In this tutorial, we will show you how to install and configure Apache Kafka on an Ubuntu 16.04 VPS.
Apache Kafka is one of the most popular open-source stream processing software platforms available. Developed by the Apache Software Foundation, Apache Kafka is written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. It’s a scalable, fault-tolerant, publish-subscribe messaging system that allows you to build distributed applications. Let’s begin with the installation.
Prerequisites:
- A server with a minimum of 4GB of RAM is required. This tutorial focuses on installing Apache Kafka on a VPS running Ubuntu 16.04 Server, but this may work with other distrubutions, or newer versions of Ubuntu Server.
- SSH root access to the server, or a user account with administrative access.
Step 1. Getting Started
First, you need to log in to your server via SSH as the root user or as an admin account:
ssh root@IP_ADDRESS -p PORT_NUMBER
and replace “IP_ADDRESS” and “PORT_NUMBER” with your actual server IP address and SSH port number.
Before we begin with the installation, let’s make sure that your Ubuntu 16.04 server is up-to-date by running the following commands:
apt-get update apt-get upgrade
Step 2. Install Java
Kafka is a Java based application, so we will need to install OpenJDK 8 on our server. To do this, run the following command:
apt-get install openjdk-8-jdk
To verify if the installation is completed, you can execute the following command:
java -version
The following output should be displayed on your screen:
openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.16.04.1-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
Step 3. Install ZooKeeper
ZooKeeper is an open-source service for maintaining configuration information, providing distributed synchronization, naming and providing group services, and so on. We will be using this service to manage the Kafka cluster state.
We can install ZooKeeper from the Ubuntu’s default repository with the following command:
apt-get install zookeeperd
The service should be running on port 2181 by default. To confirm if the installation is completed and if the service is running, we can run the following command:
netstat -tnlp | grep :2181
The following output confirms that the service is running:
tcp6 0 0 :::2181 :::* LISTEN
You can also use the following command to check the status of the service:
systemctl status zookeeper
To stop the service, you can run:
systemctl stop zookeeper
And to start the service:
systemctl start zookeeper
Step 4. Create a Kafka User
For security reasons, it is recommended that we create a dedicated user for our Kafka service.
To create a new user with a dedicated home directory, execute the following command:
useradd kafka -m
Once the user is created, you can set a new password for it with:
passwd kafka
Make sure you enter a strong password and press [Enter]. Next, add the user to the sudo group with:
adduser kafka sudo
And log in as the kafka user with:
su kafka
Step 5. Install Apache Kafka
Let’s create a new directory for our Apache Kafka installation with:
mkdir ~/kafka
You can then navigate to this directory with:
cd ~/kafka
Next, find the download link of the latest Apache Kafka installation from this page: https://kafka.apache.org/downloads
You can use the wget command to download the file. The command below might be an older version of Apache Kafka:
wget http://www.trieuvan.com/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz
Extract the archive in the current directory with:
tar -xvzf kafka_2.11-2.1.0.tgz --strip 1
You can also delete the original installation file with:
rm kafka_2.11-2.1.0.tgz
To create a systemd startup service for Kafka, run the following command:
nano /etc/systemd/system/kafka.service
Enter the following content and save the file:
[Unit] Requires=zookeeper.service After=zookeeper.service [Service] User=kafka ExecStart=/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
The service will also check and make sure that the ZooKeeper service we have set up previously is also up and running before it attempts to start the Kafka service.
You can now try to start the Kafka service with the following command:
systemctl start kafka
To check the status of the Kafka service and make sure it is properly started, run the following command:
systemctl status kafka
You should see the following output on your screen:
● kafka.service Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled) Active: active (running) since Wed 2018-11-28 12:31:10 CST; 9s ago Main PID: 10583 (java) CGroup: /system.slice/kafka.service
To make sure that the Kafka service gets started automatically after a server reboot, run the following command:
systemctl enable kafka
That’s it. Apache Kafka has been installed and it is now running on your server.
Of course, you don’t have to install Apache Kafka on Ubuntu 16.04 if you use one of our Managed VPS Support services, in which case you can simply ask our expert Linux admins to install and configure Apache Kafka for you, as well as set up any additional software that you might need. They are available 24×7 and will take care of your request immediately.
PS. If you liked this post on how to install Apache Kafka on Ubuntu 16.04, please share it with your friends on the social networks using the buttons below or simply leave a reply. Thanks.
