How to Install Apache Kafka on Ubuntu 16.04

In this tutorial, we will show you how to install and configure Apache Kafka on an Ubuntu 16.04 VPS.

Apache Kafka is one of the most popular open-source stream processing software platforms available. Developed by the Apache Software Foundation, Apache Kafka is written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. It’s a scalable, fault-tolerant, publish-subscribe messaging system that allows you to build distributed applications. Let’s begin with the installation.

Prerequisites:

• A server with a minimum of 4GB of RAM is required. This tutorial focuses on installing Apache Kafka on a VPS running Ubuntu 16.04 Server, but this may work with other distrubutions, or newer versions of Ubuntu Server.
• SSH root access to the server, or a user account with administrative access.

Step 1. Getting Started

First, you need to log in to your server via SSH as the root user or as an admin account:

ssh root@IP_ADDRESS -p PORT_NUMBER

and replace “IP_ADDRESS” and “PORT_NUMBER” with your actual server IP address and SSH port number.

Before we begin with the installation, let’s make sure that your Ubuntu 16.04 server is up-to-date by running the following commands:

apt-get update
apt-get upgrade

Step 2. Install Java

Kafka is a Java based application, so we will need to install OpenJDK 8 on our server. To do this, run the following command:

apt-get install openjdk-8-jdk

To verify if the installation is completed, you can execute the following command:

java -version

The following output should be displayed on your screen:

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Step 3. Install ZooKeeper

ZooKeeper is an open-source service for maintaining configuration information, providing distributed synchronization, naming and providing group services, and so on. We will be using this service to manage the Kafka cluster state.

We can install ZooKeeper from the Ubuntu’s default repository with the following command:

apt-get install zookeeperd

The service should be running on port 2181 by default. To confirm if the installation is completed and if the service is running, we can run the following command:

netstat -tnlp | grep :2181

The following output confirms that the service is running:

tcp6 0 0 :::2181 :::* LISTEN

You can also use the following command to check the status of the service:

systemctl status zookeeper

To stop the service, you can run:

systemctl stop zookeeper

And to start the service:

systemctl start zookeeper

Step 4. Create a Kafka User

For security reasons, it is recommended that we create a dedicated user for our Kafka service.

To create a new user with a dedicated home directory, execute the following command:

useradd kafka -m

Once the user is created, you can set a new password for it with:

passwd kafka

Make sure you enter a strong password and press [Enter]. Next, add the user to the sudo group with:

adduser kafka sudo

And log in as the kafka user with:

su kafka

Step 5. Install Apache Kafka

Let’s create a new directory for our Apache Kafka installation with:

mkdir ~/kafka

You can then navigate to this directory with:

cd ~/kafka

Next, find the download link of the latest Apache Kafka installation from this page: https://kafka.apache.org/downloads

You can use the wget command to download the file. The command below might be an older version of Apache Kafka:

wget http://www.trieuvan.com/apache/kafka/2.1.0/kafka_2.11-2.1.0.tgz

Extract the archive in the current directory with:

tar -xvzf kafka_2.11-2.1.0.tgz --strip 1

You can also delete the original installation file with:

rm kafka_2.11-2.1.0.tgz

To create a systemd startup service for Kafka, run the following command:

nano /etc/systemd/system/kafka.service

Enter the following content and save the file:

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
User=kafka
ExecStart=/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target

The service will also check and make sure that the ZooKeeper service we have set up previously is also up and running before it attempts to start the Kafka service.

You can now try to start the Kafka service with the following command:

systemctl start kafka

To check the status of the Kafka service and make sure it is properly started, run the following command:

systemctl status kafka

You should see the following output on your screen:

● kafka.service
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2018-11-28 12:31:10 CST; 9s ago
Main PID: 10583 (java)
CGroup: /system.slice/kafka.service

To make sure that the Kafka service gets started automatically after a server reboot, run the following command:

systemctl enable kafka

That’s it. Apache Kafka has been installed and it is now running on your server.

Of course, you don’t have to install Apache Kafka on Ubuntu 16.04 if you use one of our Managed VPS Support services, in which case you can simply ask our expert Linux admins to install and configure Apache Kafka for you, as well as set up any additional software that you might need. They are available 24×7 and will take care of your request immediately.

PS. If you liked this post on how to install Apache Kafka on Ubuntu 16.04, please share it with your friends on the social networks using the buttons below or simply leave a reply. Thanks.