Introduction

Modern IT is fully developed in terms of services and products. There are times when large amounts of data need to be streamed. It is called event-streaming, more precisely. There are many technologies used for that. Apache Kafka is an open-source tool developed by the Apache Foundation. It is needed on high-end gaming servers for log monitoring, machine learning, analytics, etc. It has a wide application in IT.

Installation

Let’s see how we can install Apache Kafka inside a Docker container. Docker is so popular and lightweight.

Step 1 – Install Docker.

Install Docker and verify it’s working. Run the below command on the command-line interface (CLI).

#docker -v

If not installed, do install it from the official doc: https://docs.docker.com/engine/install/

Step 2 – Create an independent directory.

Create a directory under /opt/ or wherever you want it to be installed.

#mkdir /opt/kafka
#cd /opt/kafka

Step 3 – Create a docker-compose file.

It’s time to write docker-compose.yml, where you can define the entire application stack in a single file. Create the docker-compose file with the below directives.

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.4
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
  kafka:
    image: confluentinc/cp-kafka:7.4.4
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

The Zookeeper and Kafka are bound together. It runs on port 22181 and 29092 respectively. Rest assured, all additional settings are in place. 

Step 4 – Start the container using docker-compose.

You can simply start it using the command below and see the output. 

#docker-compose up -d

It is successful. Time to verify that Docker is running. 

#docker ps

Step 5 – Verify that Kafka is running.

You can verify it using netcat (nc) command. 

#nc -zv localhost 22181

Step 6 – Connect to Kafka.

Kafka is running, use any kafka client to connect to it. Remember it is a single node cluster. 

Conclusion

Apache Kafka is a large data handling and streaming utility. It is open-source and reliable. It can be installed on VPS and dedicated servers in single-node and multi-node configurations.