Kafka producer consumer command line message send/receive sample

Kafka is a distributed streaming platform, used effectively by big enterprises for mainly streaming the large amount of data between different microservices / different systems. Kafka can process upto 2Million records per second.

Pre requisite:

---

Kafka must be installed / setup in your machine. Go through the below detailed link to install kafka in your machine.

Kafka installation / setup guide on windows / mac can be found here with the detailed instructions and screenshots

Start Zookeeper

---

Path to run the zookeeper start command:

Command to run zookeeper service

zookeeper-server-start.bat ..\..\config\zookeeper.properties

Verifying zookeeper status: started with port – 2181

Start kafka broker

---

Path to run the kafka broker start command

Command to run kafka broker

kafka-server-start.bat ..\..\config\server.properties

Verifying kafka broker status: started with port – 9092

Create a topic: way where producer and consumer talk

Path to run the kafka topic command

Command to create a kafka topic

kafka-topics.bat --create --topic ngdev-topic --zookeeper localhost:2181 --replication-factor 1 --partitions 3

• ngdev-topic: kafka topic name to be created
• zookeeper: we already started above with 2181 port, here linking the topic with zookeeper.
• replication-factor: 1 here, can be any number, its where distributed streaming comes into picture.
• partitions – Each kafka topic contains n number of partitions, here creating the ngdev-topic with 3 partition. Remember if consumer would like to receive the same order it is sent in the producer side, then all those messages must be handled in the single partition only. If messages are shared across the partitions then consumer can’t receive the messages in the exact order where producer sent it.

Verifying kafka topic creation status: By listing all topics of the zookeeper

kafka topics can be listed using this command

kafka-topics.bat --zookeeper localhost:2181 --list

Start kafka producer

---

Path to run the kafka producer – start command

Command to start kafka producer

kafka-console-producer.bat --broker-list localhost:9092 --topic ngdev-topic --property "key.separator=:" --property "parse.key=true"

• kafka producer is created inside the broker, so localhost:9092 is given, this is where kafka broker is running (as above)
• key separator and all basically to retain the same order

Verifying kafka producer status: you can see “>” then started successfully

Start kafka consumer

---

Path to run the kafka consumer – start command

Create another instance and run the kafka consumer with this command

Command to start kafka consumer

kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic ngdev-topic --property "key.separator=:" --property "print.key=true"

• Same key separator mentioned here for ordering purpose and then mentioned the bootstrap server as kafka broker 9092 running instance.

Verifying kafka consumer status: No exceptions then started properly

That’s awesome. You are done.

Let’s send and receive some messages,

key separator

We have started producer and consumer with “:” as key separator, so you will not be able to post/send the messages without the key here (“:”).

To show that I have posted few messages without key and its throwing this exception (No key found on line 1:). If you post the messages with any key separating the “:”, will be properly sent from the producer and the same has been received successfully in the consumer.