Kafka is a distributed streaming platform, used effectively by big enterprises for mainly streaming the large amount of data between different microservices / different systems. Kafka can process upto 2Million records per second.
Table of Contents
Pre requisite:
Kafka must be installed / setup in your machine. Go through the below detailed link to install kafka in your machine.
Start Zookeeper
Path to run the zookeeper start command:
Command to run zookeeper service
zookeeper-server-start.bat ..\..\config\zookeeper.properties
Verifying zookeeper status: started with port – 2181
Start kafka broker
Path to run the kafka broker start command
Command to run kafka broker
kafka-server-start.bat ..\..\config\server.properties
Verifying kafka broker status: started with port – 9092
Create a topic: way where producer and consumer talk
Path to run the kafka topic command
Command to create a kafka topic
kafka-topics.bat --create --topic ngdev-topic --zookeeper localhost:2181 --replication-factor 1 --partitions 3
- ngdev-topic: kafka topic name to be created
- zookeeper: we already started above with 2181 port, here linking the topic with zookeeper.
- replication-factor: 1 here, can be any number, its where distributed streaming comes into picture.
- partitions – Each kafka topic contains n number of partitions, here creating the ngdev-topic with 3 partition. Remember if consumer would like to receive the same order it is sent in the producer side, then all those messages must be handled in the single partition only. If messages are shared across the partitions then consumer can’t receive the messages in the exact order where producer sent it.
Verifying kafka topic creation status: By listing all topics of the zookeeper
kafka topics can be listed using this command
kafka-topics.bat --zookeeper localhost:2181 --list
Start kafka producer
Path to run the kafka producer – start command
Command to start kafka producer
kafka-console-producer.bat --broker-list localhost:9092 --topic ngdev-topic --property "key.separator=:" --property "parse.key=true"
- kafka producer is created inside the broker, so localhost:9092 is given, this is where kafka broker is running (as above)
- key separator and all basically to retain the same order
Verifying kafka producer status: you can see “>” then started successfully
Start kafka consumer
Path to run the kafka consumer – start command
Create another instance and run the kafka consumer with this command
Command to start kafka consumer
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic ngdev-topic --property "key.separator=:" --property "print.key=true"
- Same key separator mentioned here for ordering purpose and then mentioned the bootstrap server as kafka broker 9092 running instance.
Verifying kafka consumer status: No exceptions then started properly
That’s awesome. You are done.
Let’s send and receive some messages,
key separator
We have started producer and consumer with “:” as key separator, so you will not be able to post/send the messages without the key here (“:”).
To show that I have posted few messages without key and its throwing this exception (No key found on line 1:). If you post the messages with any key separating the “:”, will be properly sent from the producer and the same has been received successfully in the consumer.
