Kafka package provides two different set of commands (.bat for windows) and .sh for Mac in the same distribution.
Table of Contents
Windows user ? Run from \bin\windows
Need to run the kafka commans from this path.
c:\ngdeveloper\softwares\kafka_2.12-2.5.0\bin\windows
Mac user ? Run from \bin
Run the commands from below path
c:\ngdeveloper\softwares\kafka_2.12-2.5.0\bin\
I am listing down only the windows commands below, but if you need to run on Mac, just find the equivalent command yourself in the above mentioned folder.
Command to start Zookeeper
zookeeper-server-start.bat ..\..\config\zookeeper.properties
Not starting ? Make sure you are running it from this path: c:\ngdeveloper\softwares\kafka_2.12-2.5.0\bin\windows
Kafka command to start the kafka Broker:
kafka-server-start.bat ..\..\config\server.properties
Is Kafka broker started successfully? do checkout this and compare your output.
Need to start multiple instances of the kafka server, then do follow this link
Kafka command to create a kafka topic
- ngdev-topic: kafka topic name to be created
- zookeeper: we already started above with 2181 port, here linking the topic with zookeeper.
- replication-factor: 1 here, can be any number, its where distributed streaming comes into picture.
- partitions – Each kafka topic contains n number of partitions, here creating the ngdev-topic with 3 partition. Remember if consumer would like to receive the same order it is sent in the producer side, then all those messages must be handled in the single partition only. If messages are shared across the partitions then consumer can’t receive the messages in the exact order where producer sent it.
kafka-topics.bat --create --topic ngdev-topic --zookeeper localhost:2181 --replication-factor 1 --partitions 3
Is it created try listing down and verify it.
Kafka command to list all the kafka topics ?
kafka-topics.bat --zookeeper localhost:2181 --list
Kafka command to start/create a kafka producer
- kafka producer is created inside the broker, so localhost:9092 is given, this is where kafka broker is running (as above)
- key separator and all basically to retain the same order
kafka-console-producer.bat --broker-list localhost:9092 --topic ngdev-topic --property "key.separator=:" --property "parse.key=true"
Is Kafka producer created successfully ? do verify with this screenshot with yours.
Kafka command to start/create a kafka consumer
Same key separator mentioned here for ordering purpose and then mentioned the bootstrap server as kafka broker 9092 running instance.
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic ngdev-topic --property "key.separator=:" --property "print.key=true"
Possible exceptions you may face:
Exception: Exception in thread “main” kafka.Zookeeper.ZooKeeperClientTimoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZookeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
Solution:
Zookeeper is not running in your local machine / server machine. Do start the zookeeper to resolve this exception.
Windows command to start up the zookeeper:
zookeeper-server-start.bat ....\config\zookeeper.properties