Kafka and Kafka connect command cheatsheet

These commands I used many times. This page notes down them for quick reference in future.

Kafka Connect

Kafka connect uses Rest API so we can interact with a simple curl command.

Commands for connectors

#sudo apt instal jq -y
curl -X GET http://localhost:8083/connectors| jq
curl -X GET http://localhost:8083/connectors?expand=status | jq
curl -X GET http://localhost:8083/connectors?expand=info | jq
curl -X GET http://localhost:8083/connectors/<connector-name> | jq
curl -X DELETE http://localhost:8083/connectors/<connector-name> | jq

1

Get list of all connectors

2

Get status of all connectors

3

Get info of all connectors

4

Get a connector name and use it here to get information of it

5

Delete a respective connector

To know more on jq read here.

Kafka commands

Commands for Kafka

docker exec -it kafka bin/kafka-topics.sh --list --bootstrap-server localhost:9092
docker exec -it kafka bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic debezium.commerce.products --from-beginning
docker exec -it kafka bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test_debezium.commerce.products --from-beginning | grep -i "\"price\":"

1

List all topics connected to the bootstrap-server

2

Consume messages with the console consumer

3

Consume messages with the console consumer but with a grep filter.

Kafka configuration

kafka configuration

  zookeeper:
    image: debezium/zookeeper:2.4
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - my_network
  kafka:
    container_name: kafka
    image: debezium/kafka:latest
    ports:
      - "9093:9093"
    environment:
      - ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_INT:PLAINTEXT,LISTENER_EXT:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=LISTENER_INT://kafka:9092,LISTENER_EXT://localhost:9093
      - KAFKA_LISTENERS=LISTENER_INT://0.0.0.0:9092,LISTENER_EXT://0.0.0.0:9093
      - KAFKA_INTER_BROKER_LISTENER_NAME=LISTENER_INT
    depends_on:
      - zookeeper
    networks:
      - my_network

networks:
  my_network: