A complete setup for running Apache Kafka locally using Kubernetes-in-Docker (Kind) with the Strimzi operator in KRaft mode, with:
- 3 Kafka brokers
- 1 topic with 12 partitions
- Producer that first writes 50K messages, then ~10 messages per second
- Redpanda Console
- Cruise Control
- Prometheus
- Redpanda KMinion
Manifests for each of the included services can be found in the manifests folder.
If running using Colima (Docker Desktop alternative), make sure you have enough resource, e.g.
colima start --memory 16 --cpu 6 --disk 200
Before running the setup, ensure you have the following installed:
- Colima or Docker Desktop - For running Kind
- Kind - Kubernetes in Docker
- kubectl - Kubernetes command-line tool
Run
./setup.shThen follow instructions on e.g. port-forward to access RedPanda UI.
-
View demo messages:
kubectl -n kafka logs -f deployment/demo-producer
-
Consume messages:
kubectl -n kafka run kafka-consumer --image=quay.io/strimzi/kafka:0.48.0-kafka-4.1.0 --rm -it --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server kafka-cluster-kafka-bootstrap:9092 --topic test-topic --from-beginning
Create rebalance plan:
kubectl apply -f manifests/kafka-rebalance-full.yaml
After a while, it should say:
$ kubectl -n kafka get KafkaRebalance
NAME CLUSTER TEMPLATE STATUS
cluster-rebalance kafka-cluster ProposalReady
Then approve proposals:
kubectl -n kafka annotate KafkaRebalance/cluster-rebalance-full strimzi.io/rebalance=approve
Modify number of replicas, then
$ kubectl -n kafka apply -f manifests/kafka-cluster.yaml
kafkanodepool.kafka.strimzi.io/broker configured
kafka.kafka.strimzi.io/kafka-cluster unchanged
kafkarebalance.kafka.strimzi.io/cluster-auto-rebalancing-template unchanged
$ kubectl -n kafka get kafkarebalance -w
NAME CLUSTER TEMPLATE STATUS
cluster-auto-rebalancing-template true
kafka-cluster-auto-rebalancing-add-brokers kafka-cluster PendingProposal
kafka-cluster-auto-rebalancing-add-brokers kafka-cluster ProposalReady
kafka-cluster-auto-rebalancing-add-brokers kafka-cluster Rebalancing
kafka-cluster-auto-rebalancing-add-brokers kafka-cluster Ready
Run
./cleanup.sh