Kafka

基本信息

属性	值
版本	3.5.1-debian-11
Helm 目录	k8s-apps/04-helm/03-kafka/
Namespace	yidun-infra
Release 名称	kafka-yidun
节点标签	kafka: "true"
镜像	private-registry.nis.netease.com/bitnamilegacy/kafka:3.5.1-debian-11-r1

关键配置

副本数

replicaCount: 1   # 单机模式
# replicaCount: 3  # 集群模式（自动调整）

JVM 内存

heapOpts: -Xmx2048m -Xms2048m

日志保留策略

logRetentionHours: 168       # 7天
logRetentionBytes: 1073741824  # 1GB 按大小滚动
logSegmentBytes: 1073741824    # 1GB 段大小

分区与副本

numPartitions: 1
defaultReplicationFactor: 1         # 集群模式自动改为 3
offsetsTopicReplicationFactor: 1
transactionStateLogReplicationFactor: 1
transactionStateLogMinIsr: 1

资源限制

resources:
  requests:
    cpu: 1
    memory: 2048Mi
  limits:
    cpu: 3
    memory: 3000Mi

认证

默认使用 plaintext（无认证），适合内网私有化部署场景：

auth:
  clientProtocol: plaintext
  interBrokerProtocol: plaintext

服务地址

集群内访问

# 单节点
kafka-yidun-0.kafka-yidun-headless.yidun-infra.svc.cluster.local:9092

# 三节点
kafka-yidun-0.kafka-yidun-headless.yidun-infra:9092
kafka-yidun-1.kafka-yidun-headless.yidun-infra:9092
kafka-yidun-2.kafka-yidun-headless.yidun-infra:9092

yidun-text 的 values.yaml 中的配置示例：

kafka:
  bootstrapServers: "kafka-yidun-0.kafka-yidun-headless.yidun-infra:9092,\
                     kafka-yidun-1.kafka-yidun-headless.yidun-infra:9092,\
                     kafka-yidun-2.kafka-yidun-headless.yidun-infra:9092"

常用运维

# 查看 Kafka Pod 状态
kubectl get pod -n yidun-infra -l app.kubernetes.io/name=kafka

# 进入 Kafka Pod
kubectl exec -it kafka-yidun-0 -n yidun-infra -- bash

# 查看 topic 列表
kubectl exec kafka-yidun-0 -n yidun-infra -- \
  kafka-topics.sh --list --bootstrap-server localhost:9092

# 查看消费者组
kubectl exec kafka-yidun-0 -n yidun-infra -- \
  kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

# 查看特定 topic 的 offset
kubectl exec kafka-yidun-0 -n yidun-infra -- \
  kafka-consumer-groups.sh --describe --group <group-name> \
  --bootstrap-server localhost:9092

故障排查

Kafka Pod 无法启动

# 检查日志
kubectl logs kafka-yidun-0 -n yidun-infra

# 常见原因 1: ZooKeeper 未就绪
kubectl get pod -n yidun-infra -l app.kubernetes.io/name=zookeeper

# 常见原因 2: 存储空间不足
kubectl describe pvc -n yidun-infra
df -h  # 检查节点磁盘

消息积压

# 查看 consumer group lag
kubectl exec kafka-yidun-0 -n yidun-infra -- \
  kafka-consumer-groups.sh --describe --group <group> \
  --bootstrap-server localhost:9092 | grep -v "0$"