kafka consumer group

Consumer group

How to make multi-worker in same consumer-group all recieve same topic’s message?

20181106190805256.png
In kafka the topic only send one worker in consumer-group per partition(randomly?).

It’s better set the partition number multiple of consumer group’s number

So if you need pulish/subscribe mode, make sure consumer in different group id.

How do I choose the number of partitions for a topic?

The partition count determines the maximum consumer parallelism and so
you should set a partition count based on the maximum consumer parallelism you would expect to need

Partition

  • A partition is basically a directory of log files.
  • Each partition must fit entirely on one machine.
  • Each partition is totally ordered.
  • Each partition is not consumed by more than one consumer thread/process in each consumer group.
  • Many partitions can be consumed by a single process(randomly?)
  • Another way to say the above is that the partition count is a bound on the maximum consumer parallelism.
    parallelism mean different group can consume different partitions
  • More partitions will mean more files and more machine restrict.
  • Each partition corresponds to several znodes in zookeeper.
  • More partitions means longer leader fail-over time.Synchronize thousands of znodes is not a easy job.
  • More partitions the more expensive the position checkpoint is.
  • Expand more partitions need manually synchronize data from old partitions

https://www.oschina.net/question/2558468_2145935
https://blog.csdn.net/gezilan/article/details/80412490
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowmanytopicscanIhave?