From the above model, we can look deeper into some topics about messaging system design:
- Consumer Concurrency
- Consumer Hot Issues
- Consumer Load Balance
- Message Router
- Connection Multiplex
- Canary Deployments
A producer sends messages generated by the business application systems to brokers. RocketMQ provides multiple paradigms of sending: synchronous, asynchronous and one-way.
Producers of the same role are grouped together. A different producer instance of the same producer group may be contacted by a broker to commit or roll back a transaction in case the original producer crashed after starting the transaction.
Warning: Considering the provided producer is sufficiently powerful at sending messages, only one instance is allowed per producer group and process to avoid unnecessarily initializing of producer instances.
A Consumer pulls messages from brokers and feeds them into application. In perspective of user application, two types of consumers are provided:
Pull consumer actively pulls messages from brokers. Once batches of messages are pulled, user application initiates consuming process.
Push consumer, on the other hand, encapsulates message pulling, consuming progress maintaining and other effortful work inside, leaving a callback interface to end user to implement which will be executed on message arrival.
Similar to previously mentioned producer group, consumers of the exactly same role are grouped together and named Consumer Group.
Consumer Group is a great concept with which achieving goals of load-balance and fault-tolerance, in terms of message consuming, is super easy.
Warning: consumer instances of a consumer group must have exactly same topic subscription(s).
Topic is a category to which producers deliver messages and from which consumers pull messages. Topics have very loose relation with producers and consumers. Specifically, a topic may have zero, one or multiple producers that sends messages to it; conversely, a producer can sends messages of different topics. In consumer’s view, a topic may be subscribed by zero, one or multiple consumer groups; and a consumer group, in the same paradigm, may subscribe one or multiple topics as long as instances of this group keep their subscription consistent as emphasized in the previous section.
Message is the envelope of your information to deliver. A message must be specified with a topic, which can be interpreted as address of your letter to mail to. A message may also have an optional tag set. Extra key-value pairs may also be included. For example, you may set a business key for your message and look up the message on broker server to diagnose issues during development.
Topic, internally, is logically partitioned into one or more sub-topics. We call these sub-topics “message queues”. This concept plays a major role in implementing valuable features, including fail-over, maximum concurrency, etc.
Tag, which can be thought as sub-topic, provides an extra flexibility for user. Through introducing tag, messages with different purposes from the same business module may have the same topic yet different tag. It would be helpful to keep your code clean and coherent.
Broker is the major role of the RocketMQ system. It receives messages sent from producers, store them and being prepared to serve pull requests from consumers. It also stores message consuming related meta data, including consumer groups, consuming progress offsets and topic / queue info.
Name server serves as the routing information provider. Producer/Consumer clients look up topics to find broker list to read from and write to.
When DefaultMQPushConsumer is employed, you may decide to consume messages orderly or concurrently.
Consuming messages orderly means messages are consumed the same order they are sent by producers for each message queue. If you are dealing with scenario that global order is mandatory, make sure the topic you use has only one message queue.
Warn: If consuming orderly is specified, the maximum concurrency of message consuming is the number of message queues subscribed by the consumer group.
When consuming concurrently, maximum concurrency of message consuming is only limited by thread pool specified for each consumer client.
Warn: Message order is no longer guaranteed in this mode.