Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Kafka seems to be the most popular message queue currently. Redis and memcached are more suitaable for situations where you need cache in memory.
Protocols¶
AMQP is the Advanced Message Queuing Protocol, an open standard protocol for message orientation, queuing, routing, reliability and security. The RabbitMQ messaging server is the most popular implementation of AMQP.
Servers¶
Kafka¶
Kafka leverages ZooKeeper.
celery¶
celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.
RabbitMQ¶
Redis¶
Comparisons¶
Kafka vs RabbitMQ¶
Kafka is an overkill when you need to process only a small amount of messages per day (up to several thousand). Kafka is designed to cope with the high load. Use traditional message queues like RabbitMQ when you don’t have a lot of data.
Kafka vs Redis¶
Difference between Redis and Kafka
Clients¶
rq¶
rq (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.
kombu¶
kombu The aim of Kombu is to make messaging in Python as easy as possible by providing an idiomatic high-level interface for the AMQ protocol, and also provide proven and tested solutions to common messaging problems.
pika¶
pika is a pure-Python implementation of the AMQP 0-9-1 protocol including RabbitMQ’s extensions.