Skip to main content

Message queue systems

We use RabbitMQ as our message queue system, to handle jobs and output from several microservices, including:

  • FLOSS
  • CAPA
  • olevba
  • elfparser

The remaining microservices utilize FastAPI.

RabbitMQ concepts​

Consumers can bind to queues for messages, and perform specific actions based on messages.

Producers, however, never send messages directly to queues. Instead, an exchange is used as a routing mediator. The exchange is reponsible for deciding if the message is sent to one queue, multiple queues, or is simply discarded.

Reference: RabbitMQ Exchange Types

There are many exchange types to choose from, depending on the desired routing strategy. The main exchange type that we use is topic, which determines the appropriate queue to send the message to based on the message's routing key.

Another important concept is durability. Durable queues and exchanges will be created on server restarts. However, the messages within these durable queues and exchanges may not be automatically re-created, resulting in loss of messages.

Cue, persistence of messages. Persistent messages will be written to disk instead of solely stored in memory, allowing messages to survive server reboots in unfortunate events. This comes at a cost, however, in disk I/O.

Reference: RabbitMQ persistency vs durability