Automation: Queue
Background
Concepts
Flow(Robot): contains nodes (trigger and actions).
Flow Queue: contains flow execution requests from multiple sources.
Flow Worker: consume flow queue and obtaining the flow results.
Requirements
Functional Requirements
Create Flow Queue to add flow execution requests from multiple place(Datasheet OT, Form Submit...).
Consume flow requests and executes flow(waiting).
Non-Functional Requirements
Queue tasks should be durable, doesn't get lost after shutting down the server.
Flow requests should be ordered.
When process flow requests unsuccessfully, it should be retry.
Implements
Concepts
Operation: A data structure that records the operations of a datasheet.
Changeset: The set of datasheet operations.
OP Event: Operation event is converted from operation.
Flow: contains nodes (trigger and actions).
Flow Run(Robot Histroy): The flow running record.
Process
Original robot process
New robot process
Task status
Terminology
The flow queue is based on BullMQ, which would make it durable since it stores the tasks on redis
and the library also supports repeatable tasks.
BullMQ
BullMQ is a Node.js library that implements a fast and robust queue system built on top of Redis that helps in resolving many modern age micro-services architectures. Feature:
LIFO and FIFO jobs
Priorities
Delayed jobs
Scheduled and repeatable jobs according to cron specifications
Retries of failed jobs
... ...
Q & A
What are the advantages of BullMQ compared to message broker systems like RabbitMQ?
BullMQ is a Redis-based queue specifically designed for Node.js. It is best suited for the following scenarios:
Simple and lightweight: When you need a simple, lightweight, and easy-to-use queue system with a smaller footprint.
Redis compatibility: If your project already utilizes Redis as a data store, using BullMQ could be a natural choice for leveraging the same infrastructure.
Rate limiting: BullMQ has built-in support for rate limiting, which can be useful for scenarios where you want to control the processing rate of jobs.
Error handling: BullMQ has a well-defined error handling mechanism, which provides better control over job retries and failures.
Why not use traditional message broker systems like RabbitMQ?
RabbitMQ is a better fit for complex, high-throughput, and advanced message routing scenarios. Moreover, the entity that is required to act as a job queue in this case is aimed at improving stability and decoupling modules. Currently, it is understood that BullMQ can meet the requirements of the queue.