Skip to main content

ADR-009: Use Redis and BullMQ for Background Processing

Status​

Accepted

Date​

YYYY-MM-DD

Context​

Chilarai requires support for background and scheduled processing.

Examples include:

  • Fee Generation
  • Late Fee Processing
  • Email Notifications
  • SMS Notifications
  • Document Processing
  • Future Integrations
  • Scheduled Maintenance Tasks

The platform requires:

  • Reliable execution
  • Retry support
  • Delayed execution
  • Scheduled execution
  • Operational visibility
  • Scalability

The platform should avoid introducing multiple background processing mechanisms.

Using different frameworks for:

  • Scheduled Jobs
  • Background Jobs
  • Delayed Jobs

would increase implementation complexity and operational overhead.

Decision​

Chilarai will use Redis and BullMQ as the standard platform for background processing.

Redis will serve as:

  • Queue Backend
  • Future Cache Provider

BullMQ will serve as:

  • Job Queue
  • Background Worker Framework
  • Delayed Task Engine
  • Scheduled Task Engine

All asynchronous and scheduled processing must be implemented using BullMQ.

The platform will not use NestJS Scheduler for production scheduling workloads.

Rationale​

Single Execution Model​

Using a single framework for:

  • Scheduled Jobs
  • Background Jobs
  • Delayed Jobs

reduces architectural complexity.

Developers only need to understand one processing model.

Reliability​

BullMQ provides:

  • Persistent Jobs
  • Retry Handling
  • Delayed Execution
  • Failure Recovery

Jobs are not lost when application instances restart.

Operational Visibility​

BullMQ provides visibility into:

  • Pending Jobs
  • Running Jobs
  • Failed Jobs
  • Retried Jobs

This simplifies troubleshooting and operations.

Scalability​

Workers may be scaled independently from API instances.

Future workloads can be distributed without redesigning business logic.

Redis Reuse​

Redis provides additional future capabilities:

  • Caching
  • Distributed Locks
  • Rate Limiting

The platform avoids introducing multiple technologies for similar concerns.

Scheduled Processing​

Scheduled tasks must be implemented using BullMQ repeatable jobs.

Examples:

  • Daily Late Fee Processing
  • Daily Notifications
  • Nightly Cleanup Jobs
  • Report Generation

The platform will not use NestJS Scheduler as a primary scheduling mechanism.

Background Processing​

Background jobs should be used for operations that are:

  • Long Running
  • Resource Intensive
  • Batch Oriented
  • Non-Blocking

Examples:

  • Fee Generation
  • Document Processing
  • Notification Delivery
  • Future Integration Synchronization

Domain Events​

BullMQ is not a replacement for Domain Events.

Domain Events remain in-process communication mechanisms.

Examples:

  • StudentCreated
  • AdmissionConverted
  • EnrollmentCreated
  • PaymentReceived

Domain Events should use NestJS EventEmitter.

Background jobs should only be introduced when asynchronous execution is required.

Consequences​

Positive​

  • Single background processing framework
  • Reliable job execution
  • Retry support
  • Delayed processing support
  • Scheduled processing support
  • Operational visibility
  • Scalable architecture
  • Future caching capability through Redis

Negative​

  • Additional Redis dependency
  • Additional operational component
  • Worker monitoring required

Alternatives Considered​

NestJS Scheduler​

Pros:

  • Simple
  • Built into NestJS

Cons:

  • No persistent jobs
  • No retries
  • Limited operational visibility
  • Separate execution model

Decision:

Rejected.

Synchronous Execution Only​

Pros:

  • Simpler architecture

Cons:

  • Long-running requests
  • Poor scalability
  • Reduced reliability

Decision:

Rejected.

Separate Message Broker​

Examples:

  • RabbitMQ
  • Azure Service Bus

Pros:

  • Advanced messaging features

Cons:

  • Additional operational complexity
  • Unnecessary for MVP

Decision:

Deferred.

May be reconsidered when microservices are introduced.

Implementation Notes​

Background processing components belong in the Data layer.

Example:

shared/data/jobs

├── queues
├── workers
├── schedulers
└── processors

Business modules should enqueue jobs through abstractions.

Workers should execute application use cases rather than containing business logic.

Decision Summary​

Chilarai will standardize on Redis and BullMQ for all background and scheduled processing.

NestJS EventEmitter will continue to be used for in-process domain