Redis-Backed Queuing Pipelines That Don't Fall Over

Most "slow API" problems aren't slow databases. They're slow work happening on the request path — a 200k-row CSV import, a campaign firing to tens of thousands of contacts, a code submission compiling against 40 test cases. The user is stuck watching a spinner while your worker thread is pinned.

The fix is almost always the same shape: get the work off the request, put it in a durable queue, and let a pool of workers drain it. I've built this pattern twice now under real load — once for a WhatsApp campaign platform, once for a competitive-programming judge — and the interesting parts are never the happy path.

The shape

A producer accepts the request, validates it, enqueues a job, and returns immediately with a job id. A separate processor pulls jobs off the queue and does the heavy lifting. Redis sits in the middle as the queue backend.

@Injectable()
export class SubmissionService {
  constructor(
    @InjectQueue("submissions") private readonly queue: Queue,
  ) {}
 
  async submit(dto: SubmitDto): Promise<{ jobId: string }> {
    const job = await this.queue.add("run", dto, {
      attempts: 3,
      backoff: { type: "exponential", delay: 2000 },
      removeOnComplete: 100,
    });
    return { jobId: String(job.id) };
  }
}

The controller now returns in milliseconds. The actual execution — bundling test cases, calling the judge, comparing output — happens in the processor:

@Processor("submissions")
export class SubmissionProcessor {
  @Process("run")
  async run(job: Job<SubmitDto>) {
    const result = await this.runner.execute(job.data);
    await this.repo.save(result);
    this.gateway.emitResult(job.data.userId, result); // real-time push
  }
}

The parts that actually matter

Retries with backoff. External calls fail — the judge times out, a rate limit trips, a DNS blip happens. attempts: 3 with exponential backoff turns a transient failure into a non-event instead of a lost job. But retries are only safe if the work is idempotent: design the processor so running the same job twice produces the same result. Save by a deterministic key, not by appending.

Backpressure. A queue doesn't make work free — it makes it deferrable. If producers enqueue faster than workers drain, the backlog grows without bound. Cap concurrency on the processor, and if the queue depth crosses a threshold, reject new work at the edge with a clear signal rather than silently piling on.

Progress you can actually show. For a bulk import, "processing…" for four minutes is unacceptable. I write incremental progress to Redis as the worker advances, and the client polls (or subscribes) for it:

async importContacts(job: Job<ImportDto>) {
  const rows = await parse(job.data.fileUrl);
  for (const [i, row] of rows.entries()) {
    await this.insert(row);
    await job.progress(Math.round((i / rows.length) * 100));
  }
}

Where it bit me

The bugs that cost me the most weren't in the queue logic — they were at the edges. A timezone mismatch between the JVM and the database (PKT vs UTC) meant jobs scheduled for "midnight" fired an hour off, and it only showed up in analytics. An AES-GCM decryption failure — a tag mismatch — surfaced as a cryptic worker crash because the payload was being re-encrypted on retry with a fresh nonce that the consumer didn't expect.

The lesson: the queue is the easy part. What breaks is serialization, time, and encryption at the boundaries — the assumptions that hold on the request path and quietly stop holding when work moves across a process and a few seconds of latency.

When not to reach for a queue

If the work finishes in under ~200ms and can't fail partway, a queue is just latency and operational surface you don't need. Reach for it when the work is slow, bursty, retryable, or when you need to survive the process restarting mid-flight. That's the bar. Everything else is premature infrastructure.