Redis as a Queue: Lists vs Streams

Redis

Chapter 7 · Redis as a Queue: Lists vs Streams

📬 Redis as a Queue: Lists vs Streams

Chapter 6 drew the line: pub/sub is fire-and-forget, a queue persists until processed. node3-6 mentioned "Redis Streams" as one queue-like option without teaching it. This chapter covers both real options — Chapter 3's Lists reused as a simple queue, and the more capable Streams structure with consumer groups.

Simple List-Based Queues

Chapter 3's RPUSH/LPUSH already build an ordered list — reused as a queue, producers RPUSH jobs onto one end, and workers BLPOP (blocking left-pop) from the other, waiting efficiently rather than polling in a loop.

# Producer
127.0.0.1:6379> RPUSH jobs "send-email:104"
(integer) 1

# Worker — blocks until a job appears, 0 = wait forever
127.0.0.1:6379> BLPOP jobs 0
1) "jobs"
2) "send-email:104"

This is the simplest possible queue — and it has real limits: once BLPOP delivers a job, it's removed from the list immediately. If the worker crashes before finishing, that job is simply gone — no retry, no record it was ever handed out, and no way for a second worker to notice and pick up the slack.

Redis Streams

A Stream is an append-only log — closer to a lightweight Kafka-style log than a simple list. XADD appends an entry (auto-assigned a unique, timestamp-based ID); XREAD reads entries without removing them, meaning multiple independent readers can each track their own position through the same history.

127.0.0.1:6379> XADD orders * customer "C104" total "42.50"
"1720000000000-0"
127.0.0.1:6379> XREAD COUNT 10 STREAMS orders 0
1) 1) "orders"
   2) 1) 1) "1720000000000-0"
         2) 1) "customer" 2) "C104" 3) "total" 4) "42.50"

The * in XADD tells Redis to auto-generate the entry's ID. Unlike a list's BLPOP, reading a stream doesn't consume the entry — the same data can be read again later, or by a completely different reader.

Consumer Groups

Consumer groups solve the list queue's crash-recovery problem directly: multiple workers cooperatively consume from one stream, each entry delivered to exactly one worker in the group, while the stream itself keeps the full history intact.

# Create a consumer group starting from the beginning of the stream
127.0.0.1:6379> XGROUP CREATE orders workers 0

# A worker reads its next assigned entry
127.0.0.1:6379> XREADGROUP GROUP workers worker1 COUNT 1 STREAMS orders >

# The worker acknowledges successful processing
127.0.0.1:6379> XACK orders workers 1720000000000-0

If worker1 crashes before calling XACK, the entry stays in a pending state, visible via XPENDING — another worker can reclaim it with XCLAIM and finish the job, exactly the crash-recovery guarantee the simple list queue never had.

Lists vs. Streams

List Queue

Simple, fast, no history — once popped, a job is gone forever with no record it existed, no acknowledgment, no crash recovery.

Stream + Consumer Group

Full history retained, multiple cooperating workers, acknowledgment-based reliability, and reclaimable jobs if a worker crashes mid-processing.

Command	Structure	Purpose
`RPUSH` / `BLPOP`	List	Simplest producer/consumer queue
`XADD`	Stream	Append an entry to a stream
`XREAD`	Stream	Read entries without removing them
`XGROUP CREATE`	Stream	Create a consumer group on a stream
`XREADGROUP`	Stream	Read as part of a consumer group — each entry to one worker
`XACK`	Stream	Acknowledge an entry as successfully processed
`XPENDING` / `XCLAIM`	Stream	Inspect / reclaim entries a crashed worker never acknowledged

A Simple List Queue Is Enough When

Jobs are low-stakes and losing one occasionally on a crash is an acceptable trade-off for maximum simplicity — a single-worker background task, for instance.

Reach for Streams + Consumer Groups When

Multiple workers need to share the load reliably, a crashed worker's job must be recoverable, or the message history itself has value (auditing, replay).

💻 Coding Challenges

Challenge 1: Build a Simple Queue

Write the commands for a producer to push two jobs ("resize-image:1", "resize-image:2") onto a list called image-jobs, and for a worker to block-pop one of them.

Goal: Practice the basic RPUSH/BLPOP producer/consumer pair.

→ Solution

Challenge 2: Diagnose a Lost Job

A team uses a simple RPUSH/BLPOP list queue. A worker crashes mid-processing, and the job it was handling is never completed or retried. Explain why this happened, and what Redis structure would have prevented it.

Goal: Practice connecting the list queue's known limitation to a concrete failure and its fix.

→ Solution

Challenge 3: Design a Consumer-Group Flow

Design the commands for three workers (worker1, worker2, worker3) to cooperatively process entries from a stream called notifications as a single consumer group called notifiers, including how a worker confirms it finished an entry.

Goal: Practice the full XGROUP CREATE → XREADGROUP → XACK lifecycle for multiple cooperating workers.

→ Solution

🎯 What's Next

The next chapter is Persistence & Durability — RDB snapshotting vs. AOF, and when Redis data loss is (and isn't) an acceptable trade-off.