Project
Buffalomq
A lightweight Redis-backed job queue for Python, with delayed jobs, retries, worker recovery, and a built-in monitoring dashboard.
- Python
- Redis
- Lua
- FastAPI
- React
Overview
BuffaloMQ is a lightweight job queue for Python applications that need to process work in the background. It supports delayed jobs, retries with exponential backoff, queue controls, and a built-in monitoring dashboard.
I built it to explore a simpler approach to background processing: a small Python API, minimal dependencies, and Redis-native operations. The goal was to make job processing easier to understand and operate while addressing practical concerns such as worker crashes, long-running tasks, and inspecting failed work.
My contribution
I built the Python package around producer and worker APIs, with Redis storing job data and queue state. I implemented delayed scheduling, retry handling, pause and resume controls, and the ability to return failed jobs to the waiting queue.
For worker recovery, I implemented visibility timeouts, a sweeper for expired active jobs, and heartbeats that extend processing leases for long-running tasks. Lua scripts handle job state transitions atomically inside Redis.
Technical notes
Keeping state transitions atomic:
Moving a job between states involves several related Redis operations. If those operations happen separately, concurrent workers or an interruption can leave inconsistent state. I used Lua scripts to perform these transitions atomically, keeping coordination close to the data.
The tradeoff is that some core behavior lives outside Python, making the scripts another part of the system to maintain and debug.
Recovering work after a worker stops:
Jobs receive a visibility timeout when a worker claims them. Heartbeats extend that lease during long-running tasks, while a sweeper recovers jobs whose leases expire.
This highlighted an important boundary: recovery can cause a job to run again if processing succeeds but completion is not recorded. Applications therefore need idempotent processors where repeated execution could create duplicate effects. The next useful validation is targeted failure testing around worker termination, lease expiry, and Redis interruptions.
Published on
Back to projects