Cauli: a task queue for Python where one Rust binary runs every lane.

The problem

Celery needs a separate worker per pool type. A mixed workload means several deployments, several config files, and a decision about prefetch that quietly trades reliability for speed. Async def tasks it cannot run at all.

Cauli routes with kind="cpu" on the task instead. Async runs on an event loop with uvloop, blocking runs on a thread pool, CPU bound forks to child processes for real multicore. One process, one flag.

What it looks like

from cauli import Cauli

app = Cauli(redis_url="redis://localhost:6379/0")

@app.task(max_retries=5)
def send_email(to: str):
    ...

@app.task()                       # async def just works
async def call_api(url: str):
    ...

@app.task(kind="cpu", timeout=120)
def crunch(data: list[int]):
    ...

Then run the worker. -c 50 means 50 tasks in flight, the unit Sidekiq and Hangfire use, not 50 processes. Process, thread and slot counts all derive from it.

pip install cauli
cauli-worker -A myproj.tasks:app -c 50 --print-plan

The prebuilt worker binary ships in the wheel. No Rust toolchain, no compiler, no second step.

Dispatch throughput

No database, no ORM, no web framework. The task body is one redis.incr, so this measures the queue runtime and nothing else. The cores column is what tells you whether a number is a real ceiling or just a saturated box.

WSL2, 6 shared vCPUs. Redis, the workers and the harness all compete for the same cores, so treat every number as directional.

The ratio is not the interesting part. Celery is limited by Celery at 1.13 cores, leaving four of six idle, and cannot turn more hardware into more throughput on this workload. Cauli saturates the machine instead. That is the structural difference and the ratio is just its consequence.

Cauli's actual ceiling is unknown. Every row is box bound at 4.6 to 4.9 cores while sharing six cores with the broker. The defensible claim is "at least 27,060 per second using 4.71 cores", not "Cauli maxes out here".

Correctness under a hard crash

kill -9 the worker at 160 of 500 uniquely tagged tasks, restart it, count what comes out. Counts, not rates, so nothing depends on how busy the box was.

Celery's default loss is not a number, it is a formula, and it predicted three configurations exactly: 8, 20 and 80 tasks lost. It loses what the worker had prefetched into memory, with no path back. The uncomfortable part is that prefetch is Celery's main throughput knob, so the harder you tune it for speed, the more it loses when a box dies.

Cauli loses none at any concurrency, because recovery is driven by the Redis consumer group pending list rather than by whatever happened to be inside a worker process.

Where it loses

A table that only shows wins is rigged.

• Memory, unless you pin --procs. Each worker process embeds its own CPython and costs about 22.6 MiB before it holds a single task. Celery gevent is one process with N greenlets and pays that floor once.
• CPU bound parity at 50ms. Cauli and Celery are 1 percent apart on 50ms tasks and 6 percent at 10ms. The lead only exists for short tasks.
• The wrong config stalls under a CPU burst. Route CPU work with kind="cpu" or a 50ms burst pushes the naive async lane to 18× baseline p99.
• Throughput falls off a cliff above 104 slots per process. A run reaches 91 to 99 percent and then hangs instead of slowing down.
• Django needs pgbouncer at high concurrency. At --procs 12 --io-threads 80 the Django lane asks Postgres for 960 connections.

What is not measured is stated too. No soak result survived a host outage, there is no published latency table, and no duplicate delivery test exists yet. Absence is not evidence.

Limitations, up front

• Redis only. No RabbitMQ, SQS or Pub/Sub.
• Linux only worker, x86_64 and aarch64, glibc 2.35 or newer. The client runs anywhere.
• At least once, not exactly once. Idempotency keys narrow the window, they do not close it.
• No workflow primitives. No chains, groups, chords or canvas.
• No web UI. Stats go to stdout as structured logs.