Templates / Queue drain
Draining a queue with no worker to run it
A queue is easy; somewhere to run the consumer is not. On platforms with no long-lived process, a frequent call to a route that drains a bounded batch is how most people close the gap.
The recipe
- Request
- POST https://{{domain}}/api/queue/process
- Schedule
- Every minute (* * * * *)
- Headers
- Authorization: Bearer {{secret}}encrypted
- Timeout
- 60s
- Attempts
- 3
- Counts as success
- 2xx
You supply
- Your domain
- example.com
Host only — no https:// and no trailing slash.
- Shared secret
- s3cr3t-valuestored encrypted
Whatever your endpoint checks the caller against.
Why this job exists
The usual queue architecture assumes a daemon: a worker process that boots once, blocks on the queue, and processes jobs as they arrive. It is efficient and it is what every queue library documents first.
Serverless platforms do not offer that. Functions are invoked, run briefly, and are frozen or killed. There is no process to keep blocked on a queue, and paying for a container that exists solely to run a worker often costs more than the rest of the application.
The workaround is to invert the trigger. Instead of the worker waiting for jobs, something calls the worker and it processes whatever is waiting, up to a limit, then returns. The cadence sets your worst-case latency: a one-minute tick means a job enqueued just after a run waits up to a minute.
Setting it up
Write the route so a single invocation is bounded in both count and time. The failure to design for is a queue backlog that makes every invocation run until it is killed, losing whatever was in flight:
A drain route with an explicit budget
export async function POST(request: Request) {
if (request.headers.get('authorization') !== `Bearer ${process.env.QUEUE_TOKEN}`) {
return new Response('Forbidden', { status: 403 });
}
const deadline = Date.now() + 45_000;
let processed = 0;
while (processed < 50 && Date.now() < deadline) {
const job = await queue.reserve();
if (!job) break;
try {
await handle(job);
await queue.ack(job);
} catch (error) {
await queue.retry(job);
}
processed += 1;
}
return Response.json({ processed, drained: processed < 50 });
}Returning the count makes the run history a throughput graph. A response that consistently reports the batch limit means the queue is growing faster than the cadence drains it — raise the cadence or the batch, or both.
Checking it actually works
- Enqueue a job by hand and confirm it is processed within one cadence interval.
- Read the processed count across recent runs. Steady low numbers are healthy; a run of maximum-size batches means you are behind.
- Confirm a failing job is retried rather than acked — the drain must not swallow errors to keep its own response green.
Where it goes wrong
Unbounded batches
A drain that loops until the queue is empty will, on the day the queue is deep, run until the platform kills the function. Anything in flight at that moment is neither acked nor retried. Always bound by both count and elapsed time, and leave headroom under the timeout.
Overlapping drains
If a drain takes longer than the cadence, the next one starts alongside it and both reserve jobs. This is safe only if your queue has real reservation with a visibility timeout. If it does not, take a lock at the top of the route and return early when it is held.
A poison job blocks the head
One job that always throws will be reserved, fail, and be retried forever, consuming a slot on every run. Cap attempts and move exhausted jobs to a dead-letter queue.
Questions
How short can the cadence be?
One minute is the floor here, and on paid plans that is usually the right answer for user-visible work. For anything measured in seconds — a chat response, a payment confirmation — polling is the wrong shape and you want a real worker.
Is this worse than a real worker?
Yes, on latency and on efficiency. It is better on cost and operational surface for workloads that tolerate a minute of delay, which is most background work: emails, thumbnails, exports, webhooks.
What if a drain is still running when the next fires?
Overlap is fine when the queue reserves jobs properly. When in doubt, take a short-lived lock in the route and have the second invocation return immediately — a skipped tick costs a minute.
Running it here
This is a preset in the product, not an illustration. Pick it in the dashboard, fill in the 2 values above, and the job is created, scheduled and enabled — with retries, a record of every run showing status, latency and response, and an alert the first time one fails. Secret values are encrypted at rest and never rendered back.
The free plan runs 3 jobs hourly and needs no card. Where a recipe wants a finer cadence than your plan allows, it is slowed to the fastest schedule you are permitted rather than rejected.