Templates / Search reindex
Rebuilding a search index overnight
Incremental indexing drifts. Updates get missed during a deploy, a webhook fails silently, a delete never propagates — and search slowly starts returning things that are not there.
The recipe
- Request
- POST https://{{domain}}/api/search/reindex
- Schedule
- At 02:30 AM (30 2 * * *)
- Headers
- Authorization: Bearer {{secret}}encrypted
- Timeout
- 300s
- Attempts
- 1 — no automatic retry
- Counts as success
- 2xx
You supply
- Your domain
- example.com
Host only — no https:// and no trailing slash.
- Shared secret
- s3cr3t-valuestored encrypted
Whatever your endpoint checks the caller against.
Why this job exists
Search indexes are a cache of your data, and like every cache they go wrong in one direction: they hold things the source no longer has. Each individual cause is minor — a failed webhook, an update inside a transaction that rolled back, a bulk import that bypassed the indexing hook — but they accumulate, and nothing removes them.
The symptom is bad in a specific way. Users search, get a result, click it, and land on a 404 or something they should not see. That reads as broken far more than an empty result page does, and it erodes trust in search generally.
A periodic full pass fixes drift by construction rather than by finding each cause. It is not elegant, and it is much cheaper than guaranteeing that every write path indexes correctly forever.
Setting it up
Index into a new index and swap an alias when it completes, rather than clearing and refilling in place. The in-place approach means search is empty or partial for the duration of the rebuild, and if the rebuild fails halfway it stays that way:
Build then swap
const next = `items_${Date.now()}`;
await search.createIndex(next, mapping);
await search.bulkIndex(next, await loadAllItems());
// Only now does anything user-facing change.
await search.moveAlias('items', next);
await search.deleteOldIndexes('items_', { keep: 2 });
return Response.json({ index: next, documents: await search.count(next) });This recipe allows one attempt and a five-minute timeout, deliberately. A reindex that failed halfway should be looked at rather than automatically repeated — retrying a heavy rebuild against a database that is already struggling is how a maintenance job becomes an outage.
Checking it actually works
- Compare the document count in the response against the row count in the source. A steady gap means something is being skipped by the loader.
- Search for something you deleted recently and confirm it is gone after a rebuild.
- Confirm the alias swap happened — the most common silent failure is a rebuild that succeeds into an index nothing queries.
Where it goes wrong
Clearing the index first
Delete-then-rebuild leaves search broken for the length of the rebuild and indefinitely if it fails. Always build alongside and swap.
Loading everything into memory
Reading the whole table to build an array works until the table grows. Paginate or stream, and index in batches — this job is written once and runs for years.
Reindexing during business hours
A full pass is heavy on the database it reads from. 02:30 in the schedule timezone is the default here for that reason; if your traffic peaks elsewhere, move it rather than accepting the convention.
Questions
Does this replace incremental indexing?
No. Incremental keeps search current within seconds; this repairs what incremental missed. Removing the incremental path and reindexing nightly means a day-old index, which is usually unacceptable.
Nightly is too often for my data.
Then run it weekly. The right cadence is roughly how long you are willing to serve a stale result, and for a catalogue that changes monthly that can be quite long.
What about Algolia, Typesense or Meilisearch?
The pattern is the same on all of them — they all support building into a secondary index and swapping. Only the client calls change.
Running it here
This is a preset in the product, not an illustration. Pick it in the dashboard, fill in the 2 values above, and the job is created, scheduled and enabled — with retries, a record of every run showing status, latency and response, and an alert the first time one fails. Secret values are encrypted at rest and never rendered back.
The free plan runs 3 jobs hourly and needs no card. Where a recipe wants a finer cadence than your plan allows, it is slowed to the fastest schedule you are permitted rather than rejected.