Templates / Feed refresh

Polling a feed on a sensible cycle

Polling is still how most of the web notifies you. Fifteen minutes is the usual compromise between freshness and being rude to somebody else’s server.

The recipe

Request: POST https://{{domain}}/api/feeds/refresh
Schedule: Every 15 minutes (*/15 * * * *)
Timeout: 30s
Attempts: 3
Counts as success: 2xx

You supply

Your domain: example.com
Host only — no https:// and no trailing slash.

Why this job exists

Push is better than pull and is frequently unavailable. RSS and Atom have no push mechanism, most partner APIs offer no webhooks, and plenty of the ones that do will not send them to a small integrator. Polling is what remains.

The failure mode is doing it badly. Polling every minute against a feed that updates twice a day is thousands of pointless requests, and it is how integrations get rate-limited or blocked — an outcome that is tedious to reverse because it usually involves emailing somebody.

The other failure is polling without conditional requests. A feed sends its full body on every fetch unless you ask properly, so a naive poller downloads the same payload ninety-six times a day.

Setting it up

Store the ETag and Last-Modified from each response and send them back on the next request. A feed that has not changed then answers 304 with no body, which is faster for you and dramatically cheaper for the publisher:

Conditional polling

const cached = await state.get('feed:example');

const response = await fetch(FEED_URL, {
  headers: {
    ...(cached?.etag && { 'If-None-Match': cached.etag }),
    ...(cached?.lastModified && { 'If-Modified-Since': cached.lastModified }),
  },
});

if (response.status === 304) {
  return Response.json({ changed: false });
}

const items = parse(await response.text());
await ingest(items);

await state.set('feed:example', {
  etag: response.headers.get('etag'),
  lastModified: response.headers.get('last-modified'),
});

return Response.json({ changed: true, items: items.length });

Deduplicate on ingest by the feed item GUID rather than trusting the publication date. Plenty of feeds republish items with a new timestamp after an edit, and date-based deduplication turns one edit into a duplicate post.

Checking it actually works

Confirm you are receiving 304 responses between updates — if every fetch returns 200 with a body, the conditional headers are not being sent or not being honoured.
Check that a republished item updates rather than duplicating.
Read the publisher’s rate limits and any polling guidance in their robots or docs before shortening the cadence.

Where it goes wrong

Polling far faster than the feed changes

Match the cadence to the publication rhythm. A news wire justifies minutes; a company blog does not, and hourly is plenty. Fast polling on a slow feed is the behaviour that gets an integration blocked.

Reprocessing everything on every fetch

Without a stable identifier for what you already have, each poll re-ingests the whole window. Track the GUIDs and process only what is new.

Trusting the feed to be well-formed

Feeds break — unescaped entities, truncated XML, an HTML error page served with an XML content type. Parse defensively and fail the run rather than ingesting garbage.

Questions

How often is too often?

If the publisher documents a limit, follow it. Otherwise, roughly a quarter of the shortest interval between updates is a fair rule, and slower is rarely a problem.

What about WebSub or webhooks?

Use them where offered — they are strictly better. Keep a slow poll as a backstop, because push mechanisms drop messages and rarely tell you.

Can I poll several feeds in one job?

You can, but one job per feed gives you per-feed history and alerting, so a single broken publisher does not fail the whole run and hide the others.

Running it here

This is a preset in the product, not an illustration. Pick it in the dashboard, fill in the one value above, and the job is created, scheduled and enabled — with retries, a record of every run showing status, latency and response, and an alert the first time one fails. Secret values are encrypted at rest and never rendered back.

The free plan runs 3 jobs hourly and needs no card. Where a recipe wants a finer cadence than your plan allows, it is slowed to the fastest schedule you are permitted rather than rejected.

Create a free account →Check a cron expression →