NORTHSTACKNORTHSTACKAny system that pulls data from someone else's service inherits their failures. Feeds go down, formats change without warning, and rate limits arrive unannounced. The naive version of this job is a script on a timer, and it works right up until the day something upstream is different, at which point it either crashes loudly or, far worse, writes bad data quietly.
The harder problem is what happens on the second run. A pipeline that fetches the same day twice needs to produce the same result twice. Without that, every retry risks duplicating records, and every recovery from an outage becomes a manual reconciliation job.
A two-stage pipeline that separates fetching from parsing. Raw payloads are written to disk and recorded in a ledger before anything is interpreted, so a parsing bug never means re-fetching, and a fetch is never repeated unnecessarily.
Every write is an upsert on a natural key, and every fetch is checked against the ledger first. Running the same day twice is a no-op rather than a duplication. A run that crashes halfway leaves its own breadcrumbs, with attempt counts and the last error recorded per item, and the next scheduled run picks up exactly where it stopped without being told to.
Thirty-seven scheduled jobs handle ingestion, aggregation, model fitting, integrity checks and backups. Missed runs execute on next boot rather than being skipped. Parsers classify their own failures, so a genuine upstream format change raises an alert while routine unknown values are logged for later review, rather than both being treated as noise.
Backups run nightly to object storage, with the upload confirmed rather than assumed, retention rotated automatically, and monthly snapshots kept indefinitely.
An aggregation job stopped working and nobody noticed for nine days.
It hadn't crashed. One statement in it had grown slow enough, against a table that had been getting larger for weeks, to exceed the time budget the job was given. The process manager killed it on timeout, exactly as configured, and moved on. The job's own logs showed it starting every morning. Nothing reported a failure, because from the system's point of view nothing had failed.
What made it invisible was the same thing that made it survivable: the views it refreshes still existed and still returned data. They were just steadily getting older, and stale data looks identical to fresh data unless you check the timestamp.
The fix was three parts. A realistic time budget for a job whose runtime scales with a growing table. Failure alerting wired directly into the process manager rather than left to the script. And a scheduling change, moving it clear of another memory-hungry job it had been silently competing with.