Data Tips #25 - Handling re-runs

Today I want to look at a topic that happens to all teams running a data platform: how do we handle re-runs in the best manner?

Start with the purpose of the data platform. Generally it is to provide the right data at the right time to the right people. In a perfect world nothing would go wrong from the source into the platform. In reality, two scenarios keep coming back:

• We do not receive all data from the source, and the source does not send only the missing data. For instance, we may get a full day of financial transactions every night.
• There is an error in the data somewhere in the chain, so we have to reload data historically.

Both cases can be handled manually. It is easy to clean the wrong data and run the pipeline again with the correct data. But of course we do not want to handle anything manually.

Separate the platform into two layers

First, separate the data platform into an Ingestion layer and a Data layer (detail, dimensional model, use case).

In Ingestion we keep everything we have received, even if that means duplicates. Getting data into the platform should be robust, and we should never lose what a source has sent us. Ingestion is append-only and immutable.

In the Data layer we build the model that consumers read. This is where deduplication, late-arriving data and corrections are resolved - deterministically, from what Ingestion holds.

Make re-runs idempotent

The practical rule: a pipeline run for a given period must be safe to execute any number of times and always produce the same result. That means keying loads on business time rather than run time, replacing partitions instead of appending blindly, and keeping a technical timestamp so you can always tell which version of a record you are looking at.

When re-runs are idempotent, reloading history is no longer a risky operation performed by the one person who remembers how it works. It becomes a parameter.