Data Tips #24 - Data Retention & Data Deletion

In this article we are looking at Data Retention and Data Deletion inside the data platform. There are two typical scenarios we need to cater for.

Data lifecycle / retention window. There is normally a time period for which we are allowed, or obliged, to keep the data. A typical period is three years. This is essentially removing a whole chunk of data once the expiration date has passed.

Deletion of specific parts of the data. This occurs when a rule says that some part of the data must be deleted. The classic example is the right to be forgotten - the GDPR mechanism that allows a customer to have their data removed. Another is a customer who has stopped being a customer and must be removed after a given time. This removes parts of the data regardless of the time window.

Delete everywhere, not just in the mart

To be fully compliant we need to delete the data from all parts of the platform: the archive, the normalised storage and the use case storage. It is very common to see deletion implemented in the layer the business looks at, while copies quietly live on upstream.

Every data object should have a data retention rule set. For non-sensitive dimensional data the rule may simply be "keep". For anything personal or regulated, the rule needs an owner, a period and an implementation.

Make it a platform capability

Deletion is not a project - it is a capability. Model it once: a registry of data objects, their retention rules and the identifier used to find a person or entity across layers. Then implement one mechanism that walks that registry. Otherwise every new use case becomes a new compliance discussion.