Retention, archiving and the data that has to still be there

How long process data is kept is a policy question that is normally answered by a default. The answer surfaces when someone needs data from beyond the retention window — an incident investigation, a warranty claim, a regulatory request, a long-term degradation analysis — and finds it gone.

The competing requirements

Operational troubleshooting needs recent data at full resolution. Most queries are within days or weeks.

Incident investigation needs full resolution around specific past events, and the need can arise long after the event, particularly where a problem is discovered late.

Annual work-hour assumptions affect resource plans, maintenance forecasts and lifecycle budgets. Further details are available in the linked article.

A useful external reference for timing and synchronisation is NIST time and frequency services.

Performance and reliability analysis needs years of data, and usually at reduced resolution — daily or hourly aggregates are sufficient for equipment degradation trends.

Regulatory and environmental reporting may carry a statutory minimum retention with evidential requirements about completeness and integrity. These vary by jurisdiction and by sector and are the one category where the requirement is external rather than a judgement.

A tiered scheme

The common structure keeps full resolution online for a defined recent period, then reduces resolution for a longer period, then retains a defined subset indefinitely.

Reduction is normally by aggregation: replacing the detailed record with periodic minimum, maximum, average and standard deviation. This preserves the ability to see long-term trends and loses the ability to see individual events, which is the correct trade for older data.

The subset retained indefinitely should be defined explicitly — regulatory tags, key production figures, critical equipment measurements — rather than emerging by accident.

Archiving is not backup

These are routinely confused and serve different purposes.

An archive is the long-term retention of data that has been moved out of the active system, and it must remain readable and queryable.

A backup is a copy taken for recovery after failure, and it must be restorable.

A site with good backups and no archive can recover from a server failure and cannot answer a question about last year. A site with an archive and no backup can answer historical questions right up until the moment the system fails.

Readability over time

Archived data has to remain readable for its retention period, which may exceed the life of the system that wrote it.

Data archived in a vendor's proprietary format, on media that requires that vendor's software, becomes inaccessible when the system is replaced. This is a real and recurring failure: sites discover during a migration that fifteen years of history cannot be brought forward.

The mitigations are to confirm the export path before committing to a platform, to periodically verify that archived data can actually be read, and for the small set of genuinely long-retention data to hold a copy in an open format.

Testing the restore

An archive that has never been read is an assumption. The test is to select a random period from several years ago and retrieve it.

Doing this annually finds the problems — missing volumes, unreadable media, a format the current version no longer imports — while they are still correctable.

Deleting deliberately

Retention has an upper as well as a lower bound. Data kept beyond its purpose occupies storage, slows queries and, where it contains personal data such as operator identities in event logs, may carry obligations about how long it may be held.

A retention policy should state what is deleted as well as what is kept, and the deletion should happen on a schedule rather than when someone notices the disk is full.

Writing it down

The policy is a short document: the categories, the retention for each, where each is stored, who owns it, and the review cycle. Most sites do not have one, and the effective policy is whatever the system was configured with by an integrator in a hurry.

Storage growth and forecasting it

Historian storage grows with tag count, sampling rate and retention, and the growth is frequently discovered when a disk fills rather than predicted.

A simple forecast — current consumption, growth rate over the last year, projected date at which capacity is reached — takes an hour and turns a future incident into a budget line.

It also makes the trade-offs concrete: extending retention by two years has a storage cost that can be stated, which allows a decision rather than a preference.

Who decides retention

Retention appears to be a technical setting and is substantially a business decision, because the consequences of insufficient retention fall on operations, legal, environmental and finance rather than on the control systems group.

The practical approach is a short consultation: asking each of those functions what they may need and for how long. The answers are frequently longer than the current setting, and occasionally shorter, and both are useful to know.

Data that must be demonstrably complete

Where data supports regulatory reporting, the requirement is usually not only that it exists but that its completeness and integrity can be demonstrated.

That places additional requirements: gap detection and reporting, records of any substitution, controls over who can modify stored values, and an audit trail of changes. Whether these apply depends on jurisdiction and sector, and where they do, they should be established at configuration rather than retrofitted before an inspection.

Retention and system replacement

Retention periods that exceed the expected life of the historian platform create an obligation that must survive a migration, and that obligation is frequently overlooked when the platform is chosen.

Where data must be kept for fifteen years and the system will be replaced in eight, the retention requirement is really a requirement about export capability and archive format.

Establishing that at procurement, rather than at migration, is what makes the obligation achievable.

What happens when storage fills

Historian behaviour when storage is exhausted varies: some systems overwrite the oldest data, some stop collecting, some fail.

Knowing which applies matters, because two of those three lose current data rather than old data, which is considerably worse.

Monitoring free capacity with an alarm at a threshold that allows time to act converts a failure into a scheduled task.