Compression: what the historian threw away and when it matters

Process historians store less than they receive. Compression algorithms — most commonly a variant of swinging door — discard points that can be reconstructed by interpolation within a defined tolerance, which reduces storage dramatically with no visible loss on a normal trend.

The tolerance is a configuration setting. Set too tight, storage grows without benefit. Set too loose, real process behaviour is discarded permanently and irrecoverably.

On most systems it is set to a product default applied uniformly to every tag.

How swinging door compression works

The principle is simple. The algorithm holds the last stored point and evaluates whether each new value can be represented, within a specified deviation, by a straight line from that point. While it can, the value is not stored. When a value falls outside the corridor, the previous point is stored and a new corridor begins.

Time records can help teams compare planned engineering effort with the work actually completed. For a related reference, see Monitask time tracking software.

A useful external reference for timing and synchronisation is NIST time and frequency services.

The result is that a steady signal stores very few points and a rapidly changing one stores many, which is exactly the behaviour wanted. The deviation setting — often called the compression deviation or exception deviation — determines how much detail survives.

Two stages, frequently confused

Most architectures apply two filters. The exception filter sits at the data collection point and decides which values are sent onward. The compression filter sits at the archive and decides which of those are stored permanently.

Both have deviation settings and they multiply. An exception deviation of one percent followed by a compression deviation of two percent produces an effective tolerance larger than either, and the interaction is frequently not understood by whoever set them.

Where default settings cause trouble

A uniform percentage-of-range deviation behaves very differently across tags.

On a temperature with a range of zero to five hundred degrees, a one percent deviation is five degrees, which discards most of the behaviour of a loop controlling to within two degrees.

On a pH measurement with a range of zero to fourteen, one percent is 0.14 pH units, which may be the entire control band.

On a valve position of zero to a hundred percent, one percent is reasonable.

The correct basis is the measurement's noise and the resolution needed to see the behaviour that matters, not a fraction of the instrument range.

The investigation that finds out

The characteristic discovery is during an incident investigation. An event happened over forty seconds, the trend shows four points, and the shape of what occurred cannot be determined.

By then nothing can be done. This is the strongest argument for reviewing compression settings on the tags associated with protective functions, critical equipment and known upset scenarios, ahead of any need.

Tags that should not be compressed

Some data should be stored at every sample regardless of storage cost.

Digital states — running, tripped, open, closed — should record every transition. Compression on a digital signal can discard a brief transition entirely, which means an event genuinely disappears rather than losing resolution.

Tags used for regulatory reporting, where the record must be demonstrably complete.

A defined set of critical process measurements around protective systems and known hazard scenarios.

Counters and totalisers, where an interpolated value is meaningless.

Checking what you currently have

Two practical tests. First, compare a raw trend from the control system against the historised version of the same period for a handful of important tags; the difference is what compression removed. Second, look at the stored point count per day for each tag — tags storing very few points are heavily compressed and worth examining.

Most historian products expose compression statistics directly, and reviewing the tags with the highest compression ratios usually identifies the misconfigured ones quickly.

Setting it deliberately

The workable approach is tiered rather than uniform. A small critical set with compression off or set very tight. A larger set of important process measurements with deviations derived from the measurement's noise band. The long tail on defaults, where the cost of being slightly wrong is low.

That requires someone to classify the tags, which is the same exercise required for sampling rates and retention, and is best done once for all three.

Setting the deviation from the measurement, not the range

The defensible basis for a compression deviation is the measurement's own noise: the deviation should be slightly larger than the noise band, so that random fluctuation is not stored while genuine movement is.

Establishing that requires looking at raw data for the tag over a steady period and observing the spread. This is a few minutes per tag, which is impractical for a hundred thousand tags and entirely practical for the few hundred that matter.

For the remainder, a sensible default applied by measurement type — one value for flows, another for temperatures, another for positions — is better than a single uniform figure and costs nothing to implement.

Interaction with control system deadbands

Filtering can occur at several points: the field device, the control system's own change-of-value threshold, the historian's exception filter, and the historian's compression.

Each removes data, and the effects compound. A site investigating why a trend looks coarse frequently finds three separate filters, each individually reasonable, none of which was set with knowledge of the others.

Mapping the full path for a few critical tags, and confirming what each stage is doing, is a worthwhile exercise and one that few sites have done.

What to check after any system change

Compression settings are frequently reset to defaults by an upgrade, a migration or a bulk configuration change, and nothing announces it.

Comparing stored point counts before and after any significant change to the historian is a quick check that catches this. A tag that stored ten thousand points a day last month and stores two hundred now has had its configuration altered, whatever anyone intended.

Documenting what was chosen and why

Compression settings are among the least documented configuration decisions in a control system, which is why they are so often defaults.

Recording the basis — the measurement's noise band, the resolution required, the resulting deviation — takes a line per tag for the tags that matter, and makes the setting reviewable rather than mysterious.

It also protects the setting from being reset during an upgrade, because someone can tell that it was deliberate.

Digital signals and event capture

Digital states deserve separate treatment from analogue measurements. A brief transition that is compressed away is not a loss of resolution; it is an event that did not happen as far as the record is concerned.

Configuring digitals to store every transition, without compression, costs very little storage because transitions are infrequent, and it preserves exactly the information investigations depend on.