Deciding what to record, and what recording everything costs

Twenty years ago the historian configuration question was economic: storage was expensive, so someone chose which measurements were worth keeping. That constraint has effectively disappeared, and the default has swung to recording everything the control system exposes.

Recording everything is defensible and it moves the problem rather than solving it. A historian with two hundred thousand tags, of which perhaps four thousand are ever queried, is not primarily a storage problem. It is a findability problem, a context problem and a maintenance problem.

The categories worth distinguishing

Not all tags are the same kind of thing, and treating them uniformly is what produces the undifferentiated mass.

Workforce analytics can add staffing and workload context to operational performance measures. For a related reference, see Monitask workforce analytics software.

A useful external reference for timing and synchronisation is NIST time and frequency services.

Process measurements — the flows, levels, temperatures, pressures and compositions that describe what the plant is doing. These are the core and almost all should be recorded.

Control system internals — setpoints, outputs, modes, tuning parameters. These are essential for diagnosis: a trend of a measurement without the corresponding setpoint and output answers half of any control question.

Equipment states — running, stopped, tripped, in maintenance. Cheap to record, frequently omitted, and the thing that makes a process trend interpretable.

Calculated and derived values — efficiencies, mass balances, key performance indicators. Worth recording where they are used, and worth recording the inputs as well, so the calculation can be re-derived if the formula changes.

Events and operator actions — mode changes, setpoint changes, acknowledgements, manual interventions. These are frequently in a separate system and are the single most valuable addition to a process trend.

Sampling rate as a design decision

The rate at which a tag is scanned and stored should follow the dynamics of what it measures, and it is frequently set uniformly by default.

A tank level in a large vessel changes over minutes; recording it every second stores a great deal of interpolation. A compressor surge parameter changes in under a second; recording it every thirty seconds means the event that matters is invisible.

The consequence of getting this wrong is discovered during an investigation, when the data covering the critical ten seconds turns out to be three points.

The fast-data problem

Some events require sub-second resolution to understand: trips, surge, fast pressure transients, sequence-of-events analysis on protective functions.

Recording every tag at that rate is impractical. The usual approach is a small set of tags at high rate, or an event-triggered high-resolution capture that stores a window around a defined trigger.

Deciding which tags need this, before the event rather than after, is a specific engineering exercise and one of the few historian decisions that genuinely cannot be corrected retrospectively.

Retention by category

A single retention period applied to everything is either wasteful or insufficient.

Regulatory and environmental data may have a defined statutory retention that must be met and evidenced. Process data used for troubleshooting is most valuable for the recent past and declines in usefulness. Data used for long-term performance analysis needs years but not at full resolution.

A tiered scheme — full resolution for a period, reduced resolution beyond it, indefinite retention for a defined subset — matches the value profile and is supported by most historian products.

The tags nobody can identify

The characteristic condition of a mature historian is a large number of tags whose meaning nobody knows: they were added by a project, named according to a convention that has changed, and describe equipment that may or may not still exist.

These are worse than useless because they appear in searches and dilute results. A periodic review — tags with no queries in two years, tags whose source no longer exists, tags with no description — is unglamorous and improves the usability of everything else.

Metadata is what makes it findable

A tag with a cryptic name and no description is data that exists and cannot be located by anyone who does not already know its name.

The minimum useful metadata is a plain-language description, the engineering unit, the equipment it belongs to, and the process area. With those, a search for the pump discharge pressure on a named unit returns the right tag. Without them, it returns nothing and the user asks a colleague.

Deciding at commissioning, reviewing later

Historian configuration is set during commissioning by people with other priorities and is rarely revisited. Most of the problems described here are the result of defaults applied once.

A review after the first year of operation — which tags are actually queried, which sampling rates proved inadequate, which retention periods are wrong — costs a day and corrects decisions taken when nobody yet knew how the plant would be run.

Who will use it, and how they will ask

Historian configuration decisions are usually made by control engineers, and the eventual users include process engineers, reliability engineers, production planners, environmental staff and, occasionally, external auditors.

Each asks differently. A control engineer asks by tag. A process engineer asks by equipment and by period. A reliability engineer asks for a specific machine across years. An auditor asks for a defined parameter over a statutory period, with evidence of completeness.

Configuring only for the first of those produces a system that serves its builders and frustrates everyone else, which is the common outcome and the reason many historians are used far less than their capability would allow.

The cost that is not storage

With storage inexpensive, the real costs of recording everything are elsewhere. Query performance degrades as the tag count grows. Search results become unusable when they return forty similar tags. Every tag is a thing that must be maintained when the plant changes, and the ones nobody uses are the ones nobody maintains.

There is also a licensing dimension: several historian products are licensed by tag count, which turns an apparently free decision into a recurring cost that is discovered at renewal.

Reviewing what is actually queried

Most historian products can report which tags have been accessed and how often. That report is the empirical basis for a configuration review and it is rarely produced.

The typical finding is extreme concentration — a small percentage of tags account for nearly all queries — which does not necessarily mean the rest should be removed, since some exist precisely for the rare investigation. It does mean that the frequently used tags deserve attention to their sampling, compression and metadata, and that is a much smaller job than reviewing everything.

Governance of the tag list

Historian tag lists grow through projects, each adding what it needs, with no mechanism for consolidation or removal.

Applying the same discipline used for alarms works here: a tag addition is a change, with a stated purpose and an owner, and a periodic review removes tags whose purpose has expired.

The review is low effort where the purpose was recorded at creation and nearly impossible where it was not, which is the argument for capturing it at the point of addition.

Recording the reason a tag exists

A one-line purpose against each tag at the point it is added — what question it answers, who asked for it — costs almost nothing and makes every later review possible.

Without it, a review of ninety thousand tags has no basis for removing any of them, because nobody can establish whether a given tag is still needed.