Timestamps: when two systems disagree about what happened first
Sequence-of-events analysis depends entirely on clocks agreeing. On most sites they do not, and nobody knows by how much.
Determining what happened first is the central question in most incident investigations. Did the pressure rise before the valve moved, or after? Did the trip precede the alarm or follow it?
Answering it requires that the systems recording those events agree about the time. On a typical site, the control system, the safety system, the historian, the alarm server, the operator station and the various intelligent field devices each maintain a clock, and the differences between them are frequently larger than the events being analysed.
Where the time comes from
A coherent architecture has one authoritative time source and everything synchronises to it, directly or through a hierarchy.
Payroll rounding rules are jurisdiction-specific and should be reviewed separately from operational time records. Further details are available at this link.
A useful external reference for timing and synchronisation is NIST time and frequency services.
In practice sites accumulate several: a GPS receiver feeding one system, a corporate network time server feeding another, and a handful of devices with clocks set by hand at commissioning and drifting since.
The first useful exercise is simply to establish what the current architecture is, which frequently reveals devices that are synchronised to nothing at all.
Multiple independent time sources produce systems that are each internally consistent and mutually contradictory, which is worse than being obviously wrong.
Accuracy needed depends on the question
For trend analysis over hours, a second or two of disagreement is irrelevant.
For determining the order of events during a trip sequence, milliseconds matter, and this is the case where dedicated sequence-of-events recording with high-precision time stamping is used.
For alarm analysis, sub-second precision is rarely needed but consistency is: comparing alarm times against process data requires both to be on the same clock.
Timestamping at source
Where a value is timestamped determines what the timestamp means. A value timestamped at the field device records when the measurement was taken. A value timestamped at the historian records when it arrived, which includes network and polling delay.
The difference is normally small and becomes significant during the fast events where the analysis matters most, and during communication disruption where a backlog of values may arrive together and receive nearly identical arrival timestamps for measurements taken seconds apart.
Source timestamping should be used where the devices support it, and where they do not, the polling interval sets a floor on the meaningful resolution.
Time zones and clock changes
The recurring practical failure is daylight saving. A system recording in local time has an hour that occurs twice and an hour that does not exist, once each per year.
Analysis crossing those boundaries produces events out of order, gaps and duplicates. Investigations conducted months later, by someone who did not know the system runs in local time, reach wrong conclusions.
Storing in a fixed reference such as UTC and converting only for display avoids the problem entirely. Retrofitting that to a system that has recorded in local time for a decade is difficult, which is an argument for getting it right at commissioning.
Checking the current state
A practical audit takes an hour. Trigger a single identifiable event and record how each system timestamps it. Compare the clocks on every relevant device directly. Look for events in the historian that appear out of causal order.
The typical finding is that most systems agree within a second, one or two are minutes out, and at least one device has a clock that was set at installation and has drifted substantially.
Monitoring the synchronisation
Time synchronisation fails silently. A device that loses contact with its time source continues to run on its internal clock and drifts without any indication.
Where the infrastructure supports it, alarming on synchronisation loss or on clock offset exceeding a threshold turns a silent failure into a visible one. This is one of the few genuinely useful diagnostic alarms and is rarely configured.
Building a time architecture
A coherent arrangement has one primary reference — commonly a GPS-disciplined clock on the control network — with a small number of servers synchronising to it and everything else synchronising to those.
The design questions are which device is authoritative, what happens if it fails, whether the control network takes time from the corporate network or the reverse, and how devices that cannot use the standard protocol are handled.
That last category — older devices with proprietary synchronisation or none at all — is usually where the drift lives, and identifying them is the practical first step.
Sequence of events recording
Where the order of fast events matters — determining the first-out cause of a trip — normal historian resolution is inadequate and dedicated sequence-of-events capability is used, with time stamping at the input module to millisecond precision.
This requires the modules involved to share a high-accuracy time reference, which is a specific architectural requirement rather than something ordinary synchronisation provides.
Deciding which inputs need it is an engineering judgement made during design. Retrofitting it after an incident where the sequence could not be determined is possible and considerably more expensive.
Verifying after changes
Time synchronisation is disturbed by system changes: a replaced server, a network reconfiguration, a firewall rule that blocks the synchronisation protocol without anyone noticing.
Including a clock comparison in the post-change checks for any network or server work is a small addition that catches a failure which otherwise remains hidden until an investigation needs the timestamps.
Time in reports and exports
Time zone and format problems propagate beyond the historian into every report and export built on it.
A report showing timestamps without a zone, or exported to a spreadsheet where the format is reinterpreted, produces figures that cannot be reconciled with the source. This is a common cause of apparent discrepancies between systems that are actually agreeing.
Stating the zone explicitly on every report, and using an unambiguous format, removes an entire class of confusion for very little effort.
Establishing a baseline
Before any improvement, record the current offset of every system against the intended reference. That table is the baseline, and it is what allows the improvement to be demonstrated and any regression to be detected.
Repeating the measurement annually, and after any significant infrastructure change, keeps the picture current for very little effort.