Factory acceptance testing that finds problems

Factory acceptance testing is the last point at which a control system problem can be fixed cheaply, in the integrator's workshop, with the people who built it present and the schedule not yet critical.

It is frequently conducted as a demonstration: the integrator drives, shows the system working, and the client's representatives observe and sign. That version reliably finds nothing, because a system being demonstrated by the person who built it will do what it is asked.

The client should drive

The single change that most improves a FAT is that the client's engineers operate the system rather than watching it operated.

The timing and pace of communication influence how handovers, escalations and instructions are understood. Further details are available on this page.

For process facilities, the surrounding management framework is covered in OSHA process safety guidance.

Someone unfamiliar with the configuration finds the things a familiar person navigates around automatically: the display that does not exist, the sequence that requires an undocumented step, the alarm that does not annunciate because the tag was misspelled.

Write the test cases in advance

A FAT protocol should exist before the test, be reviewed by both parties, and specify what is being tested, how, and what constitutes a pass.

Test cases written during the FAT tend to follow what the system does rather than what it should do. Written in advance from the functional specification, they test the specification.

Test the abnormal path

Normal operation is the easy case and is what a demonstration covers. The value is in the rest.

• Loss of communication between components, and recovery.
• Redundancy changeover, initiated deliberately, with the system loaded.
• Controller failure and restart, including what happens to outputs during the transition.
• Bad quality inputs, and how calculations and displays respond.
• Invalid operator entries: out-of-range setpoints, contradictory commands.
• Sequences interrupted midway, and restarted.
• Power loss and restoration, on each component in turn.

The redundancy test is the one most often skipped and the one most often found to be defective when it is performed. Redundancy that has never been tested under load is an assumption.

Simulation quality determines what can be tested

A FAT tests the control system against simulated process behaviour. The quality of that simulation determines what can meaningfully be tested.

Static simulation — inputs set by hand — verifies configuration, displays and alarms. Dynamic simulation, where the simulated process responds to control action, allows control strategies and sequences to be tested properly.

Dynamic simulation costs more and is worth it where sequences are complex. Deciding this early matters, because retrofitting a dynamic model during the FAT is not possible.

Record everything, including the passes

The test record should show each case, the observed result and the outcome. Passes matter as much as failures, because the record is the evidence that a given behaviour was verified.

Failures become punch list items with an owner and a retest requirement. The FAT is not complete when the tests have been run; it is complete when the failures have been corrected and retested.

Resist compressing it

The FAT sits at the point where a project first discovers it is late, and it is the obvious thing to shorten because it appears to be a formality.

Every hour saved there is spent at site with a multiplier, because the same fault found during commissioning involves travel, plant access, permits, and a schedule where the critical path runs through it.

Bring the operators

A FAT attended only by engineers verifies technical function. Bringing one or two experienced operators finds usability problems while changes are still cheap.

Their contribution is specific: the display that does not show what they need, the operation that takes six actions, the alarm description they cannot interpret. Every one of those found at FAT is one not found in the first month of operation.

What to test that the specification does not cover

A FAT protocol derived from the functional specification tests what was specified. Several things worth testing are not usually specified at all.

Response of the system under load: how the displays and alarms behave when a large number of points change simultaneously, which is the flood condition the plant will produce.

Behaviour at limits: maximum tag counts, maximum trend pens, maximum simultaneous users.

Recovery time: how long a controller takes to return to service after a restart, and what the plant would be doing during that interval.

Bringing the site's own documentation

A FAT is also a test of the documentation, and the way to test it is to use it. Operating the system from the manuals and drawings supplied, rather than from the integrator's knowledge, finds the gaps.

Documentation deficiencies found at FAT are corrected as part of the project. The same deficiencies found after handover are the site's problem.

The punch list starts here

Items found during FAT should enter the same punch list that will run through commissioning, with the same categorisation and closure discipline.

Keeping a separate FAT list that is closed at the end of the test loses the items that were deferred to site, which are precisely the ones most likely to be forgotten.

Testing the alarm configuration

A FAT is the natural point to verify that alarms are configured as the rationalisation specifies: correct thresholds, correct priorities, correct descriptions, correct suppression logic.

Doing it here rather than at site is substantially cheaper, and it catches the systematic errors — a priority mapping applied wrongly across a whole area, descriptions truncated by a field length limit — while they are one correction rather than four hundred.

It also exercises the alarm rationalisation database as a specification, which frequently reveals gaps in it.

Who attends and for how long

FAT attendance is a cost and is frequently minimised, with one client engineer sent for a few days of a two-week test.

The people whose attendance produces most value are the engineer who will support the system afterwards and an experienced operator. Both find things nobody else will, and both gain familiarity that reduces the support burden later.

Treating FAT attendance as training as well as testing changes the economics of sending them.