The Deepwater Horizon disaster was caused in part by a disabled alarm system. The alarm system was disabled for the wrong, yet all-too-common, reason. It was disabled because false alarms are an annoying disturbance. The technician in charge of maintaining the system, Mike Williams, testified “he had asked before about the settings and was told the company didn’t want a false alarm waking people at night.”

The lesson to be learned here is that there is only an “A” and “B” option. There is no “C” option. Here is a list of the only two options that should be considered when fine-tuning an alarm (or any monitoring) system:

A) It’s a real problem, so fix the problem.
B) It’s not a real problem, so fix the alarm.

Yes, it’s a challenge to maintain the right balance. If the threshold you set for an alarm is not optimized, you will get too many alerts. And even when you have the alarm threshold set appropriately, sometimes you will get a false alarm. The idea is that you want to be warned before you have a problem. In order to be warned before you have a problem, the thresholds should be set to somewhere before where real problems occur, but not too much!  This means that you will have some “false alarms”, but there shouldn’t be too many, otherwise they will start to be ignored. You should strive for the right balance. Ninety percent (90%) of your alarm notifications should be actionable, where “actionable” can include “investigating significant threats of possible problems”. If more than 10% of your alarms can be reasonably ignored, then you have a problem and it is time to do an alarm / monitoring audit.

Regular monitoring audits are part of a good monitoring process. It’s often the last thing that system architects and designers think of, since they don’t actually operate the real-world systems, so it’s up to operations staff usually to establish good monitoring practices as part of good operating practices.

If you have problems with your monitoring/alarm systems, don’t shut them off. To get expert help with establishing good monitoring practices, please contact Rex Consulting today for an analysis.

© Copyright 2020 Rex Consulting, Inc. – All rights reserved