Industrial System Malfunction Diagnosis: A Practitioner’s Guide to Fault Detection, Troubleshooting, and Root Cause Analysis

The Detection → Diagnosis → RCA Continuum

Fault detection and diagnosis (FDD) is a systematic process for identifying, isolating, and characterising malfunctions in industrial systems. But detection and diagnosis serve different purposes.

Fault detection answers: Is something abnormal happening? Your alarm fires. A trend deviates by more than two standard deviations. A control loop starts hunting (oscillating around its setpoint). Detection recognises that a problem exists, ideally within 30 seconds to 5 minutes, quickly enough to prevent escalation.

Fault diagnosis goes further: what specific fault is occurring, and where? The pump isn’t just “failing.” The pump is cavitating because NPSH (Net Positive Suction Head, the pressure available at the pump inlet) dropped 8 feet below requirements. The heat exchanger isn’t “underperforming.” Tube-side fouling reduced the heat transfer coefficient by 30%. Diagnosis isolates the specific fault for targeted action.

Root cause analysis asks the deeper question: why did this fault occur? RCA techniques such as the 5 Whys and fishbone diagrams (also called Ishikawa diagrams, developed by Kaoru Ishikawa at Kawasaki shipyards in 1968) trace causal chains back to their origins. Without proper RCA, you keep replacing failed bearings every 4 months without asking why. And they’ll fail again at the same interval.

Here’s the honest take from industry surveys of 200+ process facilities: roughly 85% are decent at detection (alarms exist), approximately 60% are okay at diagnosis (experienced operators figure things out within a shift), and only about 25% conduct effective root cause analysis. That’s why the same equipment keeps breaking, and the same money keeps disappearing.

Abnormal Event Management (AEM) encompasses timely detection, diagnosis, and correction while plants remain in controllable operating regions. The petrochemical industry has rated AEM as its number one problem since the 1990s. Thirty years later, the billions in annual losses prove the industry hasn’t solved it.

What’s the difference between fault detection and fault diagnosis? Fault detection identifies that an abnormal condition exists (something is wrong), while fault diagnosis determines what specific fault is occurring and where (what’s wrong and where). Detection happens in seconds to minutes through automated monitoring. Diagnosis requires 15 minutes to 4 hours of investigation using process knowledge and systematic analysis.

Traditional and Advanced Detection Approaches

Threshold-based monitoring is the simplest approach: set limits and trigger alarms when they are exceeded. Temperature exceeds 350°F? Alarm. The problem: threshold alarms detect faults only after significant equipment degradation has occurred. By the time your bearing temperature alarm fires at 180°F, the bearing may have only days until catastrophic failure.

Statistical process control (SPC) detects deviations from normal patterns, not just absolute limits. A temperature trending upward at 0.5°F per hour, even 40°F below alarm limits, indicates developing problems. SPC catches faults weeks earlier than thresholds but requires 30 to 90 days of historical data to define “normal.” Implementation typically runs $15,000 to $75,000, using software such as Honeywell Uniformance or AspenTech Aspen ProMV, though pricing varies by scope and vendor.

Condition monitoring uses specialised measurements like vibration, thermal imaging, and oil analysis to assess equipment health directly. A vibration signature change of 0.1 inches/second can indicate bearing wear 30 to 90 days before temperature rises—performing a root cause analysis of equipment vibration helps identify whether the issue stems from imbalance, misalignment, or bearing defects. Basic vibration setup typically costs $5,000 to $15,000 per machine, using SKF Microlog or Emerson CSI 2140 analysers.

How much does condition monitoring cost? A basic vibration monitoring setup typically runs $5,000 to $15,000 per machine, though costs vary by equipment complexity and vendor. Comprehensive thermal imaging programs often cost $20,000 to $50,000 annually, including equipment and trained thermographer time. ROI often exceeds 200% to 300% in well-implemented programs due to avoided failures. Verify current pricing with vendors for your specific application.

Machine learning takes detection further. Neural networks and support vector machines identify complex fault patterns that statistical methods miss. The Tennessee Eastman Process, developed by Eastman Chemical Company in 1993, serves as the benchmark for validating fault detection methods. If evaluating ML tools, ask whether they’ve been validated against Tennessee Eastman, achieving strong detection rates with low false alarms. Vendors who can’t demonstrate this validation haven’t proven their algorithms work in realistic conditions.

Quick sidebar: the hype around AI-based fault detection often exceeds reality. Yes, machine learning can achieve high accuracy in controlled conditions with clean data. But real plants have sensor drift, significant data gaps, and operating modes not in training data. AI augments human expertise. It doesn’t replace the process knowledge your experienced operators carry.

Selecting the Right Detection Method

Most facilities need layered detection: threshold alarms as a last defence, statistical monitoring for early warning, and predictive analytics (when data quality supports it) for advance notice of developing problems.

Why model-based detection works: these methods compare real-time measurements against physics-based predictions. Deviations indicate faulty sensors or developing problems. You can detect faults affecting unmeasured variables by observing their impact on measured ones. Skip model validation? Your system generates excessive false alarms, and operators ignore everything.

AVEVA delivers industrial software, including Asset Information Management and Predictive Analytics, that identify anomalies before failure. Modern platforms can forecast time-to-failure with reasonable accuracy when properly implemented. But these tools only work if your data infrastructure captures sensor readings consistently with proper timestamps.

The 7-Step Troubleshooting Model

1. Define the problem. “The pump is broken” isn’t a definition. “Pump P-101 discharge pressure dropped 15 psi over 4 hours while suction remained at 25 psig and motor amps stayed at 45A.” Takes 5 to 15 minutes. Skip it? You’ll chase the wrong cause for hours.
2. Gather data. Current readings, 24-hour and 7-day trends, 12-month maintenance history, operator observations, and changes in the last 30 days. Verify in the field when possible. Takes 20 to 60 minutes.
3. Analyse data. Generate 3 to 5 hypotheses about potential causes. Takes 15 to 45 minutes.
4. Assess sufficiency. Have enough information to propose a solution? If not, return to step 2. Takes 5 minutes but saves hours.
5. Propose a solution. What’s the most likely cause and fix? Takes 10 to 30 minutes.
6. Test the solution. Implement and verify. Takes 15 minutes to 8 hours, depending on process impact.
7. Document findings. Record in your CMMS (Computerised Maintenance Management System). Takes 30 to 60 minutes. Skip documentation? The next troubleshooter starts from scratch in 6 months.

The 2018 Husky Energy explosion and fire at the Superior, Wisconsin, refinery, which injured 36 workers, demonstrates how diagnostic failures during abnormal operations cascade into disaster. The U.S. Chemical Safety Board (CSB) investigation found that a spent alkylation unit was brought online without adequate diagnostic verification of equipment condition. Canadian operators should note that similar incidents have occurred domestically. The Transportation Safety Board of Canada’s (TSB) investigation into the 2013 Lac-Mégantic rail disaster revealed systemic failures in monitoring, inspection, and safety culture that parallel industrial diagnostic breakdowns.

For process industry incidents specifically, Energy Safety Canada and the Canadian Association of Petroleum Producers (CAPP) publish safety alerts and lessons learned that inform diagnostic best practices across Western Canada’s energy sector. The Canadian Centre for Occupational Health and Safety (CCOHS) provides additional guidance on incident investigation and root cause analysis methodologies.

Field Verification and Cognitive Traps

Here’s something automation enthusiasts hate to hear: your senses are diagnostic tools. Look for discolouration and leakage. Listen. That 3 kHz whine means bearing problems. The 500 Hz rumble indicates cavitation. Feel for vibration above 0.3 inches/second or temperatures above 150°F. Experienced operators detect problems that sensors miss entirely.

Instrument readings require verification—a core competency in instrumentation and controls engineering. A level transmitter might correctly measure differential pressure while the actual level exceeds the transmitter’s upper tap, meaning DP no longer reflects true level. Sight glasses and alternative measurement methods provide critical verification. Trust your instruments, but verify critical readings through alternative means.

Confirmation bias kills troubleshooting. You think you know what’s wrong, so you only see supporting evidence. Spend 5 minutes actively seeking evidence that would disprove your theory. Research by Kahneman and Tversky found that this approach significantly improves diagnostic accuracy.

Anchoring distorts analysis. The supervisor says, “probably the seal,” based on 30 seconds of thought, and suddenly everyone’s looking at seals even when symptoms don’t fit. Document your own observations before asking opinions.

Canadian Regulatory Framework for Process Safety

CSA Z767 Process Safety Management is Canada’s national standard for preventing major accidents in process industries. Published by the Canadian Standards Association (CSA Group), Z767 establishes requirements for hazard identification, risk assessment, and management of change that directly impact diagnostic capabilities. Alberta’s Occupational Health and Safety (OHS) legislation references CSA standards and requires employers to ensure equipment is properly maintained and monitored.

The Alberta Energy Regulator (AER) oversees upstream oil and gas operations in Alberta and enforces Directive 071: Emergency Preparedness and Response Requirements, which includes provisions for equipment monitoring and failure prevention. In Ontario, the Technical Standards and Safety Authority (TSSA) regulates operating engineers and facilities under the Operating Engineers Regulation.

For comparison, the U.S. framework under OSHA’s 29 CFR 1910.119 (Process Safety Management) establishes similar requirements, and many Canadian facilities operating across the border maintain compliance with both frameworks. Regulations change frequently, so verify current requirements with qualified professionals and your provincial regulatory authority.

Critical Diagnostic Checkpoints

Before startup (2 to 8 hours before introducing the process):

• Verify critical instruments are functional and calibrated within 12 months; having the right industrial control system troubleshooting tools on hand streamlines this verification process.
• Confirm that level indicators and pressure transmitters read correctly against field checks. Takes 30 to 90 minutes per major vessel.
• Ensure alarms are set for startup conditions (wider limits during filling)
• Review shutdown maintenance that might affect diagnostics

During startup (continuous for 4 to 24 hours):

• Apply first principles continuously. If levels rise faster than expected, investigate within minutes, not after overfill.
• Verify readings through redundant transmitters, sight glasses, and flow calculations.
• Staff critical startups with additional operators beyond normal requirements

Why startup discipline matters: Abnormal conditions progress faster during transients than during steady-state. An unexpected rise in level gives you limited time to respond. Rush the investigation, and you’re making evacuation decisions rather than troubleshooting ones.

Choosing the Right RCA Technique

The 5 Whys is simplest: ask “why” repeatedly until you reach the fundamental cause. Takes 30 minutes to 2 hours.

The bearing failed. Why? Overheated to 250°F. Why? Inadequate lubrication. Why? The PM schedule wasn’t followed. Why? Staffing shortage caused the technician to be pulled for higher-priority work. Why? Budget cuts reduced headcount without adjusting workload.

The 5 Whys works for single-cause failures but may miss multiple contributing factors present in many equipment failures.

Fishbone diagrams organise causes into six categories: Equipment, Procedures, Personnel, Materials, Environment, and Management. Takes 2 to 4 hours with a cross-functional team. This is where multi-discipline engineering perspectives prove invaluable, as failures rarely confine themselves to a single domain. Generates numerous potential causes for investigation.

Fault Tree Analysis works backwards through logical gates (AND/OR conditions) to identify failure combinations. Takes 4 to 16 hours. Reserved for incidents with significant consequences or safety implications.

TapRooT and other structured methodologies provide systematic approaches to RCA that many Canadian energy companies have adopted. The Canadian Centre for Occupational Health and Safety (CCOHS) publishes guidance on incident investigation techniques suitable for various industry sectors.

How long does root cause analysis take? Simple 5 Whys: 30 minutes to 2 hours. Fishbone sessions: 2 to 4 hours with a cross-functional team. Formal Fault Tree Analysis: 4 to 16 hours. Complex incidents with multiple factors: 20 to 40 hours across several days.

For most failures, combine approaches: use a fishbone to generate hypotheses, use 5 Whys to drill into likely causes, and document in your CMMS to capture learnings.

Moving Beyond “Operator Error”

Here’s the unpopular opinion: “operator error” is almost never the root cause. It’s a symptom of system failures. These include inadequate training (4 hours instead of the required 16), confusing procedures (37 pages with 12 conflicting sections), poor interface design (critical alarm buried among dozens of others), and fatigue from excessive overtime.

The Transportation Safety Board of Canada’s investigations consistently identify organisational and systemic factors underlying incidents initially attributed to human error. The TSB’s Watchlist highlights recurring safety issues, including inadequate safety management systems. Facilities that stop at “operator error” experience significantly higher repeat incident rates than those that investigate system causes.

When RCA points to operator error, keep asking why. What system condition allowed or encouraged that error? What would have caught it before the consequences? Spend the extra hours on system factors. Or repeat this same RCA in months.

Building Your Data Infrastructure

Every advanced capability depends on data quality. Without data collected consistently, stored accessibly, and contextualised meaningfully, predictive analytics produces noise instead of insights.

Asset Information Management (AIM) creates the foundation: a single source of truthSingle source of truth (SSOT) refers to the practice of structuring information models and associated data schema such that every data ele... consolidating engineering documents, maintenance records, and sensor data. When troubleshooting pump P-101, you need immediate access to the P&ID, datasheet, maintenance history, and real-time data. Most facilities have this scattered across many disconnected systems. Finding everything takes hours instead of minutes.

Vista Projects, an integrated engineeringThe process of integrated engineering involves multiple engineering disciplines working in conjunction with other project disciplines to e... firm headquartered in Calgary serving Western Canada’s energy sector and international markets, has delivered multi-discipline services since 1985. Our AVEVA partnership provides asset information management, consolidating diagnostic information into accessible systems. Implementation costs vary significantly by facility size and complexity, typically ranging from $200,000 to over $1 million with 12 to 24 month timelines. ROI often pays back in 18 to 36 months through faster troubleshooting, though results vary by implementation quality.

Case studies from organisations across North America demonstrate significant savings through AI-enhanced analytics when properly implemented over 18 to 30 months with strong change management.

The honest take: AI-powered diagnostics deliver value for organisations ready to use them. Many organisations aren’t ready yet. Their historian has significant gaps. Their CMMS data quality is poor. Start with reliable collection and information management. Add predictive analytics when foundations are solid, typically year 2 or 3 of a transformation program.