SCADA Performance Collapse at Shift Change: Diagnosing the Concurrent Load Spike on GE HMI and Honeywell Experion

Alarm Management, DCS, GE Proficy iFIX, HMI Performance, Honeywell Experion PKS, Industrial Automation, Root Cause Analysis, SCADA, Shift Change
เมษายน 13, 2026

The Symptom: SCADA Freezes Only During Shift Handover

Operators report that the SCADA system becomes unresponsive during shift change. Graphics pages load slowly. Alarm acknowledgments take 5–10 seconds. Tag values freeze momentarily on trend displays. The PLC continues running normally. Field instruments show correct values. The freeze resolves itself after 15–20 minutes.

This pattern points to a server-side load spike, not a field communication problem. The GE HMI Proficy iFIX system and Honeywell Experion PKS both experience this issue when multiple events collide at the shift boundary. Understanding each event helps isolate the dominant cause.

Root Cause 1: Concurrent Workstation Restart and Tag Re-subscription

At shift change, the outgoing operator closes the SCADA application. The incoming operator launches it fresh. On GE HMI Proficy iFIX, each workstation restart triggers a full tag subscription download from the SCADA server. If 8 workstations restart within a 2-minute window, the server processes 8 concurrent subscription requests.

Each subscription involves database query, tag lookup, and real-time data routing setup. The GE Proficy Cim-Manager service handles these requests. Under heavy load, the queue depth exceeds the buffer limit. New subscription requests wait in the queue. The delay manifests as slow graphics loading on the client side.

Mitigation: Configure workstations to use application resume instead of full restart. In Proficy iFIX, set the Auto-Resume option in the iFIX workspace properties. This reuses existing tag subscriptions instead of rebuilding them from scratch. The startup time drops from 45 seconds to under 10 seconds.

Root Cause 2: Alarm Acknowledgment Burst and Database Write Storm

During handover, the outgoing operator acknowledges accumulated alarms. A typical shift generates 200–400 alarm events. If 80% remain unacknowledged, the incoming operator confirms them in a batch. Each acknowledgment generates a database transaction recording the operator name, timestamp, alarm ID, and comment.

In Honeywell Experion PKS, alarm events write to the Alarm Database (ADB). Each write requires an index update. Under burst load, the index update queue grows faster than the disk I/O can process. The Experion Alarm Server responds with increased latency. Alarm pop-ups appear with a 3–5 second delay. Our Honeywell controller module cards and C300 system components are designed to maintain stability under these load conditions.

On GE Proficy iFIX, alarm acknowledgments write to the historical database via the SCU (System Configuration Utility). If the historian is on the same server, disk I/O contention worsens. The Proficy Process Historian may drop data during this window.

Mitigation: Enable alarm shelving for low-priority alarms before shift end. In Experion PKS, configure the alarm shelving feature in the Alarm Configuration tab. Shelf up to 50% of unacknowledged low-priority alarms. This reduces the acknowledgment burst by eliminating non-critical events.

Root Cause 3: Automatic Shift Report Generation and Query Overload

Both GE and Honeywell SCADA platforms support automatic report generation at scheduled times. Common shift reports include production totals, downtime summaries, alarm statistics, and energy consumption. These reports query the historical database for thousands of data points.

If the report generation engine runs on the SCADA server, it competes for CPU and disk resources with real-time operations. A typical shift report query on GE Proficy Process Historian takes 8–12 seconds to execute. During that window, the SCADA server has reduced capacity for tag updates and alarm processing.

In Honeywell Experion PKS, the Reports Workstation generates reports via the Experion Reports Server. If the Reports Workstation shares the same server as the Experion Application Server, the query load degrades real-time performance. The Server Health Monitor shows CPU usage spiking to 85–95% during report generation.

Mitigation: Schedule shift reports to run 10 minutes after shift change, not at the exact handover time. This decouples the report query from the alarm acknowledgment burst. Verify the schedule in the Experion Report Designer or the Proficy Process Historian report scheduler.

Step-by-Step Diagnostic Procedure

When the freeze occurs, collect diagnostic data immediately. Do not wait until the next shift.

Step 1: Open Windows Task Manager on the SCADA server. Record CPU usage, memory usage, and disk queue length. Identify whether the bottleneck is CPU, RAM, or disk I/O.
Step 2: Check the Windows Event Viewer for database write failures or service timeouts. Look for Event ID 833 (database latency exceeded) in the Experion logs or Event ID 4096 in Proficy iFIX logs.
Step 3: Review the SCADA tag subscription count. In Experion PKS, use the Station Manager to display active subscription statistics. In GE Proficy iFIX, use the System Diagnostics screen to check the Cim-Manager queue depth.
Step 4: Examine the alarm log for the burst pattern. Count the number of alarm acknowledgments within a 5-minute window around shift change. If the count exceeds 50 per minute, the database write storm is the dominant cause.
Step 5: Check the report generation schedule. Look for reports configured to start at shift change time. Shift the start time by 10–15 minutes to reduce overlap.

Conclusion and Action Advice

SCADA performance collapse at shift change stems from three concurrent events. First, workstations restart and re-subscribe to all tags simultaneously. Second, operators batch-acknowledge accumulated alarms, generating a database write storm. Third, automatic shift reports query the historical database at peak load time.

On GE HMI Proficy iFIX, enable Auto-Resume to eliminate full subscription rebuilds. On Honeywell Experion PKS, enable alarm shelving and reschedule reports 10 minutes after shift change. Both platforms benefit from moving the historical database to a dedicated server. This separates real-time tag processing from report query workloads. Reliable Honeywell I/O modules ensure field data integrity is maintained even during server-side load spikes.

Document the diagnostic findings in the plant's CMMS. Create a shift change performance procedure that distributes the three load events across a 30-minute window. Monitor the results for two weeks. If CPU usage still exceeds 80% during shift change, consider upgrading the SCADA server or adding a second server for load distribution.