A problem with a routine update to our alerting system caused that system to process some out-of-date site and circuit metrics. The result caused a handful of messages to be sent for site/circuit problems in the past. The resolution to this problem unfortunately required we restart the process in a way that required sending duplicate notifications for all ongoing site/circuit issues.
We apologize for any inconvenience or confusion this issue may have caused. At this point all systems are operating normally, though some notification emails may still be in transit and may continue to arrive after this message. We have already identified the root cause of the problem and are working on the correct means to address it to prevent this problem from reoccurring.
Of note, the alerting system is completely separate of our network and all systems processing customer traffic. At no point was customer traffic affected by the alerting issues.
Posted Jan 06, 2017 - 17:06 PST
Investigating
We are investigating an issue with a alerting system that may be causing delays in notifications and possibly the resending of old alert notifications. We will update as more information is known.