This circuit has been stable for over 12 hours and Zayo reports that no further issues are expected. We just re-enabled our BGP session on it and everything looks good. We will continue to monitor as usual and disable it if any further issues occur.
Posted Sep 02, 2016 - 21:59 PDT
Update
We are still monitoring this circuit and have not re-enabled it in our network since it's not yet stable. Customer traffic remains unaffected, routing over our redundant paths. Zayo has informed us that this event is not a reoccurrence of the recent DDOS related issues, but instead a router hardware issue. We will keep the circuit disabled until it's stable.
In case you're curious about how we're monitoring this, here are some detail: We monitor every one of our upstream transit circuits and core network devices with a custom globally-distributed monitoring system hosted across multiple cloud providers. This system alerts our team to issues about global internet routing to/from our network. This provides an added layer of awareness so we can make any needed traffic-engineering decisions, on top of our SD-WAN software that automatically re-routes customer traffic around most internet issues.
Posted Sep 02, 2016 - 09:20 PDT
Monitoring
We have identified and mitigated packet loss on one of our upstream internet transit circuits (Zayo) in our LA datacenter. We have disabled this circuit and it will remain disabled until the circuit is stable.