What Happened?
On Monday, March 31 at 2:17 PM EST our upstream routers were unable to effectively route traffic to the Internet. The issue started when peer routers flooded unproductive routes, which affected our upstream route tables.
How was it Fixed?
The immediate fix re-converged routes at 3:12 PM EST. The immediate fix was determined and implementation was started in less than 30 minutes. A full diagnose and confirmation of the issue was obtained from the logs. A more permanent resolution was then designed and carefully tested.
How was the Problem Resolved for the Future?
Our network team has applied more stringent access lists in those peers. Also, our stricter configuration will terminate a BGP peer if they show a sudden, unexpected increase in routes, further protecting our customers from this type of occurrence in the future.
We apologize for the inconvenience and trouble this disruption caused to our clients. We sincerely thank you for your patience and understanding as we worked through the issue.
