Service Status

Network Downtime affecting our Montreal and Toronto facilities

CLOSED: Peer1′s Incident Report

At 10:57 UTC, on 10th October 2015, the Peer 1 Hosting’s Technical Operations Centre (TOC) began receiving alerts from our monitoring systems of instability in the Peer 1 Network. The incident was immediately escalated to the Network Operations Engineers (NOE) who began an investigation.

Initial findings pointed to the source of the network disruption being on the West Coast of North America and with San Jose appearing to be the most heavily impacted area.

Further investigations by the NOE team traced the ultimate cause of the problem to be an error in the configuration of a BGP session for a customer in our Vancouver Data Center facility.  This incorrect configuration resulted in large amounts of traffic being routed via our San Jose network nodes to our Vancouver facility.  The root cause of the issues was cleared from the network at 11:35 UTC.

Additional safeguards have been introduced on all customer routing instances to ensure this issue cannot reoccur.

UPDATE October 11 @ 10:15 AM EDT: Our network has been stable since connectivity was restored yesterday morning. Peer1 has confirmed a full RFO will be provided tomorrow. We will post the details here.

UPDATE 10:55 AM EDT: Peer1 confirmed 1 hour ago that all connectivity across their network had been restored:

We would like to update you with the fact that our network engineers have managed to restore network stability across the Peer1 network infrastructure.

Our engineers have traced the problem to the San Jose area and have put some alternate routing in place to normalize the network. Customers in San Jose may see their traffic taking alternate routes at this stage but all customers should have connectivity.

At this stage we believe that all connectivity has been restored. Should you still be seeing connectivity issues with your solution please log a ticket through your normal means so that we can investigate your issue.

We haven’t detected any other issue affecting our cabinets connectivity since our previous update earlier today. Peer1′s

We expect Peer1 to provide more details of the issues that caused this downtime and will post another update later today as soon as we know more.

UPDATE 8:10AM EDT: All our services remain online. We are sorry it’s taking this long to provide details of this incident. We will post an update as soon as we hear back from Peer1.

UPDATE 7:48AM EDT: At this moment our services in Montreal and Toronto are online and we didn’t see any traffic spike in our network. We are sure this issue was out of our network limits and we are still on the phone waiting for an update from Peer1.

UPDATE 7:37AM EDT: Our network is reachable again. We are running diagnostics to make sure everything is working properly. Still no confirmation from Peer1, we are on-hold on the phone.

UPDATE 7:35AM EDT: Peer1 sites are also down at the moment, we suspect this is a major issue affecting their network. We are still on the phone waiting for an update.

We are investigating a network issue affecting both our Montreal and Toronto facilities. We are on the phone with Peer1 at the moment and will post an update shortly.