Service Status

Power outage in our Canix Datacenter

CLOSED 5:56PM EST (Dec 8th): We continued today verifying that everything was working properly in the servers that were affected by the power outage yesterday and worked with some customers to resolve specific issues. At this moment all our services are back up in our Canix facility.

The power outage was the result of a UPS failure which caused power to be lost to certain circuits in the room where our cabinets are located. Though the outage was very brief, the short unintended interruption restarted most of our servers in this facility and caused hardware issues on some of them.

This is the third power outage we have had in this facility this year. We are going to work even harder to make sure all our customers are successfully moved to our Peer1 facility by the end of this month. Customers currently using our Canix facility will receive an email from us during the next 5 days to schedule the migration of their services to the Peer1 facility. Customers wanting to speed up the migration of their services can open a ticket in our support system and we will work immediately to migrate their services. 

We have no words to apologize enough for these issues. Please contact us if you have any question or want to know more details regarding this power outage or our facilities.

Thanks,

Speedyrails Team

UPDATE 04:00AM EST (Dec 8th): We have restored the shared hosting accounts that were still down. At this moment all our services are back up. Our team is exhausted but we couldn’t stop until getting every single site back up. We will take a quick nap and will be back at work to double check all the details in the Canix facility. If you are still experience any kind of issue please open an Urgent ticket in our support system and we will take a look at your services immediately. We will be posting more updates in the morning. Thank you all for your patience and trust in our service.

UPDATE 00:27AM EST (Dec 8th): Good news! At this time all the svn repositories hosted at svn.speedyrails.ca and most of the shared accounts are back up. We are working on a small number of shared accounts that are still having problems and will publish another update as soon as those accounts are up.

UPDATE 10:37PM EST: We have been able to restore the RAID arrays of the damage server and we are very optimistic at this point. We expect the affected services (shared databases and svn repositories) to be back up within 1 hour.

UPDATE 9:10PM EST: We are still working hard to restore the failed shared DB server. Please note that shared hosting customers still hosting the repositories at svn.speedyrails.ca are also affected at this time. We expect to have an update regarding the shared DB server within 2 hours.

UPDATE 4:46PM EST: Getting the shared db server back up is taking longer than planned. We have backups less than 12 hours old for each shared database. If you have a shared hosting account and you can live with a database restored from a backup done earlier today please let us know and we will get your site back up. Once we restore access to the shared database server we will give you access to the original database so you can manually synchronize it with the new one (we can help with that too!)

UPDATE 4:29PM EST: All our VPS customers are back up but we are still verifying that every single application is running properly. It will take a few hours until we can complete a partially manually scan of all the applications to make sure there are no issues. If your application is having any issue please open a ticket and we will check right away. 

Shared customers are still experiencing issues due a failure in one of the shared database servers. Our team is working to get everyone back up as soon as possible.

UPDATE 4:03PM EST: Most of our VPS customers are back up. Shared hosting customers are still experiencing some issues due a shared database server that is having problems after the outage, we are working to get this server back up as soon as possible.

UPDATE 3:27PM EST: All physical servers are back up and we are starting up the affected customers as fast as possible. If you opened a ticket and your VPS is back up and working properly please let us know. If your VPS is still down rest assured we are doing our best to get it back up as soon as possible.

We just experienced a partial power outage in our Canix datacenter. We are starting the affected VPS as fast as humanly possible while also investigating what happened.