Service Status

NAS Outage in our Montreal facility

CLOSED 9:55PM EST: We are happy to confirm that we have been able to restore the data of all the affected customers. Our team was able to void data loss during this hardware failure. We have discontinued our NAS server and have setup temporary storage for backups and for customers using block storage connected to their servers. We will work hard to setup a much better storage service for our customers and hope to have it available during the third quarter this year.

We are grateful for the patience and trust all affected customers showed us today. We are pretty tired but happy with the results. Rest assured we will continue working hard to offer you the best managed hosting service!

UPDATE 6:31PM EST: We have been able to restore the raid array and so far no data has been lost!!! We are going to start moving all data to new servers, starting with customers that store application data and following up with backups. Every affected customer will be contacted to explain the changes to their files ando/or backups storage. We will post another update later today. Thanks for your patience!

UPDATE 5:00PM EST: Still working on this issue. We are going slow to avoid data loss but the risk is still present. We will post another update in about 2 hours.

UPDATE 2:30PM EST: Our team continues working to restore access to the data stored in our NAS server in Montreal. Data loss seems like a real possibility at this moment but we are doing our very best to avoid it.

UPDATE 11:55AM EST: Our engineers continue working to restore the corrupted RAID5 array and bring all the data back online. Feel free to contact us if you have any question.

UPDATE 10:00AM EST: Unfortunately the RAID array is corrupt and we are working to properly restore it without any data lost. We are truly sorry for the inconveniences caused to some of our customers during this downtime. We will continue posting periodic updates while we work to fix this ASAP.

UPDATE 9:30AM EST: fsck is still running but we don’t think it will take more than 30 minutes to have everything back up. We will post another update in 30 minutes. Thanks for your patience.

UPDATE 8:55AM EST: The drives were successfully moved to a new server. The Array is reported as Optimal and the server reports all the logical volumes. A fsck is running as part of the server start up and should take a few more minutes to finish. As soon as this is completed we expect the server to be back up and will check every affected customer to make sure everything is working properly. We will post another update in about 30 minutes.

UPDATE 8:30AM EST: Engineers on site tried replacing the damanaged RAID card but that didn’t work. We are replacing the whole server now, this will take a bit longer but we expect to have the server back up within 1 hour.

UPDATE 7:30AM EST: So far we have been unable to restore access to our NAS but we have confirmed a RAID card failure is the cause of this downtime. We have spares on site and datacenter engineers will replace the damaged card within 1 hour. We will continue posting updates here.

6:30AM EST: Our NAS server in Montreal has been unavailable since 6:20AM EST due to apparently a hardware issue. Our team is working hard to restore access to the server and restore the storage of affected customers.

Thanks,
Speedyrails Team