[Problem] Performance issue due to SAN storage I/O
Incident Report for CloudNordic
Resolved
We are pleased to announce that the replacement SAN storage node was implemented at 08:30 CET, by the SAN storage technician, and all systems are again fully operational.

During the incident no CloudNordic services were implemented except for a small performance degrade to our Shared webhosting services, as no Exchange (including IMAP/POP) or dedicated servers were impacted.

The only impact was to our own website, which was unfunctional for about 30-40 minutes.

Datacenter: Skanderborg, Jutland

Start: 16-05-2018 @ 12:30 CET
End (expected): 17-05-2018 @ 09:30

Best Regards, CloudNordic Helpdesk & Support
Posted May 17, 2018 - 09:55 CEST
Update
Status 17-05-2018 CET | SAN storage replacement node from Amsterdam arrival delayed

Late last night we were notified by our SAN storage partner, that the SAN storage node unfortunately was delayed arriving from Amsterdam.

However, all systems have been up and running on the secondary SAN storage node in the meantime. At this time the secondary SAN node is however strained from performing for both nodes, as they usually load balance the technical task of running servers, websites and mail etc., but is handling the job satisfactory.

Our technical staff is and has been on high alert since yesterday, and is still keeping in touch with our SAN storage provider, whom we have been told should arrive at the datacenter around 08:30 CET with the replacement part for primary SAN storage node.

Datacenter: Skanderborg, Jutland

Start: 16-05-2018 @ 12:30 CET
End (expected): 17-05-2018 @ 12:00 CET

Affected services: Shared webhosting, Shared POP3/IMAP e-mail (however not Exchange) and virtual servers could be affected by performance issues.

We apologize for any technical issues and disturbances experienced due to this - and will resolve them soonest possible.

Best Regards, CloudNordic Helpdesk & Support
Posted May 17, 2018 - 08:12 CEST
Update
Our SAN storage provider is currently equipping a technician with a replacement for our primary SAN node, as it is not functioning correctly. He will be on an airplane from Amsterdam shortly, and should arrive in the datacenter before midnight today with the replacement unit.

At this time the secondary SAN node is strained from performing for both nodes, as they usually load balance the technical task of running servers, websites and mail etc.

However, all services are up and running, as the secondary node is working hard to maintain uptime for these services until the primary node has been replaced.

Our technical staff is keeping in touch with our SAN storage provider and will meet up at the datacenter upon arrival from Amsterdam.

Datacenter: Skanderborg, Jutland

Start: 16-05-2018 @ 12:30 CET
End (expected): 16-05-2018 @ 00:00 CET

Affected services: Shared webhosting and virtual servers can be affected by performance issues.

We apologize any technical issues experienced due to this - and will resolve them soonest possible.

Best Regards, CloudNordic Helpdesk & Support
Posted May 16, 2018 - 14:08 CEST
Monitoring
We have restarted the primary node for the SAN storage, which is a long process, as a lot of failsafe measures is being monitored while the unit is restarting.

However, the SAN node is not responding correctly, why we working on resolving the issue together with our SAN provider, whom we have been in contact with since the start of the incident.

Our technical staff is therefore still working on the issue and hope to resolve it soonest possible.

Datacenter: Skanderborg, Jutland Start: 16-05-2018 @ 12:30 CET
End (expected): 16-05-2018 14:30 CET

Affected services: Webhotel, E-mail and DNS (however not Exchange) services

We apologize for the technical issues and will resolve them soonest possible.

Best Regards, CloudNordic Helpdesk & Support
Posted May 16, 2018 - 13:41 CEST
Update
We are continuing to work on a fix for this issue.
Posted May 16, 2018 - 12:59 CEST
Identified
One of the primary nodes in the SAN storage has crashed and we are experiencing delayed performance issues, due to I/O (read-write) problems on the SAN, as the secondary node is working hard to maintain uptime for services.

Our technical staff is working on the issue and hope to resolve it soonest possible.

Datacenter: Skanderborg, Jutland

Start: 16-05-2018 @ 12:30 CET

End (expected): 16-05-2018 14:30 CET

Affected services: Webhotel, E-mail and DNS (however not Exchange) services

We apologize for the technical issues and will resolve them soonest possible.

Best Regards, CloudNordic Helpdesk & Support
Posted May 16, 2018 - 12:58 CEST
This incident affected: CloudNordic services (Shared Web Hosting, www.cloudnordic.com, Virtual Servers).