After a thorough investigation, we identified the root cause of the incident: a software issue between the AMD platform and IOMMU caused the PCI devices backing the NVMe drives on cluster node ams-9950x-odessana to malfunction on a software level, forcing the system to switch into a read-only mode and render the VMs unoperable.

The issue was initially misdiagnosed as a drive failure, which led to an extended delay while we requested remote hands at the data center to replace the suspected faulty drive. Further diagnosis revealed the actual cause. We adjusted the BIOS settings accordingly, then gracefully recovered and resynced the RAID array on the node.

All of the VMs are back up since 00:04 AM UTC, and all customer data is safe. However, we strongly recommend running a file system check on any virtual machines hosted on ams-9950x-odessana to rule out any latent issues.

We sincerely apologize for the disruption and the extended downtime. We will be providing two extra days of service time to all customer services that were affected by this outage.

We are continuing our work on resolving the issue.

We found the cause of the issue and are working with the data center engineers to diagnose and resolve it as soon as possible.

We are investigating the issue on the ams-9950x-odessana node in the Netherlands.