Over the past week, the virtual machine management platform "VMmanager", which we use to manage our VPS infrastructure, was affected by a severe security incident.

Based on the extensive analysis carried out over the past seven days, we found no evidence that any customer VMs were affected. None of the customer VMs were accessed, read, or copied by the attackers. Our early detection and response prevented the attack from progressing beyond the management platform.

As promised, we are providing a full post-mortem of the incident.

🕐 October 12 - initial compromise

At 03:01 UTC on October 12, an attacker gained access to our VMmanager platform through a chain of vulnerabilities in the software.

The attacker moved through several API endpoints, exploiting vulnerabilities within the platform's code, until they finally reached the platform's internal containers. One of these endpoints allowed an arbitrary task to be launched with an attacker-controlled payload through the ISO mount functionality - all without ever having an account and actually authorizing into the platform. This ultimately gave the attacker the ability to execute their own commands on the platform. Using this access, the attacker created two administrator-level API tokens, both valid until 2036. Their activity was disguised as ordinary platform tasks and mixed in with older legitimate tasks.

The attackers later attempted to delete their traces, but we were able to recover the deleted records during our investigation. The last recorded use of these tokens was at 00:20 UTC on October 13. At this stage, we were not aware of the ongoing attack, and neither were the developers of the VM platform, along with its many of hosting providers who used the VM platform.

🕐 October 14 - detection and containment

On October 14, our team was conducting a security review of a different ISO-related vulnerability, unrelated but very similar to the vulnerability chain used in this attack.

During this review, we noticed the two unusual API tokens that had been issued to an administrator account without our involvement. We immediately preserved the tokens as evidence and revoked them from the platform.

A few minutes later, we restricted public access to VMmanager. Access to vm.senko.digital was blocked for all users and was replaced with a 403 page about ongoing maintenance, asking customers to use the Client Area to manage their services instead. At the same time, we applied additional hardening to further reduce the platform's attack surface.

At the time, we did not observe any logs showing the attacker's payload being executed or any additional access to the panel. This turned out to be critical: the attacker's access to the platform had already been revoked 3 days before the later stages of the malicious campaign were observed against other hosting providers - another hacker bites the dust.

🕐 October 16-17 - the wider campaign is real

As our investigation continued, we established that this was not an isolated attack against Senko Digital. VMmanager is a commercial product used by many hosting providers and at least several dozen providers were affected by the same campaign. The timing and progression of the attacks varied between providers, so October 17th should not be understood as the universal start date of the campaign.

During the investigation, we worked closely with other affected hosting providers, sharing information about the vulnerabilities, attacker activity and malicious payloads found on their systems. Evidence from other affected providers showed that, after gaining access to VMmanager installations, the attackers' toolkit was capable of progressing beyond the management panel.

Among the components found were:

1. hidden backdoors
2. fake system services
3. tools designed to collect data from customer VM disks
4. tools designed to export the platform database
5. mechanisms that automatically restored attacker-controlled API keys

On one affected provider's infrastructure, for example, the investigation showed platform-level access on October 16th, followed by payloads on October 17th that enumerated cluster nodes, interacted with Docker environments and retrieved database credentials. This information was extremely valuable, as it gave us additional clues on what we needed to look for on our own infrastructure. By this point, however, the attacker's access to our platform had already been revoked on October 14th. The later stages of the campaign therefore did not execute against our VM platform.

🕐 October 17 - infrastructure-wide audit

As an additional precaution, on October 17th we replaced the platform's management SSH key on all 94 cluster nodes.

With the information gathered from our own investigation and from other affected providers, we performed an extensive audit of our entire infrastructure. This included, but was not limited to, all of our cluster nodes, the old platform server, the platform database, access credentials, management mechanisms and everything else the attackers were known to have targeted on other providers' systems. Every one of our cluster nodes came back clean.

We found no evidence of:

1. anything being executed inside customer virtual servers
2. customer VM disks being read or copied
3. the platform database being dumped
4. injected code or backdoors on the platform server
5. injected code or backdoors in the database
6. VM passwords, VNC passwords, or IP allocations being retrieved
7. the platform's management SSH key being stolen

The attacker's toolkit did contain functionality capable of reaching cluster nodes directly, but our records show that this functionality was never used against our infrastructure.

As the compromised VMmanager server was already quarantined, we subsequently prepared a new installation of VMmanager on a completely new server, isolated from the public internet from the very beginning. This ensured that the new installation was not exposed to the same vulnerability as soon as the panel was brought online.

We reviewed the database backup from the compromised server, found no embedded malicious code or injections, cleared all existing access keys and sessions, replaced the management SSH key again and reset all VMmanager user passwords.

For full transparency, on October 17th we temporarily lifted the Cloudflare restriction for approximately 5.5 hours so that our customers could access the panel and finish the operations they needed - our VM platform was already hardened to be able to withstand further kinds of such API attacks as seen in the initial intrusion on October 12th.

We found no signs of renewed unauthorized access during this period.

What our (and other affected hosting providers') customers should do

Although we found no signs that customer data was accessed, we still recommend taking the following precautions:

1. Change your server passwords, especially if you are still using the password provided during server creation or reinstall
2. Switch to SSH key authentication and disable password-based SSH logins
3. Rotate credentials stored on your servers, including but not limited to API keys, tokens, database passwords and private keys

These are precautionary recommendations and do not mean that we found evidence that these credentials were compromised.

Current status

VMmanager will remain unavailable for public access indefinitely.

Your virtual servers continue to operate normally, and basic VM operations remain available through the Client Area.

Because the temporary unavailability of VMmanager affects our customers' ability to properly manage your virtual servers, we will be launching our brand-new Client Area in open beta within the next few days. It will allow customers to manage their services much more efficiently, with a substantially improved UI and UX.

Conclusion

The most important conclusion from our investigation is straightforward: the attack was contained to the VM platform. It did not spread to our cluster nodes or customer virtual machines.

We will continue monitoring the situation and will provide further updates if our findings change.

If you have any questions or concerns, you can contact our support team through the Client Area, Telegram support bot or over e-mail at [email protected].

We sincerely apologize for this incident and for the disruption it has caused.

Thank you for your patience, understanding and continued trust in Senko Digital!