Recent Server Downtime

As many of our customers noticed, all web sites operated by Cracked Egg Studios, LLC, including VaultWiki.org, were unavailable from December 18 - December 24, 2013.

Server Compromise

On December 12, our server began experiencing database corruption and by December 14, we learned that our IP address had been added to a number of email block lists. We spent the next few days investigating and our host found that a virus had hijacked our internal Tomcat instance and changed its security settings so that it could bypass our mail software to send SPAM.

On December 18, our web sites were replaced with a generic maintenance message suggesting we would be back in a few hours. Unfortunately, the infection was more persistent than initially thought, so our host's estimate of 6 hours quickly turned into the better part of a week.

How This Affects Users

While there was no evidence that the virus or an unauthorized user had accessed sensitive information (in fact we believe the virus simply propagated from an infected workstation that was connected via SSH), we recommend that any users who submitted any Install, Upgrade, or Import service requests during the month of December should change their site and FTP login information where possible. If you have ever purchased one of these services before December and want to be extra cautious, please do not hesitate to do the same.

We will also be requiring that all users on our sites reset their passwords, in case user salts and passwords were collected.

Obviously this did delay the development of the next release. We had already lost several weeks in December due to malfunctioning workstation hardware.

Changes to Our Implementations

With this recent event, we noted some ways that our server security could be increased and have already done so.

In addition, we will be adding another level of separation to the services that were affected by this infection. Moving forward, we expect to move our download area to a completely separate Amazon server that can achieve the same functionality without having Tomcat installed at all. Further, we intend to begin processing email traffic through an off-server service as well. These changes should begin rolling out during January.