Hi, I am Lucas Carlson, founder and CEO of PHP Fog and the guy who hasn’t slept in almost 4 days. This is my story.
Saturday, March 19th, 2011 – 10:01pm – It was a dark and stormy night in Queensland, Australia. Elliot, a 16 year old student, should have been preparing for his final exams on Monday. Instead he was in a race with John, a 16 year old student living in New York, and “turby” to deface the PHP Fog site the fastest.
Before we continue, it’s important to explain how our infrastructure works so you will understand the nature of the exploit. Rather than running a shared hosting environment like the ones you may be familiar with, PHP Fog provides dedicated servers for each one of our customer applications. Each application stack also includes a caching layer, a load balancing layer, a database layer and a shared failover environment. The entry point of this exploit was our shared failover environment, which we will be discussing in more detail as the story progresses.
Thursday, March 17th, 2011 – 8:20pm – In a forum administered by John, some of our beta customers apparently did not understand that we intended to provide our users dedicated EC2 instances. They felt proud of themselves after uploading code to those servers that ran a remote shell and proceeded to compromise the servers we assigned them (the first of many violations of both US and Australian law).
The way they did this was uploading a program and executing it with our post-deploy hooks. Internally at PHP Fog we were aware of the potential security threat behind post-deploy hooks and were about to disable them indefinitely on Friday, March 18th but our software for deploying the site update malfunctioned and we decided to put it off for the weekend. What unfortunate timing.
However the servers that people run their applications on do not have access to other parts of the system, so these kids hit a dead end. Their exploits got them no further than simply signing up for Amazon’s free tier of EC2 service. A security hole, to be sure, but not a very exciting one.
Saturday, March 19th, 2011 – 10:32pm – Elliot was different. He caught a lucky break – ironically enough when the dedicated server he had running on our system died.
Some more background is necessary. The failover system we use works as follows: every application on PHP Fog was simultaneously deployed in both their dedicated instance as well as a shared hosting environment. Nginx was configured so that if your dedicated instance stopped responding for any reason (hardware or network failure) it would automatically redirect requests to the shared hosting environment.
The failover system has almost never been used in the history of PHP Fog and then only in rare occasions where we needed to move people into new hardware did it get utilized. For weeks we had been working on securing this environment with various industry standard tools and we were days away from replacing this server with a locked down, more secure version. Another timing mistake.
Of the thousands of servers we had running, Elliot ’s dedicated server died and his app was in failover mode. When he followed the instructions provided on John’s page, he broke into our shared hosting environment that had not yet been locked down.
This failover server should have been taken offline a long time ago. It was a relic that I had built as a proof of concept. We were replacing it, but I should have just taken it down until we had the replacement. Unfortunately and stupidly, I had an old copy of the site code on that server which had our PHP Fog system passwords that I also stupidly had not deleted or changed. This was really naive and irresponsible of me. The old code-base, all our proprietary intellectual property, was posted for around 5 minutes to twitter.
You can be sure every single system password at PHP Fog has been changed and they are not put on servers any more and I have more than learned my lesson here.
Saturday, March 19th, 2011 – 10:46pm – Less than 15 minutes later, our systems super-star Jake Olsen (not “turby” Jake) noticed something was not right.
Saturday, March 19th, 2011 – 10:49pm – Jake proceeded to boot Elliot off our systems and reboot servers.
Saturday, March 19th, 2011 – 11:09pm – Jake shut down every server on PHP Fog. Without access anywhere else, Elliot logged into our twitter account, our blog, and our DNS manager. He pointed phpfog.com to a site John called “PHPFog sucks,” he bragged about his exploit on our twitter and blog.
Sunday, March 20th, 2011 – 2:15am – Elliot sent me an IM. Apparently he was now sorry. The only thing going through my head was be nice to him, we need as much cooperation as possible right now.
2:15:12 AM Elliot : Lucas.
2:15:23 AM Elliot : Listen, before you begin, I want to apologize.
2:15:35 AM Elliot : I do this sort of thing for kicks, but I agree that this went a little too far.
2:15:41 AM Lucas: before you apologize can you at least take down the site explaining the exploit
2:15:51 AM Elliot : Unfortunately, that’s out of my control.
2:15:59 AM Elliot : I don’t run that domain, however I will talk to the owner tomorrow. He’s gone to bed.
2:16:36 AM Elliot : I don’t want any hard feelings between us, this originally started as a proof of concept to prove your platform was insecure.
2:16:44 AM Elliot : I guess I did that, but there are better ways I could’ve gone about it.
2:16:58 AM Elliot : Yes, it was me as root on your servers, and in your twitter, and etc.
2:16:59 AM Lucas: I really wish you had reached out to me before this
2:17:04 AM Elliot : So do I, now.
2:17:12 AM Elliot : You guys are funded and I could’ve lost you a lot.
2:17:21 AM Lucas: a whole lot
2:17:28 AM Lucas: a lot of people’s lives depend on this
2:17:37 AM Elliot : I didn’t touch anybody’s files.
2:17:39 AM Elliot : Only phpfog’s.
2:17:49 AM Elliot : Didn’t even look through them.
2:21:25 AM Lucas: can you give me the twitter password?
2:21:32 AM Elliot : I’ll change it back for you.
2:54:44 AM Elliot : Well, look on the bright side. At least it was us three, who got in just for kicks, and then told you how instead of someone who got in, pulled an rm -rf / on all of your stuff, and then changed all of your passwords.
2:55:16 AM Elliot : Wait, did I tell you how?
2:56:01 AM Lucas: not yet
2:56:08 AM Elliot : Want a brief?
2:56:11 AM Lucas: sure
2:56:25 AM Elliot : It relied upon a glitch in your system
2:56:29 AM Elliot : which ended up with my app
2:56:32 AM Elliot : being on your main node or something
2:56:37 AM Elliot : instead of being on its own instance
2:56:45 AM Elliot : then I used the method detailed by turby
2:56:46 AM Elliot : to gain root
2:57:00 AM Elliot : then I just searched around for a password, the one that worked for me was ••••••••••
2:57:08 AM Elliot : Then I went a little further and found ••••••••••
2:57:24 AM Elliot : then just basically logged in and posted on your blog, on your twitter, and that was about all.
3:06:20 AM Elliot : Well, we’re outta here for now.
3:06:28 AM Lucas : ok
3:06:40 AM Elliot : ‘Night lucas. Sorry about what we pulled again. :\
Our forensic analysis after the fact corroborated Elliot’s story of vandalism. We found no evidence of anyone besides Elliot breaking into our systems beyond the individual dedicated servers with no compromising information.
Even though it was a case of vandalism, none of us at PHP Fog were going to take any chances at all. Here are the steps we have taken since. We worked through the weekend and nearly non-stop since to get sites running again. At this point 99% of the sites are running and secure again.
Credit cards – We have never stored credit cards on any PHP Fog server. There was never any possibility that credit cards could have been compromised by this attack.
Rebuild every single server on PHP Fog – We shut down and re-created every single server we controlled. This numbered in the thousands. We had to be 100% sure that there were no rootkits anywhere and this was the only way to do that.
No more shared passwords, anywhere – We are no longer using shared passwords. They were a short-term stopgap measure we had been planning to replace, and now they have been replaced.
Change every ssh key/password/token/api key everywhere – In the last 3 days we basically rebuilt everything from scratch from the ground up.
Eliminate shared hosting failover server – We may never do shared hosting failover again if we can not guarantee its security. We might do a non-realtime failover to automatically launch a new instance for you, but this experience taught us what a bad idea this can be.
Eliminate post-deploy hooks – Until we can do this securely, we are removing it from our features.
Eliminate custom Apache conf and php.ini – Until we can do this securely, we are removing it from our features as well. Users may still rely on .htaccess files.
Complete lockdown and rebuild of the app’s dedicated servers – We have audited our dedicated servers to provide a much more secure environment that will be much harder to exploit through the techniques listed in the forum. We started out being quite trusting of our beta users, but have had to limit what they can do now in order to protect us all.
Upgrade internal password storage – Account passwords were cryptographically hashed, however we are clearing everyone’s password and before you can log in you will need to enter a new password. We are emailing password reset links to all of our beta users. Going forward, passwords are hashed with an even more secure algorithm.
Upgrade internal communication systems – Although these were not attacked this weekend, we have secured them anyway. SSH keys for git deploy have been generated on a per-server basis so there is no possible way to get “keys to the kingdom”. Code deploys onto dedicated servers are now read-only so compromised servers can not modify the main code repository.
App password changes – While we have no evidence that our users’ passwords have been compromised, we strongly advise every beta user at PHP Fog to change the passwords of the users in their applications (WordPress, Drupal, etc). We will also provide tools to change the database passwords. If you are using a password you share with other sites, learn from our example: change them all to strong, unique passwords and use a secure password manager such as 1Password or LastPass to store them.
Regular penetration testing – We have hired professional white hat hackers with government level security experience to attempt regular pen tests on our system, both as regular users as well as giving them special access and seeing if they can get through.
Audit of the vandalism – We found no evidence that our customers’ code or databases were accessed at all during the event. Since we keep all the customer code in cryptographically secure git repositories, it is almost impossible to modify these repositories without SHA1 hashes revealing the changes.
This is an amazing amount of work for 3 days and I am incredibly proud of our team at PHP Fog. We made sure our system was rock solid before bringing any sites back up and it took a massive effort. This is the best group of engineers I have ever seen. Thank you, guys!
I also want to thank the PHP community. We thought that we would be mocked and be bombarded by angry tirades, but the complete opposite has been the case. At the end of the day security is our responsibility, but all systems are prone to attack. Human error, bad timing, and oversight caused ours. Our beta testers have encouraged us to bounce back while denouncing the childish and criminal acts against us. We thank you all so much and will not let you down again.
We are talking to our legal counsel and the FBI and may press charges. This kind of behavior will not be accepted. Ever. There are proper disclosure protocols for handling this kind of situation and none of them were respected.
That said, we highly encourage our users to help us strengthen our security in a pro-active way. If you find a security flaw and report it using the Full Disclosure Policy to firstname.lastname@example.org with notice, we will help strengthen your security reputation in a very public way and reward you generously.