PHP Fog Blog

How We Got Owned by a Few Teenagers (and Why It Will Never Happen Again)

Hi, I am Lucas Carlson, founder and CEO of PHP Fog and the guy who hasn’t slept in almost 4 days. This is my story.

Saturday, March 19th, 2011 – 10:01pm – It was a dark and stormy night in Queensland, Australia. Elliot, a 16 year old student, should have been preparing for his final exams on Monday. Instead he was in a race with John, a 16 year old student living in New York, and “turby” to deface the PHP Fog site the fastest.

Before we continue, it’s important to explain how our infrastructure works so you will understand the nature of the exploit.  Rather than running a shared hosting environment like the ones you may be familiar with, PHP Fog provides dedicated servers for each one of our customer applications. Each application stack also includes a caching layer, a load balancing layer, a database layer and a shared failover environment. The entry point of this exploit was our shared failover environment, which we will be discussing in more detail as the story progresses.

Thursday, March 17th, 2011 – 8:20pm – In a forum administered by John, some of our beta customers apparently did not understand that we intended to provide our users dedicated EC2 instances. They felt proud of themselves after uploading code to those servers that ran a remote shell and proceeded to compromise the servers we assigned them (the first of many violations of both US and Australian law).

The way they did this was uploading a program and executing it with our post-deploy hooks. Internally at PHP Fog we were aware of the potential security threat behind post-deploy hooks and were about to disable them indefinitely on Friday, March 18th but our software for deploying the site update malfunctioned and we decided to put it off for the weekend. What unfortunate timing.

However the servers that people run their applications on do not have access to other parts of the system, so these kids hit a dead end. Their exploits got them no further than simply signing up for Amazon’s free tier of EC2 service. A security hole, to be sure, but not a very exciting one.

Saturday, March 19th, 2011 – 10:32pm – Elliot was different. He caught a lucky break – ironically enough when the dedicated server he had running on our system died.

Some more background is necessary. The failover system we use works as follows: every application on PHP Fog was simultaneously deployed in both their dedicated instance as well as a shared hosting environment. Nginx was configured so that if your dedicated instance stopped responding for any reason (hardware or network failure) it would automatically redirect requests to the shared hosting environment.

The failover system has almost never been used in the history of PHP Fog and then only in rare occasions where we needed to move people into new hardware did it get utilized. For weeks we had been working on securing this environment with various industry standard tools and we were days away from replacing this server with a locked down, more secure version. Another timing mistake.

Of the thousands of servers we had running, Elliot ’s dedicated server died and his app was in failover mode. When he followed the instructions provided on John’s page, he broke into our shared hosting environment that had not yet been locked down.

This failover server should have been taken offline a long time ago. It was a relic that I had built as a proof of concept. We were replacing it, but I should have just taken it down until we had the replacement. Unfortunately and stupidly, I had an old copy of the site code on that server which had our PHP Fog system passwords that I also stupidly had not deleted or changed. This was really naive and irresponsible of me. The old code-base, all our proprietary intellectual property, was posted for around 5 minutes to twitter.

You can be sure every single system password at PHP Fog has been changed and they are not put on servers any more and I have more than learned my lesson here.

Saturday, March 19th, 2011 – 10:46pm – Less than 15 minutes later, our systems super-star Jake Olsen (not “turby” Jake) noticed something was not right.

Saturday, March 19th, 2011 – 10:49pm – Jake proceeded to boot Elliot off our systems and reboot servers.

Saturday, March 19th, 2011 – 11:09pm – Jake shut down every server on PHP Fog. Without access anywhere else, Elliot logged into our twitter account, our blog, and our DNS manager. He pointed to a site John called “PHPFog sucks,” he bragged about his exploit on our twitter and blog.

Sunday, March 20th, 2011 – 2:15am – Elliot sent me an IM. Apparently he was now sorry. The only thing going through my head was be nice to him, we need as much cooperation as possible right now.

2:15:12 AM Elliot : Lucas.
2:15:23 AM Elliot : Listen, before you begin, I want to apologize.
2:15:35 AM Elliot : I do this sort of thing for kicks, but I agree that this went a little too far.
2:15:41 AM Lucas: before you apologize can you at least take down the site explaining the exploit
2:15:51 AM Elliot : Unfortunately, that’s out of my control.
2:15:59 AM Elliot : I don’t run that domain, however I will talk to the owner tomorrow. He’s gone to bed.
2:16:36 AM Elliot : I don’t want any hard feelings between us, this originally started as a proof of concept to prove your platform was insecure.
2:16:44 AM Elliot : I guess I did that, but there are better ways I could’ve gone about it.
2:16:58 AM Elliot : Yes, it was me as root on your servers, and in your twitter, and etc.
2:16:59 AM Lucas: I really wish you had reached out to me before this
2:17:04 AM Elliot : So do I, now.
2:17:12 AM Elliot : You guys are funded and I could’ve lost you a lot.
2:17:21 AM Lucas: a whole lot
2:17:28 AM Lucas: a lot of people’s lives depend on this
2:17:37 AM Elliot : I didn’t touch anybody’s files.
2:17:39 AM Elliot : Only phpfog’s.
2:17:49 AM Elliot : Didn’t even look through them.

2:21:25 AM Lucas: can you give me the twitter password?
2:21:32 AM Elliot : I’ll change it back for you.

2:54:44 AM Elliot : Well, look on the bright side. At least it was us three, who got in just for kicks, and then told you how instead of someone who got in, pulled an rm -rf / on all of your stuff, and then changed all of your passwords.
2:55:16 AM Elliot : Wait, did I tell you how?
2:56:01 AM Lucas: not yet
2:56:08 AM Elliot : Want a brief?
2:56:11 AM Lucas: sure
2:56:25 AM Elliot : It relied upon a glitch in your system
2:56:29 AM Elliot : which ended up with my app
2:56:32 AM Elliot : being on your main node or something
2:56:37 AM Elliot : instead of being on its own instance
2:56:45 AM Elliot : then I used the method detailed by turby
2:56:46 AM Elliot : to gain root
2:57:00 AM Elliot : then I just searched around for a password, the one that worked for me was ••••••••••
2:57:08 AM Elliot : Then I went a little further and found ••••••••••
2:57:24 AM Elliot : then just basically logged in and posted on your blog, on your twitter, and that was about all.

3:06:20 AM Elliot : Well, we’re outta here for now.
3:06:28 AM Lucas : ok
3:06:40 AM Elliot : ‘Night lucas. Sorry about what we pulled again. :\

Our forensic analysis after the fact corroborated Elliot’s story of vandalism. We found no evidence of anyone besides Elliot breaking into our systems beyond the individual dedicated servers with no compromising information.

Even though it was a case of vandalism, none of us at PHP Fog were going to take any chances at all. Here are the steps we have taken since. We worked through the weekend and nearly non-stop since to get sites running again. At this point 99% of the sites are running and secure again.

Credit cards – We have never stored credit cards on any PHP Fog server. There was never any possibility that credit cards could have been compromised by this attack.

Rebuild every single server on PHP Fog – We shut down and re-created every single server we controlled. This numbered in the thousands. We had to be 100% sure that there were no rootkits anywhere and this was the only way to do that.

No more shared passwords, anywhere – We are no longer using shared passwords. They were a short-term stopgap measure we had been planning to replace, and now they have been replaced.

Change every ssh key/password/token/api key everywhere – In the last 3 days we basically rebuilt everything from scratch from the ground up.

Eliminate shared hosting failover server – We may never do shared hosting failover again if we can not guarantee its security. We might do a non-realtime failover to automatically launch a new instance for you, but this experience taught us what a bad idea this can be.

Eliminate post-deploy hooks – Until we can do this securely, we are removing it from our features.

Eliminate custom Apache conf and php.ini – Until we can do this securely, we are removing it from our features as well. Users may still rely on .htaccess files.

Complete lockdown and rebuild of the app’s dedicated servers – We have audited our dedicated servers to provide a much more secure environment that will be much harder to exploit through the techniques listed in the forum. We started out being quite trusting of our beta users, but have had to limit what they can do now in order to protect us all.

Upgrade internal password storage – Account passwords were cryptographically hashed, however we are clearing everyone’s password and before you can log in you will need to enter a new password.  We are emailing password reset links to all of our beta users. Going forward, passwords are hashed with an even more secure algorithm.

Upgrade internal communication systems – Although these were not attacked this weekend, we have secured them anyway. SSH keys for git deploy have been generated on a per-server basis so there is no possible way to get “keys to the kingdom”. Code deploys onto dedicated servers are now read-only so compromised servers can not modify the main code repository.

App password changes – While we have no evidence that our users’ passwords have been compromised, we strongly advise every beta user at PHP Fog to change the passwords of the users in their applications (WordPress, Drupal, etc). We will also provide tools to change the database passwords. If you are using a password you share with other sites, learn from our example: change them all to strong, unique passwords and use a secure password manager such as 1Password or LastPass to store them.

Regular penetration testing – We have hired professional white hat hackers with government level security experience to attempt regular pen tests on our system, both as regular users as well as giving them special access and seeing if they can get through.

Audit of the vandalism – We found no evidence that our customers’ code or databases were accessed at all during the event. Since we keep all the customer code in cryptographically secure git repositories, it is almost impossible to modify these repositories without SHA1 hashes revealing the changes.

This is an amazing amount of work for 3 days and I am incredibly proud of our team at PHP Fog. We made sure our system was rock solid before bringing any sites back up and it took a massive effort. This is the best group of engineers I have ever seen. Thank you, guys!

I also want to thank the PHP community. We thought that we would be mocked and be bombarded by angry tirades, but the complete opposite has been the case. At the end of the day security is our responsibility, but all systems are prone to attack. Human error, bad timing, and oversight caused ours. Our beta testers have encouraged us to bounce back while denouncing the childish and criminal acts against us. We thank you all so much and will not let you down again.

We are talking to our legal counsel and the FBI and may press charges. This kind of behavior will not be accepted. Ever. There are proper disclosure protocols for handling this kind of situation and none of them were respected.

That said, we highly encourage our users to help us strengthen our security in a pro-active way. If you find a security flaw and report it using the Full Disclosure Policy to with notice, we will help strengthen your security reputation in a very public way and reward you generously.

  • Anonymous

    I’m starting to think that you are the same guy, entering the forum with different accounts. And then switching accounts and liking your own posts.

  • Isa K

    Well by that reasoning: Lucas and the team could have tracked them down individually and garroted them with an ethernet cable, or posted their personal information on 4chan and inflicted months of torturous abuse and harassment. Why are you complaining? It could have been so much worse!

    If you’re willing to congratulate the hackers for breaching the system because it could have been worse, you should be equally willing to congratulate phpFog for reporting it, because they could have done so much worse in response and choose not to.

    See how your logic is bullshit now?

  • Anonymous

    I have little or no regard on how this will affect John and Elliot. That’s up to the judge.

  • Sam

    Nobody disagrees with you that their ultimate punishment should take into consideration their age. My own opinion has taken that into account. However, they do require punishment, and not something minor. What they did was not minor, and they’ve seriously hurt people. 20 years might be too much, but a criminal record isn’t, because what they’ve done is criminal.

    I’m pretty sure people know that black hat hacking will result in jail. If they didn’t know that, they were even more moronic than we’ve already made them out to be.

  • Anonymous

    By your logic, the teenagers will be LUCKY to be separated from their families, from whom they apparently can’t learn the basics of ethics.

  • Anonymous

    I would be sad that my child was a criminal.

  • Sam

    Thanks Nick, that’s a level-headed post you made there. Hopefully they realize that what they’ve done is wrong.

  • Bilawal Hameed

    Are you saying regardless that these kids apologized to you that you are still going to destroy their lives? They even co-operated with you.

  • Ax

    …OR “let’s hack a website and ruin MY life” was what these kids should have thought. Don’t blame the victim.

  • Ax

    Maybe the kids broke in and then gave passwords back, etc. to help them fix it…once the kids realized they had committed federal crimes? Maybe the gravity of committing felonies hit them once they were in the act?

    In any case, the societally desired outcome of a case like this has been decided in the legislatures and courts over hundreds/thousands of prior similar cases, and this case isn’t actually all that special (unless you want to bet this is the first time a computer hacker has been 16?). So why reinvent the wheel? Use the template (aka laws) and the justice-compiler (the courts) and the debugger (penal system), if necessary?

  • Sam

    First of all, I’m not PHPFog. I’m just another software developer who happens to know what it means to get stomped on by some malicious skript kiddies.

    But they clearly haven’t shown remorse for their actions, and apologies don’t change the fact that they distributed private intellectual property. Sorry won’t un-destroy my priceless art that you walked into my house and wrecked. PHPFog has every right to press charges for that. The other idea is that charges will ruin their lives — they will have a black mark on their record but it’s deserved. As far as doing time in prison, they should be happy if they don’t feel the full extent of the law. PHPFog not pressing charges is not an obligation because they were wronged and wronged in a big way. It is within every right of the company to seek a just outcome.

  • Ksmith27

    Great job on this guys. And +10 for the transparency. It can be painful to do that, but the fact that you take responsibility for the screw up and make no excuses (but offer a complete explanation) is precisely what is needed build trust with your customers.

    The folks that will throw out disparaging comments here are most likely a) people who have zero concept of what it takes to build a startup, b) people who have no idea what it takes to build a truly scalable and secure platform, c) people who would never even think of being this transparent if (God forbid) they ever made a big mistake, or d) all of the above.

    This shit will be a defining moment for your team that lay a foundation for success for years to come.

  • Patrick Mandia

    The lack of accountability is a bit disconcerting, which makes buying into the service a much harder decision.

    “we were aware of the potential security threat behind post-deploy hooks and were about to disable them”

    “days away from replacing this server”

    I understand the service is in beta (alpha?) but putting the blame on timing is a bit ridiculous. What they did was wrong, but so is blaming your faults on timing.

  • Austin Hanson

    Press charges. Not pressing charges would be ridiculous.

    “Oh no, I’m 16, you can’t be mean to me!”

  • Rpg

    You guys are mere script kiddies. Next time learn to cover your tracks.


    John Du Hart

    164 Newfield Lane
    Centereach, New York 11720

    Phone: 631-766-7842

    Elliot Speck


    15 Viola Drive
    Redland Bay
    Brisbane, Queensland 4165

    Phone: +61.0458368251


    Aliases: zayfox, elliotspeck, qombat

  • Lol

    You guys are mere script kiddies. Next time learn to cover your tracks.


    John Du Hart

    164 Newfield Lane
    Centereach, New York 11720

    Phone: 631-766-7842

    Elliot Speck


    15 Viola Drive
    Redland Bay
    Brisbane, Queensland 4165

    Phone: +61.0458368251


    Aliases: zayfox, elliotspeck, qombat

  • Mark

    $10,000 worth of labour that would have been done anyway. You need to subtract the amount of effort it would have taken to confidently secure the system if this hadn’t happened. And given that PHPFog have clearly learned from the experience, I suspect the job has been done better this way.

    (Also, is $100/hr really the going rate for a sysadmin round your parts? I might need to move.)

  • Mark

    When the crime is related to IP, security and trust, being honest and co-operative lessens the harm caused. This is highly unrelated to murder or theft.

  • Mark

    I love all the analogies you come up with. Do PHPFog run a restaurant? Are they at risk of going out of business because their twitter account was defaced? This phenomenon of bored teenagers discovering exploits is something unique to computing, and has very different dynamics to traditional punishment models.

    Consider the costs to both sides, and give the kids something to learn from. And bear in mind that costs of legal action would dwarf any costs from the last 4 days.

  • Nicholas Anderson

    I assure you I am one and only, it would be pretty ridiculous, not to mention pathetic, for me to enter measures like that for an argument such as this one. I have never “liked” my own posts, as that doesn’t really matter. My only goal is to have everyone see the other side of all of this.

  • Anonymous

    Your system is a joke and you got what you deserved. Maybe if they had just told you, you wouldn’t have learned such a hard lesson.

    I don’t even know what phpfog does but from the name alone I can tell you are inept morons.

  • Anonymous

    Cornkits – spare us all the lecture on morality.

    If phpfog can’t even prevent simple exploits like this then who knows what adept systems crackers could have gotten away with. Quite simply – if you can’t secure you servers in this day and age – you deserve all you get.

    You’re right it’s not “play time”. So why the hell are you suggesting it’s acceptable to “play” when it comes to security? F***ing idiot. I bet you are yet another PHP idiot who hasn’t got a clue.

  • Sam

    What I think is the more pressing financial damage is the leak of intellectual property. The source dump allows anyone to create a PHPFog of their own.

  • Jay

    They have their intellectual property, basically the whole of the worth of their company, distributed across the internet. How do you imagine that damage can be undone?

    Not to mention the damage to their reputation. The issue isn’t whether they “would have been hacked anyway,” the issue is that hacking is WRONG. And real damage was done.

    I agree with the last statement you made, but I’m not sure you understand the costs on the side of PHPFog. As a software developer myself I empathize strongly with them. I can’t imagine what it’d be like to have everything that I’m passionate about destroyed by moronic teenagers who don’t give a damn about my feelings or how much hard work I had put into things. I would forgive them, had they proven to be truly repentant, and spared them a criminal record via dealings outside of the court. But I would still like reparations for the damage done monetarily, and in this case the perps make it goddamn hard to forgive. Their public statements are haughty, self-righteous, and refusing to take responsibility for what happened. There is no apology in that, and there ought to be.

  • Sam

    Your personal attack cornkits is childish and inane. Your writing is much like that of Jordi33 and PieClock. Are you the same person? Your inability to write with civility speaks to a childish attitude. Grow up.

    Quite simply, the argument going on here isn’t whether “PHPFog got what they deserved,” because nobody deserves to be treated the way they were treated by these hackers. If you think that anyone might deserve getting their lives trampled on, then you are a cold unfeeling person.

    Regardless of what they “deserved,” the issue here is that hacking is wrong, no arguing with that. Messing with peoples’ lives ought to result in a just punishment, and that’s what people are talking about here. The law, as heavy handed as it can be, agrees with this fact. Now all that’s left is for PHPFog to judge for themselves how they plan on doling out justice — they have every legal right to press charges, and they have every moral justification to seek reparation in some way or form.

    There are consequences to actions in life. PHPFog faced/continues to face theirs already. The kids need to own up.

  • Mark

    I can’t see that as a serious risk. The Windows source code is available through legitimate and illegitimate means, but Microsoft doesn’t worry about people creating their own copies of Windows. Besides, if I were to create a PHPFog of my own, I would implement it differently and open source it.

    The real issue is that they had access to the customer’s source code, but as far as I know, that was not leaked.

  • Mark

    Then you’ll agree it’s not about the code, but the customers. That’s why we’re called developers, and not engineers.

    Nothing was destroyed: a security boundary was breached, code leaked briefly and some childish defacement occurred. Some beta customers may leave, some may join, but an appropriate level of security has been implemented. And this has all happened during the beta period. Nothing to see here, move along, please.

  • Sam

    It’s clearly not the same thing, and what you would do isn’t indicative of what other people will do. Also if you know much about PHPFog, you realize that there are a lot of competitors in the same space. They’re still a startup — the codebase is relatively small and their IP is their competitive advantage. This hurts them in many ways — it can also affect their ability to further acquire funding.

    Thankfully the attackers had the sense not to make dumps of customer data.

    At any rate, the overall damage done to the company is not just a trivial calculation, and it shouldn’t be overlooked or made out to be smaller than it really is.

  • Jay

    Well the code serves the customers; they’re inseparably intertwined. I don’t understand what you’re trying to say regarding developers vs engineers? Software developer == Software engineer in my book.

    To say that nothing was destroyed is overlooking a lot — it’s what the attackers themselves are trying to say, and it’s gotta be made clear to them that this is untrue. They have to recognize that.

    Reputation and unique retention of intellectual property were both destroyed. The people who work for the company were hurt and will continue to be hurt by what transpired as a result of the hackers. You can’t say that what happened doesn’t matter — it’s only easy for you to say because you aren’t the one dealing with the mess.

    In the end, I think we agree that the hackers deserve to be punished. What we may not see eye to eye on is just how much damage was done and to what degree the hackers ought to be penalized as a result. I think it’s unfair to glaze over PHPFog’s suffering. What transpired here was criminal, harm-causing action, and to brush it off like a minor issue is unjust.

    I’m glad you like my analogies; PHPFog may not run a restaurant but in the example that recipe is an example of IP. It’s NOT okay to give away others’ IP. This isn’t different in any way, other than the fact that software code is several magnitudes more complicated than cooking recipes.

  • Mantor

    Don’t ruin there bright lives that they have ahead of them. They pointed out a security flaw in your servers and didn’t format everything like a real dick would have done. Some other cracker could have exploited that fault and could have done way worst things. You should smack the shit out of them then reward them.

  • qvc

    links to stuff if you missed it

  • qvc

    here’s the forum post he made originally

  • Jacky Alcine

    Wow, you guys are so tight.

  • Guest

    Do smoking, taking drugs, or binge drinking involve hurting other people directly?

  • Bountyhunters

    You do the crime you do the TIME!
    How much have cost this company and the people who built it in TIME & MONEY??

    Bounty Hunter’s

  • bananaranha

    “Elliot and John (Charlie wasn’t really involved) are both 16 year olds who are very ambitious about computers, and have a very bright future. Like all kids, they made a small mistake.”

    A small mistake is dropping a tv set and breaking it, parking in a no-park zone, etc.

    This was not a mistake –it was deliberate and it had huge costs.

    Oh, and 16 year olds are even tried as adults in many cases…

  • bananaranha

    “I’m on the side of people who don’t support ruining lives of teenagers who make ridiculous mistakes like this.”

    Who said to “ruin their lives”? Just hit ’em with something like a $100,000 fine and they’ll think twice next time…

    As in, sorry, guys, no college for you….

  • bananaranha

    So, if I leave my street viewing windows open, it’s ok for thugs to break into my house and break things?


  • bananaranha

    “”Did you really just compare hacking to murder?””

    No. He EXPLICITLY SAID: “Being 16 and stupid doesn’t get you off the hook for killing someone, for stealing something, or any other crime. Causing irreparable damage to people should result in justice served, period.

    It’s YOU who singled out “murder” in his argument, in order to confuse things.

  • bananaranha

    Talk about backwards logic…

  • bananaranha

    You are very cruel with these kids. They are 16 year old! There are kids that age that beat other people, that burn houses, that steal and cause real damage.

    And stealing IP property, hacking and closing temporarily the servers of an internet business is also “real damage”.

    Do you believe the internet is just some magical fairy land with no relation to reality and stuff that happens there affects noone because it’s all digital?

  • bananaranha

    Spare the melodrama.

    “Destroy a family”?

    How about, hand them an appropriate fine for their CRIME?

  • bananaranha

    “””When the crime is related to IP, security and trust, being honest and co-operative lessens the harm caused”””

    Huh? How being honest and co-operative fixes the IP theft?

  • bananaranha

    “””While I usually dislike people cracking something, then posting about it just to prove a point, it seems to have worked this time :)”””

    Yes, and if someone breaks into your house, and posts about it can also make you lock it better –invest in some new doors, window locks, alarms etc. It “worked”.

    But the burglary remains…

  • bananaranha

    And if it was your child, still going to high school, who happened to use skills in an area they’re passionate in (cars), in

    stealing someone’s car,

    would you or would you not support charging them as a criminal, and sending them to jail (along with rapists, murderers and whatnot)?

  • bananaranha

    “””Oh no they lost 3 days! Let’s press charges and ruin 3 kids whole LIFES.”””

    It’s “lives”.

    And 3 days can mean hundreds of millions of dollars in lost revenue for a company.

    (Imagine Amazon shutting down for 3 days).

    It’s not like the took 3 days of vacation.

  • bananaranha

    “”I love all the analogies you come up with.””

    But you don’t understand them.

    “””Do PHPFog run a restaurant?”””

    No, they run a website (another public facing service). That’s why it called an ANALOGY.

    And the “secret recipe” is the proprietary source code.

    “””Are they at risk of going out of business because their twitter account was defaced?”””

    Yes. Companies use their twitter account for keeping in touch with customers and potential customers.

    Oh, and you conveniently forgot the part about the IP theft and distribution, the rerooting of their website, the closing down of their service, etc.

    “””This phenomenon of bored teenagers discovering exploits is something unique to computing, and has very different dynamics to traditional punishment models.”””

    Says who? Surely not the law.

    “””Consider the costs to both sides, and give the kids something to learn from.”””

    A huge fine?

    “””And bear in mind that costs of legal action would dwarf any costs from the last 4 days.”””

    Not if the culprits are made to pay for it in fines…

Powered by Olark