PHP Fog Blog

On Outages, Responsibility, and Transparency

PHP Fog customers chose PHP Fog, they did not choose the datacenter vendor. We chose the datacenter vendor, which means that it is our responsibility to ensure your apps don’t go down and restore them quickly when they do.

Over the weekend, around ⅓ of PHP Fog apps unfortunately had some service disruptions for some apps due to our over-reliance on a single datacenter at Amazon Web Services. We spent the weekend fielding our customers’ concerns, investigating what went wrong, replacing virtual machines, and making sure that our customers’ apps were up and running as soon as AWS came back up.

End users rarely know that AWS is down, and why should they? An end user experiences Netflix as being down when they can’t stream their favorite TV shows. They’re annoyed that they can’t share photos of their recent night out on Instagram. They’re worried that their pins on Pinterest will disappear. And they wonder why their projects on PHP Fog are inaccessible.

It’s a fact of life that data centers can go down for a variety of reasons (natural disasters like this weekend, or sometimes human error too). Four high profile fails in that one region in the last sixteen months, with two outages in the last week.

We are very happy that Amazon has chosen to share a great report on this outage. We all need far more transparency in the Cloud on this issue and moving forward, cloud providers including PHP Fog need more: proactive communication around issues, bi-directional discussions, more detailed information, and far more clarity and openness around architecture and data center design.

All cloud providers share responsibility for the problem in the end, but since PHP Fog is the service you signed up for, it is up to us to communicate directly with you. It’s pretty clear that we are overly dependent on a single infrastructure provider. That’s simply not acceptable.

We are continuing to move as quickly as possible to make our services and products as flexible as possible so our customers have the freedom to not only easily choose infrastructures, but to also easily change infrastructure providers to meet the needs of your businesses.

For those of you that experienced issues via PHP Fog this weekend, thank you for being patient and working with us over the weekend while the immediate issue got resolved. It wasn’t just a long weekend for us, it was a long weekend for many of our customers too. Thank you so much for your understanding and your support.

  • Optimiza

    Why not deploy your PaaS in AWS like they said making it fault tolerant using different AZ’s ?

  • http://twitter.com/cardmagic Lucas Carlson

    We already implement multiple regions, which is why only a fraction of our customers experienced outages. Also, customers who deployed multiple virtual machines did not have outages because only some of their servers failed. However multi-availability zones introduces significant latency which would make it too slow for use with a hosting system like PHP Fog. 

Powered by Olark