PHP Fog Blog

Post-mortem on last week’s Git glitch

The problem

Last week, from roughly 2pm PST on Monday until 5 am PST on Tuesday, many PHP Fog users experienced significant disruptions in Git operations, mostly involving permission denials and closed connections. The result: many users had trouble deploying new apps and managing current apps.

Critical files which manage authorization to our Git infrastructure became corrupt due to a bug in recently pushed optimization code.

Looking forward

In order to prevent this scenario from occurring again, we’re implementing some tooling that allows us to more easily verify the integrity of these files, as well as a number of other components of our infrastructure. Additionally, we are instituting a policy of having all optimizations reviewed by a number of peers before implementing them.

We selected Git to be an essential element of using PHP Fog because it’s a wonderful and powerful tool. When our Git service has setbacks of the kind we saw this week, we understand the impact it can have on your development and deployment.

We apologize and give thanks to those who contacted us about the problem and to all of you for your patience and understanding.

Powered by Olark