On September 22, Buffer didn't load for many customers. The Publishing, Insights and Community pages couldn't open, so customers were unable to use the app. We're sorry for the disruption this caused.
Root cause
We were making improvements to team permissions, and several related updates went out close together. Each update passed our checks on its own. The problem only appeared once they were combined, and our release process wasn't set up to test that combination before it reached customers. When combined, the updates stopped account details from loading.
Customer impact
From about 14:41 UTC, many customers couldn't load the Publishing, Insights or Community pages. Most customers were back up and running by 15:11 UTC. Some customers weren't affected at any point.
Steps to resolution
As soon as the first reports came in, the team traced the problem to the permissions updates and reversed them. The fix reached our servers in stages, so some customers had access back earlier than others. Once every server was updated, we monitored for another half hour to make sure everything was stable before closing the incident.
Key learnings and what we're changing
Testing related updates as a group: when several updates depend on each other, we'll test them together before release, so conflicts between them are caught early.
Holding back releases automatically: we're changing our release process so new releases are paused whenever our automated checks find a problem.
Limiting impact and recovering faster: we're exploring releasing updates to a small group of customers first, with an automatic rollback if something goes wrong.
