Issue with Slow Dashboard Loading
On September 23, the Buffer dashboard was slow, and some pages failed to load. Many customers waited around 10 seconds for parts of the page to appear, and some had to reload several times. We’re sorry for the disruption this caused.
Root cause
Busy weekday afternoons had been putting growing pressure on the part of our system that powers the dashboard, and a few features were doing more work than they were designed for. On September 23, afternoon traffic pushed it past the point where it could keep up, and requests started to queue and time out.
Customer impact
From 14:20 UTC, dashboard pages loaded slowly or failed to load. Scheduling posts, managing content and managing channels were unreliable during this time. Performance was back to normal by 15:45 UTC.
Steps to resolution
We traced the slowdown to the system behind the dashboard and added more capacity to it. An unrelated outage at one of the tools we use to make changes slowed this down, so we added the capacity by hand. We then monitored for another 45 minutes to make sure everything was stable before closing the incident.
Key learnings and what we’re changing
Making the dashboard do less work: we’ve made adding a post to your queue much more efficient. That was the biggest source of extra load. We’re also made improvements to other parts of the app.
More room for busy times: we’ve increased the capacity behind the dashboard, so it can handle peak traffic with room to spare.
Spotting slowdowns earlier: We optimized a few more parts that were also under pressure to prevent other incidents. We’re exploring alerts that catch that kind of gradual build-up, not just sudden spikes, so we can act before customers notice.
