Buffer Service Disruption

Write-up

publish.buffer.com outage — site failed to load (403 errors on static assets)

Status: Resolved

Root cause

A routine frontend deployment failed to upload the site's static assets, but the deployment pipeline incorrectly reported the deploy as successful. As a result, the live site began serving a page that referenced JavaScript files that had never been uploaded — and the browser received 403 errors for every one of them. Because those assets are what bootstrap the entire app, nothing could load.

The original trigger was a transient failure from an upstream provider during the deploy: a tool the pipeline needed failed to download correctly, returning an error page instead of the expected file. The deploy script didn't stop on that error — it continued, skipped the asset upload entirely, and still finished "green." The new app version then went live referencing assets that didn't exist.

So there were really two failures stacked on top of each other: a transient upstream hiccup (rare but expected to happen occasionally), and — more importantly — a pipeline that didn't treat that hiccup as a failure. The second is the one we own.

Customer impact

publish.buffer.com was fully unavailable to all users for approximately 25 minutes ([start]–[end] UTC).

Because the failure was in the asset-loading layer, there was no graceful degradation — the app couldn't start at all, so login, dashboards, scheduling, channel management, and every other feature went down simultaneously. Backend APIs themselves stayed healthy throughout; the outage was confined to the frontend's inability to load.

Steps to resolution

The failure was spotted and an incident was declared, with engineers from multiple teams joining a call to diagnose.
We confirmed the root cause quickly — the referenced asset files were missing from storage, and the deploy that should have uploaded them had silently failed.
A rollback was initiated, but we recognized that our continuous-deployment setup could overwrite it. In parallel, a fresh build was already moving through the pipeline and uploading the assets correctly, so we let that complete.
The site was confirmed loading again and the incident resolved, all within roughly 25 minutes of detection.
A fix shipped the same day to make the deploy script fail correctly on this class of error and to retry transient upstream download failures with backoff.

Key learnings

This one was preventable, and we own it. Unlike an upstream provider outage, this came down to a gap in our own deployment safety net — specifically, a pipeline that could fail silently and still ship. We've closed the immediate hole and are hardening the rest.

What we've done and are doing:

The deploy pipeline now fails when it should. The script that silently continued past a failed step has been fixed to exit immediately on error, so a failed asset upload can no longer be reported as a successful deploy.
Retry logic for transient upstream failures. The step that broke now retries with backoff, so a single intermittent failure from an external provider won't take down a deploy.
Post-deploy verification. We're adding a check that confirms the newly deployed assets are actually servable before a deploy is considered done — closing the gap between "pipeline says success" and "the site actually loads."
Proactive monitoring. We're adding synthetic checks that verify the live page genuinely loads (not just that the server returns a response), plus alerting on anomalous error spikes for static assets, so we detect this kind of issue ourselves rather than waiting on a customer report.

Latent risk, exposed by a rare trigger. These pipeline gaps had existed quietly for a long time without causing a problem. It took an uncommon upstream failure to surface them — a good reminder that "it's never failed before" isn't the same as "it's safe," and that the blast radius of a silent deploy failure can be a full site outage.

Response was fast and well-coordinated. From detection to root-cause identification was under twenty minutes, and to full resolution about twenty-five. Synchronized troubleshooting on a call plus a clean split between engineering and customer-communications work kept the response tight.

Write-up

Buffer Service Disruption

Partial outage

View the incident

publish.buffer.com outage — site failed to load (403 errors on static assets)

Status: Resolved

Root cause

Customer impact

publish.buffer.com was fully unavailable to all users for approximately 25 minutes ([start]–[end] UTC).

Steps to resolution

The failure was spotted and an incident was declared, with engineers from multiple teams joining a call to diagnose.
We confirmed the root cause quickly — the referenced asset files were missing from storage, and the deploy that should have uploaded them had silently failed.
A rollback was initiated, but we recognized that our continuous-deployment setup could overwrite it. In parallel, a fresh build was already moving through the pipeline and uploading the assets correctly, so we let that complete.
The site was confirmed loading again and the incident resolved, all within roughly 25 minutes of detection.
A fix shipped the same day to make the deploy script fail correctly on this class of error and to retry transient upstream download failures with backoff.

Key learnings

What we've done and are doing:

The deploy pipeline now fails when it should. The script that silently continued past a failed step has been fixed to exit immediately on error, so a failed asset upload can no longer be reported as a successful deploy.
Retry logic for transient upstream failures. The step that broke now retries with backoff, so a single intermittent failure from an external provider won't take down a deploy.
Post-deploy verification. We're adding a check that confirms the newly deployed assets are actually servable before a deploy is considered done — closing the gap between "pipeline says success" and "the site actually loads."
Proactive monitoring. We're adding synthetic checks that verify the live page genuinely loads (not just that the server returns a response), plus alerting on anomalous error spikes for static assets, so we detect this kind of issue ourselves rather than waiting on a customer report.