Status: Resolved
Root cause
A dependency change merged into our monorepo unexpectedly rewrote the shared lockfile, shifting how dependencies resolved across the whole codebase. Though the change targeted one backend service, the ripple broke the frontend bundle for Publish, causing a client-side error that stopped the app from loading. Backend services stayed healthy throughout, confirming this was a frontend-only issue.
Recovery was slowed by a secondary gap: the lockfile-only revert didn't automatically trigger a new frontend build, so a first deploy attempt was a no-op and a second deploy was needed to ship the fix.
Customer impact
Publish was completely unavailable for approximately 25 minutes, the app was stuck on a loading screen and couldn't start, so users couldn't create, edit, schedule, or manage content. This was the second Publish outage that day, following an earlier, unrelated incident. Backend data was unaffected and no content was lost.
Steps to resolution
Incident declared; responders coordinated on a call.
Symptom quickly narrowed to a frontend runtime error, ruling out backend involvement.
The problematic change was identified and a revert prepared and merged.
First deploy was a no-op (the revert didn't trigger a rebuild); a second deploy shipped the fix.
Recovery confirmed roughly 25 minutes after declaration, with customer comms running in parallel.
Key learnings
Single-service dependency changes can have monorepo-wide blast radius. A bump scoped to one service rewrote the shared lockfile and altered resolution for unrelated code, including the frontend.
Large lockfile diffs deserve scrutiny. A small intent that produces thousands of lines of lockfile churn is a signal to review and test more closely before merging.
A revert isn't live until the pipeline rebuilds. We're closing the gap where certain change types don't trigger the expected builds, so a fix doesn't need a second manual deploy mid-incident.
