Date: July 9, 2026 Duration: ~3 hours (13:01–16:15 UTC) Severity: Minor
A code change intended to fix rate-limit tiering for MCP OAuth clients added a new field to the token payload generated during personal API key creation, but didn't add that field to the corresponding database schema. Because the schema enforces strict validation, every attempt to save a newly generated token was rejected, causing key creation to fail with a generic server error.
For roughly three hours, customers were unable to create new personal API keys from the Buffer API settings page. Anyone who tried received an error when clicking "New Key." Existing API keys and tokens were unaffected and continued to work normally — only new-key creation was broken. 12 customers reported the issue through support.
We reverted the change that introduced the issue and confirmed new-key creation was working again.
The revert took a bit longer to take effect than expected, since merging a fix isn't the same as it being fully deployed — a manual redeploy step was needed to get it live everywhere.
Once live, we verified the fix end-to-end.
A permanent fix followed a few days later: it re-adds the field properly (this time in both the schema and the code) and includes a regression test so the same class of mismatch fails automatically in the future.
Schema and code can drift silently. Adding a field in one place (the code) without adding it in another (the database schema) passed all existing checks and only surfaced as a failure in production. We've added a regression test that would have caught this before merge.
Detection relied on customer reports, not monitoring. We found out about this because customers told us, not because we were alerted automatically. We're working on alerting for this class of failure so we can catch it before customers are impacted.
A merged fix isn't a deployed fix. We lost some extra minutes assuming the revert had taken effect once merged, when the deployment itself hadn't fully rolled out yet. Verifying actual deployment state, not just merge state, is now part of how we handle reverts under pressure.
Acting fast on a strong suspicion paid off. We identified the likely cause quickly and started reverting before root cause was fully confirmed, which limited how long customers were impacted.
