Date: October 20-21, 2025
Duration: 15 hours, 38 minutes (8:27 AM GMT+1 Oct 20 - 12:06 AM GMT+1 Oct 21)
Severity: Major

On October 20, 2025, Buffer experienced widespread service disruptions due to multiple AWS outages in the us-east-1 region. The incident began with general platform errors and evolved into a complex situation involving failed video processing, publishing delays, and intermittent platform slowness. The issue was compounded by AWS's inability to provision new EC2 instances, limiting our ability to scale resources during the outage.

• Platform-wide errors and degraded performance

• Failed post publishing across all networks

• Users incorrectly received "account deleted" notifications

• Login and account creation failures

• Video processing failures with "problem processing video" errors

• General platform slowness, especially in channel loading and image uploads

• Frontend requiring manual refresh due to unreliable auto-refresh

• Publishing delays and inconsistent performance

• Analytics data gaps during the outage period

Important Note: Failed posts were marked as errors and required manual retry by users. This was an intentional design decision as we cannot assume users still want to publish content after a delay.

The incident was caused by cascading failures in AWS us-east-1 region:

• Primary Cause: AWS experienced failures in their internal network load balancer health check subsystem

• Secondary Impact: This led to failures in multiple AWS services including DynamoDB and MediaConvert

• Compounding Factor: AWS throttled new EC2 instance launches, preventing us from scaling resources

• Additional Complication: An unrelated spike in automated WordPress plugin traffic initially masked the AWS issues

The incident had two distinct phases:

• Phase 1: Initial outage and recovery (8:27 AM - 12:34 PM)

• Phase 2: Service regression requiring additional mitigation (4:50 PM - 12:06 AM)

While we couldn't resolve the underlying AWS issues, we implemented several mitigations:

• Deployment Freeze: Prevented any changes that could worsen the situation

• Security Challenge: Blocked suspicious WordPress plugin traffic that was adding load

• Resource Optimization: Downscaled non-critical workloads to free resources for essential services

• Dependency Removal: Eliminated hard dependency on error reporting service (Bugsnag) that was contributing to failures

• Reduced webhook failure rate from 25% to 3%

• Improved API Gateway latency through manual resource reallocation

• Optimized existing infrastructure when unable to add new capacity

• Early detection through monitoring systems

• Quick identification of third-party cause

• Effective mitigation strategies despite infrastructure constraints

• Clear communication with customers about manual retry requirements

Infrastructure Resilience:

• While multi-region or multi-cloud could prevent such incidents, the scale of this AWS outage affected a significant portion of the internet

• Our current single-region architecture remains a calculated risk

Monitoring and Response:

• The incident escalated gradually, and our systems correctly identified the progression

• Initial WordPress plugin traffic briefly obscured the true cause

We're evaluating:

• Enhanced redundancy strategies within practical cost constraints

• Improved isolation from third-party service dependencies

• Better automatic degradation capabilities during infrastructure provider issues

• Clearer escalation paths for cascading third-party failures

This incident highlighted both the risks and realities of cloud infrastructure dependency. While we cannot prevent AWS outages, we demonstrated effective mitigation strategies and maintained critical services where possible.

Key improvements implemented:

• Removed brittle dependencies that amplified the outage impact

• Improved resource allocation strategies for constrained environments

• Enhanced monitoring for third-party service degradation

We apologize for the extended disruption this caused. While the root cause was beyond our control, we recognize the significant impact on your workflows and are committed to improving our resilience against such events within practical constraints.