Incident Notification - All Sites/Experiences Down

Production Incident Summary – Blinds / Product Setup Outage

Issue Overview

• High latency and “OOPS” errors started around ~2:50 AM ET on Blinds integration (Home Depot + Blinds.com).
• Configurable (“Build & Buy”) products failed, while simple “Add to Cart” items continued to work.
• Full impact escalated to widespread 502 (Bad Gateway) errors across core experiences.
• Business impact:
  • Zero orders during impacted window (previous baseline: ~1–2/hr overnight).
  • Customer-facing errors across multiple product experiences.

Impact Scope

• Not limited to Blinds — all services relying on Product Setup / core APIs were affected.
• Systems impacted included:
  • Product Setup APIs
  • Configuration services
  • NGINX (origin routing layer)
  • Kubernetes-hosted API traffic
• Symptoms:
  • 502 errors
  • Pods failing / restarting
  • Requests timing out
  • No new transactions processing

Key Findings

• No recent application deployments correlated to the outage.
• Traffic to Product Setup APIs dropped to near-zero ~2:30 AM ET.
• Infrastructure health initially appeared “green,” masking the issue.

Root Cause

• Unplanned IAM policy change in AWS:
  • IAM role policies were modified/reapplied during CI/CD experimentation (Terraform/GitHub Actions).
  • This broke authentication/authorization paths between services and load balancers.
  • Impact primarily affected new connections, which explains delayed failure after initial change (~10 PM CT).
  • Result:
  • Services could not register/deregister with load balancers
  • Deployments failed (credential errors)
  • API traffic failed → cascading outage

Mitigation & Recovery Actions

• Immediate actions:
  • Investigated Kubernetes pods, NGINX, and Product Setup services
  • Attempted redeploys (initially failed due to credential issues)
  • Identified load balancer / IAM authorization failures
• Fix:
  • Reverted IAM role/policy changes across all ProdOps nodes
  • Reapplied correct AWS security roles to EKS nodes
  • Redeployed NGINX and core services
• Recovery:
  • Traffic gradually restored
  • Orders resumed (~7:20–7:40 AM range observed)
  • Latency reduced and experiences stabilized

What Worked

• Rapid cross-team collaboration (Infra + App + ProdOps)
• Use of redeploy attempts to surface underlying credential issue
• Node-level rollback and IAM correction resolved system-wide impact

Gaps / Risks Identified

• Monitoring gap:
  • Core API / load balancer health not fully covered by synthetic monitoring
  • System showed “healthy” despite outage conditions
• Change management gap:
  • IAM change applied outside formal change control
• Delayed detection:
  • Failure only surfaced when new connections were required
• Contact/communication friction:
  • Difficulty reaching correct on-call contacts quickly

Follow-Up Actions (Recommended)

• Implement synthetic monitoring for core APIs and LB endpoints
• Add alerts for IAM/auth failures and LB registration errors
• Enforce strict change control for infrastructure (IAM, Terraform)
• Improve ProdOps contact list and escalation paths
• Add post-mortem + guardrails for CI/CD experimentation in prod-linked environments

We've now resolved the incident. Thanks for your patience.

We've confirmed there is a problem, we're working to resolve it. Currently, there is no ETA.

Summary:

502 Bad Gateway is currently displayed for a majority of all experiences

Impact

• Incident Started: 1:50 central
• Affecting Customers: Yes
• Affecting CEC: Yes
• Ongoing Impact:
  • customer/cec is unable to navigate the site and engage with the experiences.

Detail:

[What are we doing to fix it?]

• We are working with Infrastructure team to revert a change

ETA:

we believe this revert will be completed by 6:40am

Please remain attentive to forthcoming updates, as additional information will be disseminated promptly upon availability. We sincerely apologize for any inconvenience this may cause to your work. We acknowledge the importance of your access to our services and appreciate your patience and understanding as we work diligently to resolve the issue.