Blinds.com website impacted - slow or no response

Blinds.com Outage 1/12/2025 11:10am

• #gcc-cec-techissues first notice here
• Moe created prod teams call at 11:14am
• BigMike sent out prod notice (sorry app) at 11:22am after fully identifying issue was ONLY on BCOM. tested all GCC sites from homepage to configurator
• David Grothe started restarting servers at 11:21am...timed out due to resources
• tried again at 11:25am...unable to add back to ALB
• tried again at 11:34am...unable to add back to ALB
• BigMike reached out to cloud at 11:32am for Justin and Nag to join call
• Nag got on at 11:42am
• BigMike reached out to Cyber at 11:36am for Krishna to join call
• Krishna got on at 11:42am
• Requested Krishna to investigate the traffic for malicious bots
• Requested Nag to add the missing nodes back to the load balancer and requested for rest of iis nodes to be restarted
• BigMike pulled the Akamai error string (#221.1ff92917.1736702259.2be347e7) to validate that the underlying servers were unavailable
• Nag restarted all nodes and sites were 100% operational
• BigMike sent all clear notice
• Resolution: There is a code issue with the website calling itself in a loop causing traffic congestion

We've now resolved the incident. Thanks for your patience.

We have identified the issue and currently working to restore full access

Summary:

Blinds.com website impacted - slow or no response

Impact

• Incident Started: 11:10am
• Affecting Customers: Yes
• Affecting CEC: Yes
• Other department affected: anyone browsing blinds.com pages
• Ongoing Impact:
  • increased calls to CEC due to bcom not being available
  • cec unable to checkout

Detail:

Currently investigating the issue

ETA:

Unknown ETA

Slack:

#GCC-ProdSupport

Please standby for further updates, and we'll be in touch as soon as we know more.