Incident Notification - Intermittent timeouts on Design Builder - All Categories

There is a product in millwork with a lot of choices. There is over 100 calls when this product loads in the configurator.

Regarding yesterday's Design Builder outage, it occurred when we rolled out the Design Builder to include Millwork products in new markets.

What caused the problem?

• Design Builder works by sending *requests (little messages) from the user's computer back to our main system. This happens both when the tool first loads and whenever a user interacts with it (e.g., selecting a choice).
• Millwork products have many more choices and options than the products we previously offered (like blinds).
• Because of all these extra choices, the system started sending a huge, unexpected volume of rapid requests to our main servers.
• Our system's security (called Akamai) saw this massive number of rapid requests and automatically assumed we were under a malicious attack (a denial-of-service attack, or DoS).
• To protect the system, the firewall automatically blocked the internet connection (the IP addresses) of the stores using Design Builder, which caused the tool to stop working for those locations.

The temporary solution was to exclude the stores' IPs from security policy evaluation. However, we need to redesign our configurator to permanently resolve the issue.

Update:

We are still investigating the cause of the intermittent issue in Design Builder. Currently, the issue appears to be related to a network security policy that blocks calls from Design Builder due to a high volume of requests to the servers. We have escalated the matter to our infrastructure team for further assistance, and we will provide an update as soon as more information becomes available.

Impact

• Incident Started: approximately 10:50 am • Impact on Online Customers: No • Impact on Stores: Yes

We sincerely apologize for any inconvenience this may cause to your work. We understand how important access to our service is and appreciate your patience and understanding as we work diligently to resolve the issue.

Summary: Randomly Design Builder pages won't load. In-Store associates are having issues with Design Builder loading pages. This is affecting all categories.

Impact • Incident Started: Around noon • Affecting Online Customers: No • Affecting Stores: Yes

Please stay tuned for updates, as we will share more information as soon as it becomes available. We sincerely apologize for any inconvenience this may cause to your work. We understand the importance of your access to our service and appreciate your patience and understanding as we work diligently to resolve the issue.