Incident Notification - All Sites - Intermittent Issues with sites loading

The Pricing API has come back and the queue has processed all the items.

All sites are back to normal.

Cause: Product was updated that was calculating a bad price which was causing a ton of calls to the pricing API to wait the full 60 seconds to time out.

David wrote a PR to ignore the pricing calls when set to offline pricing.

Summary:

Multiple issues are slowing the pricing API which degrades all site performance.

Impact

• Incident Started: 15 minutes ago
• Affecting Customers: Yes - Difficult to load PIP and Add to Cart
• Affecting CEC: Yes - Pricing affects all site and will increase phone traffic
• Ongoing Impact: Updates to product catalog have been paused to remove pressure to pricing catalog. All updates to catalog will start back after pricing API improves.

Detail:

While monitoring the system we noticed degraded performance cause by Pricing API being overloaded. We've paused the Product Catalog Queue and have restarted the pricing server.

ETA:

10 to 15 minutes to clear up Pricing API.

Slack:

#GCC-ProdSupport

Please standby for further updates, and we'll be in touch as soon as we know more.