Incident Notification - Credit Card Payment System Down

The issue is finally resolved and payments are coming through.

Payment Team's Summary

Executive Summary

On Thursday, October 23, 2025, starting at 6:58 PM ET, a critical failure within the Commerce Hub payment gateway caused widespread authentication errors across all card payment methods, impacting nearly every selling channel globally. Internal systems received 502 Bad Gateway errors with the message ApiKey and/or Authentication supplied are invalid. Failure rates were extremely high, severely impacting order intake and forcing Store POS systems into offline mode. While initial alerts highlighted Apple Pay (specifically 411 Apple Pay calls failed before mitigation), the issue quickly proved to affect all card types. The incident was successfully resolved around 9:05 PM ET after the team directed traffic from the fiserv-proxy directly to Commerce Hub's US Central region endpoint. Fiserv acknowledged the issue. Monitoring confirms stability.

Impact

• Business Impact: A major disruption (~2 hours) to order intake across nearly all channels and regions (US & Canada). High failure rates caused significant lost sales opportunities and operational disruption, including forcing Store POS into offline mode.
• Customer Impact: Customers attempting to pay with any card processed by this gateway experienced a high rate of failures for approximately 2 hours, preventing them from completing purchases across multiple platforms (POS, Online, Order Up). 411 specific Apple Pay attempts failed before the feature was disabled.
• System Impact:
  • High 5xx error rates on Card Broker endpoints due to failures from the upstream fiserv-proxy. The /v2/card-broker/account/verification endpoint saw over 7,300 failures (63%) during the impact.
  • The fiserv-proxy returned ApiKey and/or Authentication supplied are invalid errors, confirmed to originate from Commerce Hub.
  • One Returns processes were also impacted.
  • Both Credit and Debit card transactions failed.

THD Payments team has fixed the issue and we are seeing payments going through.

We are still monitoring the system and following the THD Payments team until this is completely back up.

Payments Team Notice

Incident Description Changed: Impact

Business Impact: A major disruption to order intake across all channels, with 30-40% of all card payment authorizations failing. This affects online sales and poses a significant risk to in-store sales if POS terminals are forced offline.

Customer Impact: Customers attempting to pay with any card processed by this gateway are experiencing a high rate of failures, preventing them from completing purchases.

System Impact: High 5xx error rates on Card Broker endpoints due to failures from the upstream fiserv-proxy. The fiserv-proxy is returning ApiKey and/or Authentication supplied are invalid errors, confirmed to originate from

Commerce Hub. Downstream services like Cart Commons are experiencing failures. In-store POS systems are impacted due to the inability to authorize card payments.

Summary:

Credit Card Payment System Down

Impact

• Incident Started: 7:44pm
• Affecting Customers: Yes
• Affecting CEC: Yes
• Ongoing Impact:
  • unable to checkout with credit card system

Detail:

[History] [What are we doing to fix it]

• We have verified that all cards are being denied
• reaching out to OrangePay support for assistance in triaging
• we are temporarily shutting off OrangePay credit card, use other means of checkout (paypal, offline order)

ETA:

Next update at 8:30pm unless additional information is present

Please stay tuned for updates, as we will share more information as soon as it becomes available. We sincerely apologize for any inconvenience this may cause to your work. We understand the importance of your access to our service and appreciate your patience and understanding as we work diligently to resolve the issue.