Before Black Friday, Check the Integrations Nobody Is Watching
Most of the peak planning we hear about from retail teams, and most of what we have written about peak this year, is about capacity. Which carriers will take the volume, what the cut-off dates are, how many people will be in the warehouse. All of that matters, but there is a quieter risk that tends to get less attention, and it sits in the connections between systems rather than in any one of them.
In most eCommerce operations an order is created in the storefront, passed to an order management system, sent to a warehouse or a store to be picked, booked with a carrier, and then tracked until it is delivered. Almost every one of those steps is a handoff between two systems, usually through a webhook or a connector. When one of those handoffs stops working, it often does not produce an error that anyone sees. Orders simply do not arrive in the next system, or tracking updates stop coming back, and the operation carries on looking normal from the inside. In a lot of cases the first sign is a customer asking where their order is.
Black Friday is on 27 November this year, a little over five weeks away, so this seemed like a good moment to go through how these failures usually happen and what we would check before the volume arrives.
Why it matters more in November
The volume is the obvious reason. Adobe forecasts US online sales of $275.1 billion between 1 November and 31 December 2026, with $47.5 billion of that in Cyber Week alone and more than $15 billion on Cyber Monday, according to its 2026 holiday forecast. A connection that stops for a few hours on an ordinary Tuesday in March is an annoyance. The same few hours during Cyber Week means many more orders sitting somewhere they should not be, and many more parcels whose customers are not being told anything.
The less obvious reason is that peak is also when a lot of things change. Teams add promotions, apps and sometimes new carriers, rotate credentials, and push last changes before a code freeze, and each of those is a chance for a connection to break in a way nobody notices straight away.
How integrations fail without anyone noticing
From what we see, quiet integration failures tend to come from a handful of places.
Retries run out. Most platforms retry a failed webhook delivery for a while and then stop. Shopify's developer documentation, for example, says Shopify retries a failed webhook up to eight times in a four-hour period, and that if failures persist the subscription is removed, after which no deliveries come through until it is recreated. The same documentation says plainly that webhook delivery is not always guaranteed, and recommends reconciliation jobs that periodically fetch the data to catch anything missed. Our own webhooks work in a similar way. Carriyo retries a failed delivery on a fixed schedule, and once the retries are used up the event waits in the Integration Monitor until someone re-sends it. None of this is a flaw in any one platform. It is how webhooks work, and it means a receiving system that is down for an afternoon can quietly miss events unless someone is watching for it.
Credentials and endpoints change. An API key expires, a secret is rotated on one side and not the other, or an endpoint moves during a site rebuild. Sometimes the change comes from the other side entirely. USPS retired its Web Tools API platform on 25 January 2026, and its developer page says service disruptions are underway, so any integration that was not moved to the newer USPS APIs may already have stopped working, or could stop at any time.
Shared infrastructure goes down. On 19 and 20 October 2025, a year ago this week, a problem in the DNS management system for Amazon DynamoDB caused a disruption in AWS's US-East-1 region that lasted from 11:48 PM PDT on 19 October to 2:20 PM PDT on 20 October, according to AWS's own post-event summary. Outages like that do not only stop the systems that are directly affected. They also leave a backlog of failed requests and webhooks behind them, and once everything is back up somebody has to work out what did not get through.
The request succeeds, but nothing happens. This one is harder to spot. A system returns a success response and then drops the event, because it did not recognise the order reference or it hit a case it was not written for. Nothing looks broken from the sending side, so the only way to catch it is to compare what was sent with what actually landed.
Who finds out first
In a lot of organisations the answer is still the customer. New Relic's 2026 Observability Forecast, which surveyed 2,575 IT and engineering leaders and practitioners across 24 countries, found that 42% of organisations still learn about disruptions through inefficient channels such as manual checks and customer complaints. That survey covers IT in general rather than retail, but in delivery the pattern is easy to recognise, because the customer complaint usually arrives as a "where is my order" message.
Customers have quite clear expectations here. Sendcloud's E-commerce Delivery Compass 2025, a survey of 1,000 UK consumers, found that 85.4% expect to be told immediately if there is a delay with their order, 45% had problems tracking their most recent order, and 64.3% said they would not order again from a retailer after a failed delivery. When a tracking feed has quietly stopped, the tracking page shows nothing new and the delay notifications never go out, so from the customer's side it looks as though the brand has lost track of the parcel, and if the feed really has stopped, that is not far from what has happened until someone notices.
What we would check in the next five weeks
None of this needs a big project, and most of it is a few days of work for someone who knows the stack. If we were going into peak with a few weeks to spare, these are the things we would want to have looked at.
- A list of every connection that moves an order or a status. That means storefront to order management, order management to the warehouse and stores, bookings with each carrier, tracking coming back, customer notifications and returns. For each one, it helps to write down who owns it and who gets the call if it stops at 2 AM on the Saturday after Black Friday.
- Alerts on failure rather than on complaints. For each connection, decide what failing actually looks like and make sure something tells a person when it happens. It is worth including the absence of activity as well, because orders per hour dropping to zero in the middle of a promotion is a failure even if no error was logged anywhere.
- Alerts that reach someone at a sensible pace. An alert email sent every hour to a shared inbox that nobody reads over a weekend does not help much, and neither does one that is so frequent people start filtering it out. Deciding who receives what, and how often, is part of the work.
- Credentials, certificates and API versions that expire before January. This is usually a short list, and it is a lot easier to deal with now than in the middle of Cyber Week.
- A way to replay what was missed. When a connection comes back, the events that failed in between need to be re-sent or reconciled, and the systems receiving them should be able to cope with the same event arriving twice. It is worth practising this once before peak, so the first time is not on the busiest day of the year.
- A date after which integration changes stop. Most teams already have a code freeze for the storefront, and it usually makes sense to include connection settings, webhook endpoints and credentials in it too.
Where we are going with this
On Friday we will write about what we have added to Carriyo to help with the second and third points on that list, so that a failing webhook or connector raises an alert in Carriyo rather than waiting for someone to notice. The list above applies whatever platform you are on, though, and we would honestly suggest working through it either way before the end of October.
Carriyo is The Intelligent Commerce Platform, from checkout to doorstep.
---
Sources
1. Adobe. "Adobe: U.S. Holiday Shopping Season to Hit Record $275.1 Billion Online, Rising 6.7% YoY" (press release, 28 September 2026). https://news.adobe.com/news/downloads/pdfs/2026/09/adi-holiday-forecast-release-9-26.pdf 2. Shopify. "Troubleshoot webhooks" (Shopify developer documentation). https://shopify.dev/docs/apps/build/webhooks/troubleshoot 3. Shopify. "About webhooks" (Shopify developer documentation). https://shopify.dev/docs/apps/build/webhooks 4. Carriyo. "Retry and idempotency" (Carriyo webhooks documentation). https://carriyo.com/docs/platform/webhooks/retry-and-idempotency/ 5. USPS. "Web Tools APIs Retired January 25, 2026" (USPS developer page). https://www.usps.com/business/web-tools-apis/welcome.htm 6. Amazon Web Services. "Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region" (post-event summary, October 2025). https://aws.amazon.com/message/101925/ 7. New Relic. "New Relic Report Finds High-Impact IT Outages Drain $74 Million Annually from Businesses" (press release on the 2026 Observability Forecast, 22 September 2026). https://newrelic.com/press-release/20260922 8. Post&Parcel. "Sendcloud: Delivery issues affect more than half of all online shoppers" (19 June 2025), reporting Sendcloud's E-commerce Delivery Compass 2025. https://postandparcel.info/159449/news/e-commerce/sendcloud-delivery-issues-affect-more-than-half-of-all-online-shoppers/