Amazon packages pile up as AWS outage spawns delivery havoc

An Amazon Web Services outage wreaked havoc on the e-commerce giant’s delivery operation, preventing drivers from getting routes or packages and shutting down communication between Amazon and the thousands of drivers it relies on, according to four people familiar with the situation.

Three delivery service partners said an Amazon.com Inc. app used to communicate with delivery drivers was down as part of Tuesday’s system failure. Vans that were supposed to be on the road delivering packages sat idle with no communication from the company, the person said. Amazon Flex drivers, independent delivery people who carry parcels in their own cars, couldn’t log into Amazon’s app to get assignments, said another person.

Amazon said late Tuesday afternoon that much of its web service was working again. “We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired,” the company said on its AWS dashboard just before 6 p.m. Eastern time. “We are seeing improvement in availability across most AWS services.” The company didn’t comment on the issues with its delivery operation.

The problems came during Amazon’s critical holiday shopping season when the company can ill afford delays that could potentially create lasting logjams. One delivery business owner on the West Coast said the company halted deliveries on Tuesday and planned to regroup on Wednesday.

Two delivery partners in earlier time zones said drivers that already had routes were instructed to put their phones in airplane mode and not log out of the Amazon routing app so they could continue making stops, but drivers who hadn’t already been assigned routes were sidelined.

The outage began at about 10 a.m. Eastern time, according to Downdetector. At the height of the outage, the web monitoring site reported more than 20,000 complaints for Amazon and more than 11,000 for the company’s cloud computing arm, Amazon Web Services. By 1:45 p.m., the reported outages had declined by about half for AWS and two-thirds for Amazon.

Multiple popular websites were also affected, including those operated by Coinbase Global Inc., Robinhood Markets Inc., Walt Disney Co. and Netflix Inc., according to Downdetector. Disney said that though people were able to get into its parks, they were having difficulty checking in online and paying for purchases. Webcast presentations from Comcast Corp. and Altice USA at UBS’s Global TMT Conference experienced disruptions earlier Tuesday, and the Charter Communications Inc. presentation was rescheduled for Wednesday.

Some Amazon services, including music and video streaming, the voice-activated Alexa platform and security arm, Ring, were affected, too.

Just after 1 p.m., Amazon said it had identified the root cause of the issue and was working to fix it. Meanwhile, the company directed customers to alternative servers in its western region that weren’t experiencing problems. The increased errors were in the eastern North American region.

Amazon, in an update on its AWS dashboard, said it still hadn’t completely fixed the issue by 6 p.m., although “many services have already recovered.” Multiple Amazon cloud-computing services had been affected, including Amazon DynamoDB and Amazon Elastic Compute.

Video streaming service Netflix experienced a 26% drop in traffic after the AWS problems were reported, showing how quickly outages can ripple outward, said Doug Madory, an analyst at the network monitoring firm Kentik in San Francisco.”It gets more and more complicated with software running these services, so when something goes sideways it can take a long time to figure out what went wrong and fix it,” he said. “Complexity has risks. You introduce unknown errors.”

AWS is the leading cloud-computing provider, selling companies computing power and software services on demand rather than maintaining their own data centers and teams in-house. Its customers include a wide range of industries and the federal government.

©2021 Bloomberg L.P.
Distributed by Tribune Content Agency, LLC.