Duo Services Update
Last week, we experienced two short outages impacting our DUO1 production server for 46 minutes on Monday, Oct. 20, and 55 minutes on Tuesday, Oct 21. While the service continued to run, authentication was degraded and timing out for a subset of users on one of our deployments. We take this very seriously, and began investigating immediately, even as some of you called in to report authentication issues.
While we were able to restore service quickly, our engineering team investigated the root cause to determine why the outage occurred. When something like this occurs, we want to ensure that we understand why it happened. Recovery was implemented in stages, so the the impact of the outage depended on which applications you protect, and which authentication factors were used. It took a day for us to determine the root cause, which was the result of a bug introduced into user onboarding processes running into database contention with backend cleanup procedures.
We patched the service to eliminate the bug, and we’re also implementing architectural changes to eliminate the possibility that database contention issues could cause an outage in the future.
We appreciate your patience and understanding during these events.
Find Your Deployment ID
We run multiple production server IDs, all of which start with “DUO.” You can find your Deployment ID by logging into the Admin Panel, and looking at the section below the navigation menu on the left-hand side of the screen. Clicking on the Deployment ID will take you to our status page.
For more information on the outage and to subscribe to receive updates related to your specific Duo deployment, please visit https://status.duo.com.
Thank you for using Duo Security.