Failing open: A lesson in attention to configurations
Duo’s AI and Security Research team takes on cases from customers and beyond, digging into authentication data and finding actionable anomalies that can be searched for, alerted on, or remediated using AI and machine learning.
When chaos strikes, we tend to attribute it to outside causes. In many cybersecurity incidents, however, internal factors come into play— including things like configuration and privilege changes. In this blog, we’ll talk about a recent investigation in which misconfigurations made all the difference.
What are we searching for?
When the AI and Security Research team received a request from a customer to review their authentication data, it appeared unusual and stood out from other recent cases in many ways. First, it was an in-progress, full scale attack that was still being contained when we received notice. Second, it contained an interesting note—that the attacker may have bypassed Duo authentication entirely.
Thankfully, the attack was noticed quickly, and the attackers’ further lateral movement was prevented. Still, the reporters confirmed that the attacker took over a vital server. The request from the reporters was to find out how Duo authentication could have been bypassed and any other pertinent information regarding the attack that we could glean for them.
The facts at hand
With the attack occurring within 3 days of the report, the team could easily sift through the authentication logs. The report also provided an affected machine name, a potentially compromised administrator account username, and one indicator of malicious activity. However, no logs existed for any activity on the affected machine during that timeframe – the customer confirmed that these were erased by the attacker.
We started our search by looking for unusual authentication behavior and errant device registrations. We made a simple query akin to the following pseudocode:
> select * from auth_data where timeframe 01/01/2025 to 01/03/2025 and user =(reportedName) and customer = (customername)
We also looked at device registration data during this period, finding a list of device registrations and removals. Malicious actors commonly either register a new phone of theirs or remove a phone belonging to the legitimate user to gain persistent access to an account.
Below: A set of device modifications surrounding the timeframe of the compromise.
A deluge of devices
Our first query above resulted in some interesting authentications. The first set were authentications that started but were not completed due to authentication timeouts, all within a few minutes of each other.
The administrator account included in the report was authenticating to a set of different Duo Authentication Proxies (Duo servers hosted by the customer), with all authentications originating from internal IP addresses. This indicates that the attacker was either using a VPN to access internal resources (and had therefore already compromised a computer directly), was using a remote access tool to pivot to an internal resource or was an insider threat who already had authorized access.
After looking through the authentication logs, we focused on the devices associated with the user. There were more than 10 devices associated with this administrator account, some of which were shared by other accounts. This type of behavior is quite unusual and raises alarms – administrator accounts and other accounts with escalated privileges should be carefully managed with the absolute minimum number of devices and users accessing them. Many devices existing on one account (and especially those shared between multiple accounts) can mean many methods of initial access or avenues of further access for an attacker.
Breach origination
We found evidence of account sharing and possible internal access. Even so, how did the attacker bypass Duo MFA protections?
After a careful review of the activity logs on the authentication proxy used to access Duo services, we saw that it had briefly lost connection with Duo services around the timeframe that the attacker gained access.
When Duo authentication proxies can’t communicate with Duo services, they can be configured to do something called ‘failing closed’ - until communications are reestablished, they need to find a different route/backup to access Duo services and gain access to resources. In this case, the authentication proxy (a RADIUS server) was configured by the customer to ‘fail open’ instead.
Failing open any application or system can be exceedingly dangerous to data security and integrity. Instead of looking for an alternative and blocking potential authentications from resources, it essentially removes all barriers preventing access to the resources. This can be useful for things that could cause catastrophic failure without access (such as industrial control or medical systems) but is risky to enable where unnecessary.
In this case, the server could have been removed from the network or denied access to the network in some other manner by the attacker. This also could have been related to some RDP authentication attempts to the server around the same timeframe.
Lessons learned
There were several things of concern within this breach that we identified as learning opportunities. The first is the issue of administrator account sharing – this is always dangerous and should be avoided wherever possible. Decreasing the possible traversal pathways of an attacker means less to contain or decommission in the event of a breach and prevents the attacker from gaining higher privileges in other areas.
Finding the server’s access failure (also known as ‘Failmode’) configuration was also a big turning point. In Duo terms, the ‘Failmode’ for a RADIUS server determines what authentications do if Duo services are unreachable. They can either be in ‘safe’ mode (fail open), in which authentications are allowed if the first factor is successful, or ‘secure’ mode, which rejects all authentication attempts if Duo services cannot be contacted.
Misconfigurations are a frequent cause of breaches and can be prevented by comprehensive compliance evaluations and regular configuration reviews on all systems.
Backing up logs was also a piece that could have made this investigation faster to resolve. Attackers frequently cover their tracks by deleting log data, and critical logs can be lost or overwritten quickly. We always recommend more comprehensive log management and forwarding configurations.
By following some simple security principles, you can keep your accounts safe and secure with Duo.
To learn more about secure configuration of your Duo server, check out our knowledgebase topic on configuration best practices. Or peruse all topics on Duo Authentication Proxies.