A Game of Phones: Fighting Phone Phreaks in the 21st Century
It is 11pm on an average Saturday night. The kids are in bed and I am on the couch not looking at the TV. My phone pings out the Slack notification sound. I know before looking at what the alert will say. There is another flood of requests to our service by someone profiting from toll fraud.
Before I started working at Duo in 2014, I had never heard of toll fraud. By the summer of 2019, it would consume my Saturday nights and nearly every other night along with it.
What Is Toll Fraud?
Telephony is a unique business. I initiate a call on a device that I carry in my pocket, and in seconds I can speak to someone on the other side of the world. Underneath that interaction is a series of connections and relationships that enable the call. The call travels over routes owned by a series of different companies through many different jurisdictions. This works because they have agreed to a system of billing that will ensure that if they make the connection happen that they will get paid for it. It is that agreement that enables connections to be so reliable to make, but it also enables toll fraud.
International Revenue Sharing Fraud (IRSF)
You may have never heard of it, but it generates more than $5 billion per year in fraudulent charges to people and services around the world.
The scammers that hit Duo practice a specific type of toll fraud known as International Revenue Sharing Fraud (IRSF). You may have never heard of it, but it generates more than $5 billion per year in fraudulent charges to people and services around the world.
IRSF starts with a carrier that bills a premium for a set of numbers (similar to 900 numbers here in the US), and is willing to pay a bounty to those who can generate calls to those premium numbers. The numbers might be real numbers that are hijacked by a transit carrier for this purpose, or unassigned numbers the carriers somehow gain control of. The bounty hunters use automation to generate calls to these numbers. A common attack is to use compromised private telephone networks to generate calls, but it is sometimes easier to abuse services, like Duo, that makes calls on behalf of their customers.
Why Duo?
Our product enables a customer to use phone calls as a second factor for authentication as an alternative to our recommended method, Duo Push. That feature enables our customers to automate phone calls to numbers they set via user enrollment. On top of that, free signups have been a cornerstone of Duo's approach since its creation. We have always wanted to be the easiest solution for customers to secure their systems. Having to listen to a sales pitch just to see a demo or create a proof-of-concept is not easy.
We want system administrators to be able to get our product up and running in 15 minutes. Providing free use of our product for 30 days without having to interact with anyone provided that speed. It was exactly this ease that created an opportunity for scammers to exploit our service, because along with that easy signup came 500 free telephony credits that scammers could happily consume by generating calls to their bounty numbers.
What To Do About It?
When the attacks started, the very first thing Duo engineers tried to do was to block the IPs that were making the requests. Within 24 hours the attackers had rented a virtual private server inside the US and were back at it. There began a back-and-forth game in which attackers would find a way around our blocks, and we would implement new ones.
A real life whack-a-mole game that cost Duo time and effort to play.
Duo engineers soon added some rudimentary number verification code that would ban you if the verification failed. After that, engineers implemented simple prefix blocks that could be quickly, but manually, added to the system that would prevent calls to numbers with prefixes on the list. That worked well enough for a while. Scammers would show up every few months, a Duo engineer would block the prefixes they were using, and the scammers would go away. Rinse. Repeat.
The Impact
Eventually, the attacks increased in frequency. We knew that someone had released a 'How to' guide for these attacks and more scammers got in on it. The costs of the attacks began to rise, both in terms of actual dollars and the distraction to our team from the effort to combat them.
In addition to the cost in dollars and effort to combat, if you have a service that enables this kind of activity, you are at risk of getting blocked by all the service providers in the chain. Telephony providers have a vested interest in preventing this type of fraud. I would routinely wake up to emails from our telephony provider's fraud department indicating a range of numbers they had blocked. Worse yet, they would warn that our access to their service could be cut off if we allow this behavior to continue.
It wasn't just our provider either, they were really just passing along the ire they received from downstream carriers who were also under pressure to eliminate fraud.
The risk of customers being affected by our efforts, or our provider's efforts to block fraudulent activity drove us to seek new solutions. Our customers rely on us to protect their assets and users, and if we are preventing them from authenticating, we’re not achieving our mission.
Even beyond the potential for an important part of our service to be made unavailable due to blocks, there is a productivity cost. It took time to analyze the calls coming in and block prefixes while making sure we keep the service working for our legitimate customers. We had to spend time on that instead of building better features, improving infrastructure, or taking vacations. In some cases, we fought scammers while on our vacations.
Evolving
Eventually, the issue came to a head. Duo's Site Reliability Engineering (SRE) team had been responsible for blocking the scammers and preventing fraud. We wanted the SRE team to focus on the stability and scalability of our service, and pushing back on scammers was a distraction.Fraud loss continued to increase. Management and Product decided to charge the Platform team with solving this problem. That is my team.
The Platform Team
We spent the first weeks taking over the manual efforts to block spammers from the SRE team. We would get some alerts that said something like "Lots of calls to number 313-555-5555" and we would go in and examine the customer's account, figure out all the numbers they are calling, and add them to the prefix ban list. The whole process took only a few minutes, but it was relentless. Scammers would start up at all hours and then go away for a while after we blocked their numbers, only to come back hours later with more numbers. The prefix blocks had been successful when phone numbers were a scarce resource, but they clearly no longer were.
We had to find a way to block scammers without impacting the real customers who wanted to try out our service. False positives had to be prevented.
Most of the premium numbers are in a certain set of countries, often in the Middle East, South America, the Caribbean, or Eastern Europe. So we blocked calls to every country outside our approved sales list for trial accounts. This frustrated them enough that they started contacting our technical support team to try to get the blocks lifted. Eventually they were able to switch to numbers in countries we didn't block. Next up we rate limited new signups aggressively, eliminating their ability to make a lot of calls quickly.
One Good Scam Deserves Another
Our efforts seemed to pay off. We had addressed limiting free accounts and trial signups, and fraud declined for a few months. Then came another pivot. The scammers began signing up for paid Duo accounts. Our systems detected paying “customers” were making high volumes of calls to premium prefixes.
The scammers were using stolen credit cards to pay for accounts and purchase large amounts of telephony credits. It was committing fraud to pay for more fraud. The controls we put in place had to be refactored for "paying" customers.
Before we had a chance to roll out those controls, they pumped up fraud earnings. Beyond just preventing the scam calls, we now had to find a way to detect fraudulent credit card transactions.
This pivot caused more headache for us. Victims of credit card fraud who had no idea who we were would see charges from us on their statements. Our business operations team had to get involved to reverse the charges and coordinate with our merchant accounts to keep us from losing the ability to accept credit card payments.
The new version of the scam was pretty simple. Scammers would buy a subscription from us, then start ordering telephony credits in increasing batches. Once the card they were using was declined, they would try another one. In one instance, a scammer tried to charge $100,000 worth of credits to a stolen card.
When we detected the fraud and blocked their account, they even contacted our technical support team to demand we unblock them or refund the charges.
Fake credit card purchases were actually pretty easy for a human to detect. The billing address would be some made up sounding street in New York, while the IP of the browser would be on the other side of the globe. The problem was detecting and blocking them fast enough to prevent them from making calls. There is no shortage of third-party services to detect credit card fraud, but the ones we tested either didn't meet our technical requirements, or hit too many false positives. We partnered with our merchant account to ratchet up their fraud controls on our account, which slowed them down a bit, but stolen credit card numbers aren't a scarce resource either, so we needed to find a better solution.
The Finale
One of the things we were always very cautious about was avoiding making the scammers take our efforts personally. We didn't want to provoke them to begin attacking our service in other ways out of spite. They never did. This kind of attack was a part of the wider trend of attacks motivated by money. It was just a business to them, and the solution was to just drive the cost of doing it so high that they would go try somewhere else.
It turned out that the thing that cost the most to them was the time and effort it takes to call us on the phone. Even the few times they had opened support escalations to complain, they preferred to use text chat. So we disabled the use of phone calls for authentication entirely for new customers.
No phone calls allowed, and the only way to get that feature enabled was to, well, call us on the phone and ask.
Only a few of the scammers are brave enough to try it, and our support engineers vet them out easily. We found that it didn't really slow down sign-ups for real customers, and the ones who need VoIP authentication don't mind asking for it.
After that change, our monthly toll fraud bills fell to less than what I pay for coffee. In January of 2020, they cost us a total of $7.00. At least for now, we won. We made it hard enough to attack our service that they have gone elsewhere to find easier targets.
I don't assume that the story is over. They could pivot and find some other way to get at us. When they do, I am confident that we can evolve to meet that challenge when it arrives. For now, I'm back to working on our platform, and using my Saturday nights to catch up on TV shows.
Try Duo For Free
With our free 30-day trial see how easy it is to get started with Duo and secure your workforce, from anywhere and on any device.