Security news that informs and inspires

Taking Hype Out of Bug Bounty Programs


The hype around the bug bounty markets pushed the idea that anyone can have a bounty program, that anyone concerned about security should open up their platform and watch the vulnerability reports roll in.

The reality isn’t so rosy, said Katie Moussouris, founder of Luta Security and an expert in developing vulnerability disclosure programs. Bug bounty programs have become shorthand for vulnerability management, as organizations replace security due diligence with “trendy bug bounties,” she said. Bounties aren't for the low-hanging fruit—organizations should continue to run their own assessments to find and fix flaws. The bounties are for high-quality submissions, but that isn’t the case for many organizations.

“Everything you thought you knew about bounties is wrong,” Moussouris said.

The programs are targeting the wrong types of flaws—a scoping problem. If the organization doesn’t have the time to triage, investigate, and fix low-level vulnerabilities, then they are going to be inundated with basic reports. If they want higher-class of vulnerabilities, the hard-to-find ones, they need to have those parameters in place to make it clear the hunters shouldn’t waste time reporting basic flaws.

Moussouris cited an example from earlier this year, of a top hunter on the HackerOne platform who received a total of $119,650 across 15 reports in one day. While the bounties ranged from $250 to $18,000, the median bounty amount was $10,000. The hunter said the various reports reflected about four hours of work—and the issues found were all SQL injection and server site request forgeries. The company had paid essentially this hunter $29,000 an hour to find low-level issues.

These aren’t new bug classes the hunter uncovered, and are issues the organization could have found internally at a far lower cost by investing in internal efforts, Moussouris said. She doesn’t begrudge the hunter making money—but said the situation doesn’t make sense from the perspective of the company with the bounty program.

“Incredibly terrible ROI for the org paying this much for 4 hours of his professional time,” Moussouris noted on Twitter at the time.

The original vision for bug bounties wasn’t complicated: Convince organizations to invite outsiders to find and report security vulnerabilities to help the organization become more secure, and pay them for helping. Some of that has happened, with more organizations launching bug bounty programs and hunters being able to join different programs. The Department of Defense launched bounties for different branches of the armed forces, and Congress voted to require the Department of Homeland Security to have its own program.

What hasn’t come to pass is better security. Organizations often launch a program without laying the groundwork for what kind of vulnerabilities are in scope, or deciding how reports will be managed. Currently, 94 percent of Global 200 companies don’t publish an email address or provide some kind of contact information for someone to report vulnerabilities. If they don’t have a process of handling incoming reports, they aren’t going to be able to handle the influx of reports through the program, Moussouris said.

In practice, the bug bounty market isn’t all that different from other gig economies as it is exploits the workforce, Moussouris said. It is very easy to violate labor laws in a gig economy. As contractors, the hunters are asked to work in roles that can be—and used to be—handled by full-time employees. Penetration testing brings in contractors to find vulnerabilities that can be exploited in order to accomplish specific tasks within a defined timeframe. The organization uses non-disclosure agreements and contracts to control the volume and what gets fixed. There is a clear understanding of what the testers are paid for and what they can or cannot do.

Bug bounty programs work if the organization can fix the bugs that are being reported. More importantly, if the hunters are doing the work but they aren’t getting paid—because it is a duplicate bug and someone else has already claimed the payment, for example, that is a a problem.

During the “Hack the Pentagon” program, 1,410 participants registered and 1,189 reports were received. Only 138 were valid reports. More deliberate planning went into “Hack the Army,” and the program fared better: 371 eligible participants, 416 reports received, and 118 valid reports.

The signal to noise was "way out of whack" because of duplicates, Moussouris said.

The bug hunters are fed the idea that anyone could hop into these programs and get paid, and potentially make a living. However, the money goes to a select skilled hunters, leaving everyone else to fight over a small share of the payments.

Moussouris worked with MIT researchers Ryan Elis, Keman Huant, Michael Siegel, and James Houghton to analyze 61 HackerOne bounty programs over a two-year period and found that just 7 percent of participants with ten or more bugs were paid for 1,622 bounties. The other 93 percent earned 2,523 bounties, which is a lopsided model.

"We have deviated so far from the vision of what we hoped bug bounties could become - a way to grow the ranks of security skilled labor,” Moussouris said.

Even HackerOne’s own figures bear this out. Over 300,000 individuals have signed up on the platform and one in ten have found something to report, the company said earlier this year. Of those who have file a report, a little over a quarter received a bounty. A thousand or so hunters have earned $5,000 or more, while about 100 have earned $100,000. The math works out to 0.03 percent of the hunters on the platform made $100,000 or more—and 0.3 percent made $5,000 or more.

That's 97.5 percent of hunters on the platform who never sold a bug, Moussouris said.

The idea that bug bounties provide “continuous coverage” because someone is always poking at the system looking for vulnerabilities “is predicated on a bottomless skilled labor market,” Moussouris said. That kind of labor market doesn’t exist, and more importantly, there wasn’t a concerted effort to develop that kind of skilled labor.

We need to “hack the labor market,” and change how bug bounties are run, Moussouris said. “Bounty smarter, not harder” will help organizations actually improve their security while fairly compensating individual bug hunters.