In many of the discussions about bug bounty programs, the focus is inevitably on how much researchers were paid. Another valuable measure—and perhaps more important—is how the organization handled the reports they received.
Code-sharing software company GitLab marked the first anniversary of its public bug bounty program with an announcement that the company had received 1,378 reports from researchers over the past year. Of the reports received, 346 were for valid vulnerabilities, said James Ritchey, a security manager for application security at GitLab. Just 16 issues were rated as critical, and 57 were rated as high severity. A little less than half, or 171 issues, were rated as low severity.
A little less than a quarter of the valid reports were for “improper access control issues,” Ritchey said, and at 23 percent, they were the most common bug type reported. MITRE defines improper access control on the Common Weakness Enumeration list as not restricting, or incorrectly restricting, access to a resource from an unauthorized actor.
GitLab said in its announcement that the program "kept our engineers on their toes, challenged and surprised our security team, and helped us keep GitLab more secure." The surprise came from the depth of findings and reports that were submitted, Ritchey said.
“We expected a lot of noise with a new, public program, but were pleasantly surprised to see the level of innovative thinking and technique we received,” Ritchey said.
Fixing the Issues
Critical vulnerabilities are tagged as “S1” severity, high as “S2”, and medium as “S3.” The goal is to mitigate an issue tagged as S1 for critical and P1 for highest priority within 24 hours and have a fix ready by—or before—the next security release, Ritchey said. The report is published 30 days after a fix has been released.
At the time the program was launched, GitLab’s former director of security Kathy Wang said the goal was to bring the mean time to mitigation for medium-high security issues to under 60 days, on average. At the time, the MTTR for critical security issues was fewer than 30 days.
The original goal for high-severity issues was 60 days. The MTTR for high-severity issues is currently 53 days, Ritchey said.
“Sometimes there are situations where easy or simple bug fixes are rolled out faster,” Ritchey said.
Triage is Hard
The fact that less than four percent of the reports were for valid vulnerabilities highlights the challenges for organizations with public bug bounty programs. Someone has to sift through the influx of reports and figure out which ones are in scope and not duplicates (reported previously). In-scope in GitLab's case refers to the GitLab product used for code-sharing and a few other components on the platform, but excludes areas such as the shop, forum, status, and support pages.
HackerOne runs GitLab’s program, so the initial triage happens on that platform. “Reports which clearly have a critical impact” are handled first, and the others are according to the order they were submitted, Ritchey said. This way, the team can identify duplicates. Impact refers to the amount of affected assets multiplied against the sensitivity level.
GitLab’s methods for determining the severity of a bug is “constantly evolving,” Ritchey said. Along with considering the vulnerability’s classification level, difficulty to exploit, GitLab considers the quantity of the affected data, the possible attack vector, the level of privileges required, and whether the attack would require user interaction. If there is a working proof-of-concept, that would potentially impact the assessment, as well.
“We often use an internal council to help make a final determination on severity if it’s in a gray area,” Ritchey said.
In a “lessons learned so far” post six months after the program was launched, GitLab’s Ethan Strike wrote about an internally developed Slack command which imported triaged reports from HackerOne and into GitLab issues so that the correct team and product managers could be assigned to work on the fix.
Ritchey also said that not having someone dedicated to triage was a problem. To address this, the team implemented a rotation schedule to make sure someone was always assigned to handle triage. The rotation reduced burnout within the team, as well, since the responsibility was spread out over the team over time.
Automation Helps
Communication is also key in a public bounty program, and one of the things that organizations frequently struggle with. GitLab automated a lot of the tasks to be able to handle the volume of reports. An automated bot provides the initial response to the researchers who submitted the report. That initial message contains an estimate on how long it will take the team to triage the report. Another bot follows up with a message providing the expected date for the fix.
“We needed a way to make it scale since there were many reports and reporters and only a handful of us on the GitLab side,” Ritchey said.
Image Credit: NESA by Makers on Unsplash