Skip navigation
Industry Events

DEF CON 24: Mudge & The Consumer Reports for Software Security

DEF CON is one of the oldest and largest hacker conventions around, taking place in Las Vegas, bringing together hackers, lawyers, law enforcement agents, civil libertarians and cryptographers.

I attended a talk by Mudge and Sarah Zatko on Project CITL (Cyber Independent Testing Laboratory). Sidenote: Mudge also gave an interesting Duo Tech Talk in Ann Arbor two years ago called A Behind the Scenes Look at Creating DARPA’s Cyber Analytic Framework.

Mudge had announced he was leaving Google to work on Project CITL, which received a contract for “Consumer Security Reports” from the Air Force, on behalf of DARPA, according to

Mudge and Sarah Zatko at DEF CON 24

Similar to Consumer Reports, formed as a nonprofit to provide unbiased product testing and ratings, Mudge and Sarah Zatko had created a very thorough set of scoring and testing criteria to produce reports on the security of current software.

Which is quite similar to what our security research team, Duo Labs, intended to do with their research earlier this year on OEM (Original Equipment Manufacturer) Updaters and the security of out-of-the-box laptops.

The Zatkos found there’s a problem in the industry - there’s no consumer reports for software security to inform customers on, for example, what the safest browsers are to use.

Part of that problem is the fact that no one has data to back up their feelings and intuition that one type of software is more secure than another.

Certifications and evaluations are manually intensive; industry and marketing labels typically only serve the bare minimum; source code review is difficult due to NDAs (Non-Disclosure Agreements) that work against consumers; and legislation is well-meaning, but often tries to fix a problem by making it illegal to look at it, which is problematic for security researchers.

In the research they presented, the Zatkos measured 20,000 third-party apps on OSX and compared the security of apps per OS plus the difficulty of finding vulnerabilities per browser. The CITL criteria for measuring security risk in software includes a safety continuum, static analysis, measurement of complexity, app armoring (includes compiler, link and loader), and developer hygiene.

The CITL Approach

Mudge compared the list of security criteria of software as analogous to that of security features for a car - it requires seatbelts, an antilock brake system, air bags, etc.

Some of the dynamic analysis components they measure include exploitability, disruptibility and runtime complexity. Depending on your environment, disruptability can be more problematic than exploitability - for example, a system crash at an offshore oil drilling company could be devastating. But a bank may be more concerned with a compromise that affects the integrity of customer/financial data vs. a service disruption.

In a comparison of app armoring of three different browsers - Chrome, Safari and Firefox on OSX. They measured which ones had ASLR (address space layout randomization), heap protection, stack guards and fortified source.

Project CITL - Browser Security

They found that one browser provider had to drop ASLR on OSX for backwards compatibility reasons - which was news to the provider, but did affect a user’s security.

They also compared the different institutions that create code - comparing Google Chrome’s framework to Microsoft Excel, they found that not all Chrome developers are security experts, and may use more risky functions (more so than Microsoft Excel’s team, who had been called out for it). One contributor to this problem is, secure functions are not always part of a typical computer science curriculum.

In one case, they looked a particular DARPA freemium pre-package of Python and big data analytics libraries, called Anaconda. Their client list included Bank of America, LinkedIn, Microsoft, HP, IBM, and many other major names. They lacked most security aspects across Linux, OSX and Windows. Turns out, they had recompiled modern source code using antiquated dev environments - as a result, they had missed nearly a decade of dev environment security updates.

Another analogy is - while nutritional guidelines can give you an estimate of what should be required for a good diet, you might still need a doctor or nutritionist to help you out. A security expert can tell you what kind of diet you should be on for secure software.