The security of IoT devices has been a running joke for many years, so much so that some researchers have given up trying to point out the weaknesses and get vendors to address the problems. Some vendors have pledged to do better and improve their development practices, but a year-long analysis of the security features in the firmware of 22 IoT device manufacturers found that not only are the vendors not making progress on security, they’re actually going backward.
The study is the work of the Cyber Independent Testing Lab, a small non-profit that performs rigorous testing of the security features and properties of a variety of products and platforms. The team wanted to see how IoT vendors were faring in adding standard hardening features to their firmware binaries, so it developed a special methodology that began with downloading available firmware updates from vendor websites, extracting Linux filesystems from the firmware, and then running each binary through the CITL’s custom analytic tools. The dataset comprises more than 3.3 million individual binaries from nearly 5,000 firmware updates from 22 vendors, including ASUS, D-Link, Belkin, QNAP, and Mikrotik, and goes back as far as 2003.
What the team found is dispiriting, if not surprising: IoT firmware hardening is getting worse rather than better. Firmware updates are more likely to remove binary hardening features than to add them, and overall there hasn’t been any trend in a positive direction for security in the 15 years covered by the CITL dataset.
“The best possibility I can theorize is that a team pulls in a new library that’s helpful with some feature and finds that it won’t build against a binary that has some additional hardening turned on, so someone removes one of the hardening flags and then it works fine so they go with it,” said Parker Thompson, lead engineer at CITL.
“Security testing for binary hardening flags hasn’t reached the state of being a regression test yet.”
The CITL study looked for the presence of a number of possible hardening techniques, such as ASLR, non-executable stacks, and stack guards. These technologies are used to mitigate the effects of certain vulnerabilities and have been in wide use in the desktop and server worlds for many years. They have begun to make their way into IoT device firmware in the last few years, but the CITL data shows that updates often remove one or more of the hardening flags and some updates significantly reduce the overall security of the firmware. For example, one update shipped in 2017 by Ubiquiti for its UAP-HD line of wireless access points removed ASLR altogether and the presence of stack guards went from about 70 percent of binaries to virtually none.
One potential reason for the backward movement could be the size of the development teams for IoT products and the rapid pace of the market’s evolution and expansion.
“If you look at a Windows build through this same lens, it looks completely different. It would have virtually complete coverage for the hardening flags. I’d posit that these are much smaller teams that have much less continuity from one project to the next,” Thompson said.
“They’re rapidly responding to a low-cost, high-volume environment. The hints that I’ve found in the filesystems suggests that these are being developed all over the world with a great amount of variation among the teams.”
“We want to bring all of these common points to light and back it up with our data."
Although IoT devices often are associated with consumer applications, a tremendous amount of IoT gear finds its way into enterprise environments, as well, whether it’s through official purchases or shadow installations by employees. Many of the firmware images the CITL study looked at are from networking devices, which are vital to enterprises and therefore quite valuable for attackers.
“We found major regressions in access points you would ship to enterprises by the crate. When you take these things in aggregate, that’s a very soft target. It’s a very low cost to find an exploit in those,” Thompson said.
“We think this is beyond the pale in what is acceptable, especially considering many of these protections are on by default and the major players have been able to accomplish this. These are not companies that are new to technology. They’re not startups that are bootstrapping and just getting into hardware. But they’re always going to be driven by market forces.”
Along with the removal of various hardening techniques across updates, CITL also discovered a large number of identical binaries that were used by at least two separate vendors. There were more than 3,700 binaries duplicated across more than one vendor, a finding that Thompson said was likely a result of so many vendors using common toolchains and upstream suppliers. That duplication has a two-fold effect: an attractive attack target spread across multiple platforms and also a common path for improvement.
“Improving the security of these common binaries would be the first major move in a positive direction for the entire market,” he said. “We want to bring all of these common points to light and back it up with our data. We want to see where the blind spots are.”
Thompson said he'd like to come back to the study in a year or so to see how things have changed, if at all.
"There's a lot more we'd like to go digging into. We bring data to the table that can help disprove groupthink on these topics," he said.
CITL's heat map shows some significant duplication of binaries across vendors.