macOS Notarization, Hardware-Backed Code Signing Keys, and Sparkle Code Signing Issues

00. Introduction

Notarization, introduced in macOS 10.14 (Mojave), verifies that software distributed outside of the Mac App Store does not contain malicious components. It also ensures that bundles, disk images, and packages are signed correctly, and that the Hardened Runtime entitlement is enabled for all executables. This system works by allowing developers to submit their Developer-ID signed executables to Apple for processing, which Apple will check for malicious components and code signing issues. When finished, the developer receives a notarization ticket from Apple that can be "stapled" to the bundle, disk image, or package. The stapling of this ticket allows the check to succeed on Macs without an internet connection. If the developer fails to properly staple the ticket, their software will still run, but the OS first checks-in with Apple's notary service.

Apple first introduced Notarization at WWDC 2018, and told developers that if they Notarized their apps end-user's would receive a "more streamlined Gatekeeper dialog." However, it was also made clear that in a future version of macOS Notarization would be required. Notarization is enabled in the current major version of macOS, 10.14 (Mojave), but it's not required in most cases. This means that signed-yet-unnotarized software will run on macOS Mojave, but users may not see a "streamlined Gatekeeper dialog." On macOS 10.15 (Catalina), however, Notarization will be required. Catalina is in beta now and will ship to customers later this Fall. This means that, by default, non-notarized bundles, disk images, and packages distributed outside of the Mac App Store will not run on macOS Catalina. This also means that, if developers haven't already Notarized their apps, they should do so now.

Additionally, as of the release of macOS 10.14.5, Apple now requires that any apps, disk images, and installer packages signed with a newly created Developer ID certificate be notarized, and that any new or updated kernel extensions be notarized too. Also, as of September 3, 2019, Apple has relaxed the Notarization prerequisites until January 2020 to make the transition easier for developers. Details regarding the updated requirements are documented on the Apple Developer News and Updates site.

01. Notarizing Your Software

Fortunately, Notarization is pretty easy to get right when using Xcode exclusively. When you export your Developer ID-signed application in Xcode, a dialog appears asking if you want to submit your app to Apple's notary service. Xcode will take care of uploading the signed payload to the notary service and attaching the notarization ticket once completed.

However, if you use the Xcode Command-Line Tools to build your software instead, perhaps because you've automated your build process as part of a CI/CD workflow, then you can Notarize your software with the xcrun altool command.

// Notarize application bundle.
xcrun altool --notarize-app --primary-bundle-id com.example.YourApp -u <apple_id_username> -p <apple_id_password> -f YourApp.zip --output-format xml

The output of the previous command will include a RequestUUID, which can be used with the --notarization-info switch to determine that status of the notarization request.

// Check Notarization status.
xcrun altool --notarization-info <RequestUUID> -u <apple_id_username> -p <apple_id_password> --output-format xml

If the notary service successfully validated your payload, you can then staple the ticket to the payload using the stapler command:

// Staple notarization ticket to payload.
xcrun stapler staple YourApp.app

02. Hardware-Backed Code Signing Keys, Notarization, and Code Signing

We love releasing and contributing to open-source at Duo Labs. One example of this is the EFIgy command-line tool and API that informs you about the state of your Mac firmware. In addition to the CLI tool, we also have a Mac app called EFIgy-GUI. Recently, when releasing a new version of the EFIgy-GUI app, I remembered that Notarization will be required as of macOS 10.15 (Catalina), so I decided to use this opportunity to figure out what it would take to Notarize the app. EFIgy-GUI is a pretty small app and we don't release new versions of it often, so we haven't gone through the trouble to entirely automate our build-process. Instead, we export un-signed archives from Xcode, then sign the bundle with a Developer-ID signing key which is stored on a YubiKey 4. The process of using YubiKeys for Mac code signing is documented by Yubico. For whatever reason, code signing through Xcode using this method doesn't appear to work, but using the CLI tools does.

The reason we sign EFIgy-GUI with a hardware-backed signing key is because we believe great care should be taken to protect Developer ID signing keys. By storing the Developer ID signing key on a YubiKey, such that it's non-exportable and requires a passphrase to authorize cryptographic signing operations, we reduce the likelihood that an adversary with code-execution could use the key to sign arbitrary code, or extract the key material. Additionally, even if the security key were stolen, the adversary would need the passphrase in order to sign anything. The reason we go to such great lengths to protect the Developer ID key material is because the stakes are so high. If the key were to get into the wrong hands, it would allow for malicious code to be signed and released as if it were from Duo. That wouldn't be good. The signing keys used to release software in the Mac App Store, (although important to protect), do not pose the same risk. This is because all software downloaded from the App Store goes through a more extensive review process before being re-signed by Apple.

03. Notarization Doing Its Job

As previously mentioned, Apple's notary service also checks for code signing issues in the payloads it inspects. Once I had gotten around to submitting EFIgy-GUI to the notary service, it was failing, saying that the Hardened Runtime is not enabled on the embedded Sparkle AutoUpdate.app. This was surprising to me, since I had seemingly signed the EFIgy-GUI app successfully, and had followed all the other prerequisites documented by Apple, such as enabling Hardened Runtime and including the --timestamp flag when code signing my bundle. After all, the application runs without Gatekeeper complaining. Plus, when I verify the code signature with the codesign tool, it reports that the app is signed correctly.

// Notarization Saying that Sparkle's AutoUpate.app doesn't have the Hardened Runtime entitlement enabled.
Distribution items ineligible: Error Domain=IDEDistributionMethodDeveloperIDErrorDomain Code=1 "Hardened Runtime is not enabled." UserInfo={NSLocalizedDescription=Hardened Runtime is not enabled., NSLocalizedRecoverySuggestion="Autoupdate.app" and "fileop" must be rebuilt with support for the Hardened Runtime. Enable the Hardened Runtime capability in the project editor, then test your app, rebuild your archive, and upload again.}

I mentioned this issue to a colleague of mine, Dave Gross, and he mentioned that I should check if the AutoUpdate.app bundle embedded within the main bundle is code signed. Sure enough, it wasn't, which explains why the notary service was complaining. This means the notary service is doing its job to help identify code signing issues. Cool!

For example, if we look at the previous version of EFIgy-GUI, which is signed but not notarized, codesign says the application bundle is signed.

$ codesign -dv --verbose=4 /Applications/EFIgy.app
<snip>
Authority=Developer ID Application: Duo Security LLC (FNN8Z5JMFP)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
<snip>

The same is true even if we pass the --deep flag to codesign when verifying the top-level application bundle.

$ codesign --deep -dv --verbose=4 /Applications/EFIgy.app
<snip>
Authority=Developer ID Application: Duo Security LLC (FNN8Z5JMFP)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
<snip>
Sealed Resources version=2 rules=13 files=5
Nested=Frameworks/Sparkle.framework
Nested=Frameworks/LetsMove.framework
<snip>

However, if we instead ask codesign to verify the embedded AutoUpdate.app bundle, it reveals that it isn't signed.

$ codesign -dv --verbose=4 /Applications/EFIgy.app/Contents/Frameworks/Sparkle.framework/Versions/A/Resources/Autoupdate.app
/Applications/EFIgy.app/Contents/Frameworks/Sparkle.framework/Versions/A/Resources/Autoupdate.app: code object is not signed at all

Apparently, this is a known problem when using CocoaPods to install the Sparkle framework, as it includes a pre-compiled, unsigned AutoUpdate.app bundle. It was a little weird to me that the way it manifested is that the notary service informed me that the Hardened Runtime entitlement wasn't enabled, but nevertheless it correctly identified a code signing issue.

04. The Fix

The fix turned out to be just code signing the AutoUpdate.app bundle before notarizing, (in addition to the other changes necessary to get Notarization working, such as enabling the Hardened Runtime entitlement and including the --timestamp flag in our codesign arguments). If you want to see the full example, take a look at the Makefile in our EFIgy-GUI GitHub repository, which is heavily based on this great GitHub Gist from Vadim Shpakovski. Below shows how the Makefile is used. Note that an Apple ID username and password must be set when using the notarize target.

$ make archive && make export-archive && make codesign
<snip>
buildArchive/EFIgy.xcarchive/Submissions/EFIgy.app: replacing existing signature
buildArchive/EFIgy.xcarchive/Submissions/EFIgy.app: signed app bundle with Mach-O thin (x86_64) [com.duosecurity.EFIgy]
$ DEVELOPER_USERNAME="<apple_id_username>" DEVELOPER_PASSWORD="<apple_id_password>" make notarize
<snip>
The staple and validate action worked!
rm -rf "buildArchive/EFIgy.xcarchive"/Products/Applications/EFIgy.app
mv "buildArchive/EFIgy.xcarchive"/Submissions/EFIgy.app "buildArchive/EFIgy.xcarchive"/Products/Applications/

05. Useful Notarization and Code Signing Commands

I found the following commands useful when troubleshooting the Notarization issues described in this post. Hopefully they'll be helpful to someone else, too.

// Check code signature on YourApp.app.
$ code sign -dv --verbose=4 /Applications/YourApp.app
// Check Sparkle AutoUpdate bundle code signature.
$ codesign -dv --verbose=4 /Applications/YourApp.app/Products/Applications/YourApp.app/Contents/Frameworks/Sparkle.framework/Versions/A/Resources/Autoupdate.app
// Check if Hardened Runtime is enabled.
$ codesign -vvvd /Applications/YourApp.app 2>&1 | grep "Runtime Version"
Runtime Version=10.14.0
// Check if bundle, disk image, or package is Notarized.
$ spctl -a -t exec -vvv /Applications/YourApp.app 2>&1|grep Notarized
source=Notarized Developer ID
// Check if Notarization ticket is stapled, (will be empty if stapled).
stapler validate -v /Applications/YourApp.app|grep "does not have a ticket"

06. Summary

Although not perfect, Notarization brings us closer to achieving iOS-level application security, and is a great overall security win for macOS. As the tooling and documentation for new security features like Notarization improve, developers will need to fight with it less, and we'll wonder how we ever so nonchalantly ran un-notarized code on macOS. Of course, it will take some time for us to get there, and there will be a long tail of support for running untrusted code on macOS for legacy reasons, but the "iOSification" of macOS is coming whether we like it or not. ;)