Skip navigation

Effective October 28, 2019 Duo Security will be transitioning to Cisco's Privacy Statement. View the Duo Privacy Data Sheet.

Jeremy Erickson

Jeremy Erickson

Senior R&D Engineer

Jeremy is a researcher on the Duo Labs team who enjoys working towards the death of the password. He believes usability is just as important as strong security. He prefers understanding and education to security-by-checkbox.

Building a Sysdiagnose Client

Jeremy Erickson December 18th, 2018 (Last Updated: December 18th, 2018)

01. Where to get it

https://github.com/duo-labs/apple-t2-xpc

02. Analysis of an existing sysdiagnose connection

Using our handy sniffer.py sniffer, we can now dissect the messages being sent between the imac and t2 chip.

The communication from a sysdiagnose -c command can be broken into several parts:

  • The sysdiagnose client opens an http/2 connection to the t2 chip

imac opening stream 1 for communication on port 49155.

New HTTP/2 frame
New XPC Packet imac->t2 on HTTP/2 stream 1 TCP port 49155
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00000000 00000000 00000001 (0x1)
	BodyLength: 0x14
	MessageId: 0x0
}
{ ~0 entries~
}
  • The sysdiagnose client sends an XPC message with the message REQUEST_TYPE=1
New XPC Packet imac->t2 on HTTP/2 stream 1 TCP port 49155
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00000000 00000001 00000001 (0x101)
	BodyLength: 0x30
	MessageId: 0x1
}
{ ~1 entry~
	"REQUEST_TYPE":
		uint64 0x0000000000000001: 1
}
  • The t2 chip sends a message on a different tcp port with metadata about a file transfer

New XPC Packet t2->imac on HTTP/2 stream 1 TCP port 49164
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00000000 00000001 00000001 (0x101)
	BodyLength: 0x12c
	MessageId: 0x1b
}
{ ~3 entries~
	"messageType":
		<file>
	"fileMetadata":
		{ ~4 entries~
			"proxied_dev":
				Bridge
			"name":
				stacks-2018-09-26-154602.ips
			"status":
				{ ~1 entry~
					"jobStatus":
						{ ~1 entry~
							"jobId":
								<unsolicited>
						}
				}
			"subdir":
			"/"
		}
	"<file_xfer>":
		MessageId: 0xe
		File transfer size: 0x000000000001b981 113025
}
  • The t2 chip opens a new stream and sends a metadata file to the imac

t2 opening stream 31 for communication on port 49164.
...
New XPC Packet t2->imac on HTTP/2 stream 31 TCP port 49164
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00010000 00000000 00000001 (0x100001)
	BodyLength: 0x0
	MessageId: 0xe
}
No Payload.
...
New XPC Packet imac->t2 on HTTP/2 stream 31 TCP port 49164
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00100000 00000000 00000001 (0x200001)
	BodyLength: 0x0
	MessageId: 0xe
}
No Payload.
...
New Data frame t2->imac on HTTP/2 stream 31 TCP port 49164
0000  7B226275675F74797065223A22323838 {"bug_type":"288
0010  222C2274696D657374616D70223A2232 ","timestamp":"2
0020  3031382D30392D32362031353A34363A 018-09-26 15:46:
0030  30322E3035202B30303030222C226F73 02.05 +0000","os
... 16384 bytes
... and so on until the file is transferred
  • The t2 chip sends another metadata file with system information in the same fashion, on the same other port, on a new http/2 stream.

  • The t2 chip sends a sysdiagnose tgz on the original tcp port


t2 opening stream 2 for communication on port 49155.
...
New XPC Packet t2->imac on HTTP/2 stream 1 TCP port 49155
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00000000 00000001 00000001 (0x101)
	BodyLength: 0xc0
	MessageId: 0x2
}
{ ~3 entries~
	"RESPONSE_TYPE":
		uint64 0x0000000000000001: 1
	"FILE_TX":
		MessageId: 0x5
		File transfer size: 0x00000000005b49d7 5982679
	"FILE_NAME":
		"bridge_sysdiagnose_2018.09.26_15-46-02+0000_Bridge_OS_Bridge_15P2542.tar.gz"
}
...
New XPC Packet t2->imac on HTTP/2 stream 2 TCP port 49155
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00010000 00000000 00000001 (0x100001)
	BodyLength: 0x0
	MessageId: 0x5
}
No Payload.
...
New XPC Packet imac->t2 on HTTP/2 stream 2 TCP port 49155
XPC Wrapper: {
	Magic: 0x29b00b92
	Flags: 0b 00000000 00100000 00000000 00000001 (0x200001)
	BodyLength: 0x0
	MessageId: 0x5
}
No Payload.
...
New Data frame t2->imac on HTTP/2 stream 2 TCP port 49155
0000  1F8B0800BAA9AB5B0003ECBD6B77DB48 .......[....kw.H
0010  9226DC5FA77E05579FAACF5832018237 .&._.~.W...X2..7
0020  4DB9F79528D9E694647144D955BD7BF6 M...(...dqD.U.{.
0030  F840242471CDDB80A42D4D4FEF6F7F9F .@$$q....-MO.o..
... 16384 bytes
... and so on for 5982679 bytes

The main curiosity is why are the metadata “files” that get transferred on a different tcp port?

Some clues:

  • The metadata “files” that are transferred don’t show up in the sysdiagnose output on the imac.
  • There is no tcp or http/2 handshake on the other tcp port.

Clearly, the t2 chip is sending this system diagnostic data to some other process on the imac, not the sysdiagnose client, and maintains a persistent connection to it. But who is it?

03. Who else is talking to my t2 chip!?

Normally, to track down the process that’s communicating on a particular port, we could use netstat or lsof to see which process has an open tcp port (listening or connected). However, since this mystery process is connecting to the t2, it doesn’t show up in the normal netstat list.

The t2 chip’s messages going off to the other tcp port contain what appear to be unique strings messageType and fileMetadata. The sysdiagnose binary doesn’t contain these strings, and neither do any of its linked libraries:


Duo-Labs-iMac-Pro:~ labs$ objdump -macho -dylibs-used /usr/bin/sysdiagnose
/usr/bin/sysdiagnose:
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1451.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1451.0.0)
	/System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport (compatibility version 1.0.0, current version 1.0.0)
		/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
		/usr/lib/libarchive.2.dylib (compatibility version 9.0.0, current version 9.2.0)
	/System/Library/PrivateFrameworks/LoggingSupport.framework/Versions/A/LoggingSupport (compatibility version 1.0.0, current version 829.30.14)
	/System/Library/PrivateFrameworks/CacheDelete.framework/Versions/A/CacheDelete (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1129.5.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 822.19.0)
	/System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.47.11)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 963.30.1)
	/System/Library/PrivateFrameworks/EmbeddedOSSupportHost.framework/Versions/A/EmbeddedOSSupportHost (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/PrivateFrameworks/RemoteServiceDiscovery.framework/Versions/A/RemoteServiceDiscovery (compatibility version 1.0.0, current version 1205.47.10)
		/usr/lib/libdscsym.dylib (compatibility version 1.0.0, current version 183.1.0)
		/usr/lib/libnetwork.dylib (compatibility version 1.0.0, current version 1.0.0)
		/usr/lib/libsysmon.dylib (compatibility version 1.0.0, current version 1.0.0)
		/usr/lib/libsystemstats.dylib (compatibility version 1.0.0, current version 403.47.2)
	/System/Library/PrivateFrameworks/RemoteXPC.framework/Versions/A/RemoteXPC (compatibility version 1.0.0, current version 1205.47.10)
		/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

However, there has to be some library or binary that parses the message structure that’s being sent to our mystery process.

These strings (messageType and fileMetadata) show up in /private/var/db/dyld/dyld_shared_cache_x86_64, which is apple’s giant cache of system libraries, used to improve performance. Loading dydl_shared_cache_x86_64 into IDA took more than 24 hours, but when it finished, both of these strings showed up in the OSAnalytics library. (nowhere else)

Loading the OSAnalytics library into IDA directly was much quicker, and decompiling the code around the use of those strings, it became clear that the OSAnalytics library does parse the XPC object structure passed from the T2 chip above (in which these magic strings occur).

However, the next step was to find all the binaries the imac may be running that use the OSAnalytics library to find our mystery process. Using a slightly modified search.py, we found /System/Library/CoreServices/osanalyticshelper and /System/Library/CoreServices/SubmitDiagInfo used the OSAnalytics library.

Analysis of the osanalyticshelper library showed that it was a thin wrapper around the OSAnalytics library, and didn’t appear to initiate any XPC connections (or call into the right places in OSAnalytics). SubmitDiagInfo on the other hand, is a long-running process that collects diagnostic information from the system and (sometimes) bundles it off to Apple. Further, after killing off the system SubmitDiagInfo process, a new sysdiagnose -c command would revive it, and we would see new TCP/HTTP2 handshakes using sniffer.py. This makes it pretty definitive that the SubmitDiagInfo process is what is capturing this system diagnostic information from the t2. However, it doesn’t appear to be stored to disk in the normal places (generally accessible via console.app). It remains to be investigated where this data is eventually going.

Side note: from Jonathan Levin’s awesome *OS book, I believe this is because launchd will relaunch it when a packet comes in looking for such a service, but I’m not seeing it show up in the list of services registered in launchctl list. This probably needs further investigation.

04. Finishing a sysdiagnose request

For the longest time, my custom sysdiagnose client, which uses the Twisted and h2 python libraries to handle the network connection, could initiate a sysdiagnose request to the t2 chip, but eventually the t2 chip would eventually stop sending data. This turned out to be primarily because of two issues:

  1. At the beginning of the sysdiagnose request, the imac initiates what appears to be a spurious http/2 stream on stream. It opens stream 3 and then sends an empty XPC payload (that is, an XPC wrapper with no XPC objects in it). The t2 chip then replies with another empty XPC payload. Neither the t2 chip nor the imac ever use stream 3 again for the life of the connection.

    As it turns out, opening stream 3 triggers something in the t2 chip’s sysdiagnose server that causes it to continue sending data as we would expect. So our client now does this.
  2. The t2 chip sends back a bridge_sysdiagnose_<date>.tar.gz file that is approximately 5MB in size. While the initial connection and configuration messages (REQUEST_TYPE=1 and RESPONSE_TYPE=1) fit within the stream’s initial http/2 window size, 5MB is larger than the default window size of a new stream (approx 196KB, in this case).

The solution was to use h2’s H2Connection.acknowledge_received_data() method to update the H2Connection object’s state that new data had been received, so that it would send a WINDOW_UPDATE frame to extend the t2’s window periodically throughout the file transfer.

05. Some notes on Apple’s HTTP/2 usage

I’m not quite sure why Apple chose to wrap the communication between the t2 chip and the imac in an HTTP/2 layer. Perhaps it makes it easier to maintain long-running connections, or logically simpler to send control messages over one stream and data transfers over another stream. Whatever their original motivations, in practice, their implementation complicates the ability to interact with the t2’s exposed services.

The largest issue by far is that the http/2 implementation that Apple has adopted breaks the http/2 specification in a few ways:

  • The spec dictates that stream ids should be monotonically increasing. In the particular case, of the sysdiagnose file transfer, the t2 chip opens stream 2 after the sysdiagnose client opens stream 3.

  • The spec dictates that the http/2 client must open only odd-numbered streams, and the server must open only even-numbered streams. The apple implementation may actually be compliant with this. It appeared to be non-compliant during initial investigation, but if the t2 chip reaching out to the SubmitDiagInfo service is considered the client, then the odd-numbered streams it uses would be compliant.

  • It also breaks the h2 library’s implementation in that it sends lots of http/2 frames with empty payloads. Specifically, HEADER frames. In a typical http/2 connection, the HEADER frame is used to send… HTTP headers! However, the HEADER frame is also what is used to open new streams. Apple using it to open a stream without passing any HTTP headers along breaks the h2 implementation (and the spec), which requires a patch.