Remote Fuzzer Monitoring with Windows Error Reporting (WER)
A great deal has been published about various approaches to fuzzing in recent years. Like any good research team, we’ve been keeping the cores warm in the never-ending search for bugs. One of our recent projects turned out to be a little less straightforward than we expected (we should know better by now), so we thought it might be worth sharing this cheap hack with a broader audience.
Instrumentation, crash collection and triage are all important parts of the fuzzing process. In fact, sometimes the process of monitoring and triaging application faults presents a bigger challenge than bug discovery itself. Some applications can make fault-monitoring efforts non-trivial.
Take an application like Microsoft’s Edge browser for example. Edge is a ‘Universal’ (formerly Metro) Windows App, built on top of the Windows Runtime (WinRT). WinRT applications introduce some complexity to debugging and monitoring due to some of the additional processes involved. In fact, even simply launching the process from the command line is not entirely straightforward.
For those who are unfamiliar with Microsoft Edge, we’ll provide a very brief explanation of the various processes involved and their purposes.
Edge is made up of two main executables, the main process ‘MicrosoftEdge.exe’ and a number of isolated content processes (MicrosoftEdgeCP.exe). In addition to these two executables, there are three other executables that support Microsoft Edge:
- browser_broker.exe - A medium integrity process that supports process isolation.
- ApplicationFrameHost.exe - A medium integrity process that enables WinRT applications to run in Windowed mode.
- RuntimeBroker.exe - A medium integrity process, which provides a security boundary for universal apps to enforce application permissions (can an app use your camera, microphone, etc.).
If you want a deep dive into some of Edge’s rendering internals, I recommend this presentation from IBM X-Force.
If you have ever tried to fuzz Microsoft Edge, you may have noticed some peculiar behavior from the host from time to time. For example, other WinRT processes occasionally terminate, and weird UI problems seem to appear at random. A lot of this unusual behavior is due to interactions with some of the additional components like the RuntimeBroker and ApplicationFrameHost which support all WinRT applications.
The only thing you really need to take away from all of this is that attaching a debugger solely to the main Edge processes (MicrosoftEdge.exe, MicrosoftEdgeCP.exe) may not provide complete insight into the application’s behavior. Simultaneously debugging all of these components can be a bit of a pain, but failure to do so may result in missing faults in supporting components.
At this point I’d be doing readers a disservice if I didn’t mention that SkyLined has written a great tool called EdgeDbg which wraps CDB.exe. This takes a lot of the headaches out of launching Microsoft Edge under a debugger, I highly recommend you check it out.
To make a long story short, EdgeDbg wasn’t entirely suited for some of the things we were trying to do at the time. We began to consider some alternate ways to do fault monitoring, that would be better suited to the task at hand. We needed to devise a solution that we could get up and running fast without much development effort.
Windows Event Log
Most readers are probably familiar with the Windows Event Log and some of the data it can collect. This is a fairly obvious place to look as the Event Log can record Windows Error Reporting (WER) events, which includes things like application crashes.
Unfortunately, the event log is stored in a binary XML (BXML) format that makes parsing it in real time a bit of a pain. However, the structures are documented if you want to parse them programmatically. But we wanted to capture the telemetry as quickly as possible without the burden of doing a bunch of file parsing.
My colleague Mikhail and I had a flashback to some crash dump analysis tools we used to work on. There is an alternative way to retrieve this information that requires very little effort, but gives us more flexible access to the information. It has the added benefit of allowing for easy remote logging. With that in mind, we decided to discuss a fairly lightweight (albeit imperfect) approach to fault monitoring.
If you want to know more about WER internals, Mikhail gave a presentation on this topic at Toorcon in 2011.
Corporate Error Reporting
Microsoft uses the Corporate Error Reporting V2 (MS-CER2) protocol to collect WER data, and various bits of telemetry about system behavior. The specification is open, fairly straightforward, and can be found here. We’ll provide a quick introduction to the format to get you up and running.
Reports are transmitted by an HTTP(S) POST request in a simple XML format. Let’s take a look at some of the data that we can collect this way. Here’s an example of the good old-fashioned BEX exception (the Mo prefix appears to be in relation to ‘Metro’ apps).
Microsoft.MicrosoftEdge_8wekyb3d8bbwe!MicrosoftEdge MoBEX {'Exception Code': 'c0000409', 'Package Full Name': 'Microsoft.MicrosoftEdge_38.14393.0.0_neutral__8wekyb3d8bbwe', 'Application Name': 'praid:MicrosoftEdge', 'Fault Module Name': 'EdgeContent.dll', 'Exception Offset': 'nope', 'Application Version': '11.0.14393.0', 'Fault Module Timestamp': '57b7e48f', 'Application Timestamp': '578997ee', 'Fault Module Version': '11.0.14393.103', 'Exception Data': '0000000000000007'}
In addition to the normal fault information you expect to see when looking for software flaws, WER provides some other information which can provide additional insight into the application’s behavior.
Microsoft Edge Content Process IENonFatalError {'MethodDef': 'res://c:\\windows\\system32\\f12\\f12script2.dll/23/debugger/DebuggerMerged.js', 'ModName': 'Debugger', 'AppName': 'microsoftedgecp.exe', 'Component': 'F12', 'Exception Code': "Unable to get property 'doc' of undefined or null reference", 'AppVer': '11.0.14393.0', 'Offset': '1', 'ModStamp': '5789990e', 'ModVer': '11.0.14393.0', 'AppStamp': '578997ee'}
In this case we are seeing a non-fatal error when a script in Microsoft’s ‘F12 Developer Tools,’ which provide the embedded webpage debugger (a resource inside f12script.dll) fails to resolve a property in the document.
The issue itself is not particularly exciting, but the example demonstrates some of the additional information you can glean about a process’ behavior. Here’s another example - Windows detected that the application was hung and generated a WER event:
Microsoft.MicrosoftEdge_8wekyb3d8bbwe!MicrosoftEdge MoAppHang {'Package Full Name': 'Microsoft.MicrosoftEdge_38.14393.0.0_neutral__8wekyb3d8bbwe', 'Hang Signature': '1938', 'Hang Type': '4194304', 'Application Name': 'praid:MicrosoftEdge', 'Application Version': '11.0.14393.103', 'Application Timestamp': '57b7e3b7'}
If you are using the Application Verifier, you will also see WER events. In this instance, the verifier thinks it detected a subtle case of potential heap corruption.
{'Application Timestamp': '57cf98db', 'Fault Module Version': '10.0.10586.0', 'Application Name': 'praid:MicrosoftEdge', 'OriginalBucketID': 'unknown', 'Fault Module Name': 'verifier.dll', 'Fault Module Timestamp': '5632d84f', 'Exception Offset': 'NOPE', 'Application Version': '11.0.10586.589', 'Status Code': '`5632d84f/3`', 'Package Full Name': 'Microsoft.MicrosoftEdge_25.10586.0.0_neutral__8wekyb3d8bbwe'}
You may have noticed the peculiar status code '5632d84f/3', the first portion of the code is the PE timestamp for verifier.dll, the ‘3’ indicates the actual status code (APPLICATION_VERIFIER_UNSYNCHRONIZED_HEAP_ACCESS).
This is all probably pretty familiar if you’ve ever taken the time to look at the Windows Event Viewer. But if you haven’t, it's easy to see we can collect quite a bit of useful information without actually attaching a debugger to the process at all.
Collecting WER Events
Configuring a machine to send telemetry to the CER server is simply a case of making a few minor changes to the system’s registry.
Leveraging WER while fuzzing allows for system wide crash reporting to a centralized collection server without having to re-attach a debugger between iterations. With a bit more work, one can also collection actual crashdumps over the protocol as well.
Under the Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting key, add the following entries:
-
CorporateWERServer- The IP address or hostname to send reports to. -
CorporateWERPortNumber- The port name on the server host where the HTTP server is listening. -
DisableQueue- Prevents the system from queueing transmission of any important information until later.
A complete guide to the WER registry settings can be found on MSDN.
Limitations and Tooling
There are some obvious limitations to this method. The first being, if you are not collecting the minidumps themselves, you will lack some of the additional context like stack traces, faulting instruction and register state. Triage requires a lot more effort.
Most importantly, if the application uses a top-level exception handler, there is the possibility you will trigger faults that you would only see in a debugger. Here’s a quick Windbg one-liner to check the address of the unhandled exception filter (UEF) for a process:
00007ffe`62878870 cc int 3
0:003> .call /s ucrtbased!GetImageBase KERNELBASE!SetUnhandledExceptionFilter(0); g
Thread is set up for call, ‘g’ will execute.
WARNING: This can have serious side-effects,
including deadlocks and corruption of the debuggee.
.call returns:
Unsigned int64 0x00007ff6`beab1037```
If `ucrtbased` is not in your process, substitute `ucrtbased!GetImageBase` with any other available function symbol with parameter info that takes a pointer and returns a pointer.
All of this being said, we have included a simple [python script](/assets/img/blogv2/wer.py) to listen for and parse WER reports with this post. Hopefully this will help you spin up some basic WER fault monitoring quickly, should you find yourself stuck with an application that is difficult to monitor.