Silver's Weblog

If you've got a crashed process that WER has handled but you either know the JIT debugger is set wrong or you haven't got one set up at all (so no "Debug" button on the crash dialog), you can leave the dialog open and attach any debugger of your choice to the suspended, crashed process.

However, there's a problem. By attaching the debugger to the suspended process, the process' last-exception information has been lost; this is because the way the debugger sucessfully attaches to a suspended (or hung) process is to inject a new, running thread and have that thread crash. So your debugger, and any minidumps it generates with .dump, will not be able to find the original program's exception information. Oops!

The solution is very similar to Raymond Chen's post on Sucking the exception pointers out of a stack trace. The difference is how we find the pointers and what we do with them.

Starting with the stack trace from kb:

RetAddr           : Args to Child                                                           : Call Site
000007f9`d88212d2 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForMultipleObjects+0xa
000007f9`dae8d20e : 00000000`02300000 000007f6`96e75000 00000000`00000000 00000000`000001dc : KERNELBASE!WaitForMultipleObjectsEx+0xe5
000007f9`dae8cfd2 : fffff8a0`00000080 00000000`80004005 00000000`000001ec 00000000`00000000 : KERNEL32!WerpReportFaultInternal+0x1fa
000007f9`d889fc87 : 00000000`00000000 00000000`00195c30 00000000`00000030 00000000`00195c30 : KERNEL32!WerpReportFault+0x42
000007f9`db56a43d : 00000001`00000000 00000000`00196990 00000000`0019f620 00000000`003441d0 : KERNELBASE!UnhandledExceptionFilter+0x1d7
000007f9`db4b4ad4 : 000007f9`d8eeabdd 00000000`00196990 00000000`00000000 00000000`00000000 : ntdll!LdrpLogFatalUserCallbackException+0x4d
000007f9`db4d464d : 00000000`00000000 00000000`00195de8 00000000`00195da0 000007f9`d8f7f000 : ntdll!KiUserCallbackDispatcherHandler+0x20
000007f9`db4d567c : 000007f9`d8ee0000 00000000`0019f5c0 000007f9`0000e4f0 00000000`00000000 : ntdll!RtlpExecuteHandlerForException+0xd
000007f9`db4b4bba : 00000000`00000000 00000000`00000000 000007f6`96e7e300 00000000`00000000 : ntdll!RtlDispatchException+0x392
000007f9`db4b2acd : 000007f9`d8bbc36c 00000000`00360f00 00000000`00360f00 00000000`00360f00 : ntdll!KiUserExceptionDispatch+0x2e
000007f9`d8bbc36c : 00000000`00360f00 00000000`00360f00 00000000`00360f00 00000000`00000000 : ntdll!memcpy+0x21a

The EXCEPTION_POINTERS structure address seems[1] to be passed as arguments 2 and 4 of the call to "KERNEL32!WerpReportFault". From this, the exception and context records can be displayed - but we only really need to include the EXCEPTION_POINTERS address in the minidump, since adding this one address will allow the !analyze command to work.

Create the minidump using the command ".dump /ma /xp 00195c30 dump.dmp" and you're done. (Apart from all of the actual debugging - have fun!)

[1] This is on Windows 8 64bit running the 64bit WinDbg debugging a 32bit executable. YMMV.

When measuring website performance, comparing two different page loads is a common and useful operation. However, there isn't much available to do it; some tools, like Fiddler, let you export the timeline waterfall as an image, allowing a visual inspection of two page loads to be done, but this lacks the ability to easily compare things such as when a particular request was made in each page.

A good intermediate solution is to save HTTP Archive files of each page load; these are JSON files containing the request and response timing information, headers, cookies and (depending on tool) contents. HTTP Archive (HAR) files can be exported natively from Chrome and Fiddler, and via the NetExport extension for Firebug in Firefox. Internet Explorer can export an XML transcription of the official JSON HAR format.

But what about comparing the exported HAR data? There is an online HAR Viewer but nothing to compare two HAR files. This particular issue came to light for me a few months back, when I was working on page load performance at work, but it really hit home when, at the WebPerfDays on Friday, "HAR diff" appeared on a list of tools that web performance people wanted.

Putting the two together, I have put my HAR difference tool online for all to use. Just paste in two HAR files (or just one to view it alone) and hit "Compare!". The tool summarises the differences, both in which requests were made, their timing information, and how the browser utilised its connections. The precise level of details depends on whether a proxy (like Fiddler) or browser tool (like Firebug) was used to collect the information, but most of the information is generally available from anything that exports HAR files.

Windows Live Mesh has a remote connections feature, similar to Remote Desktop, which can be accessed from the Windows Live Mesh client and the Windows Live Devices website. I've used it frequently when I needed to access my home machine from work - it connects despite no ports being forwarded in the router and it doesn't mess up my multi-monitor configuration at home (normal Remote Desktop will change the resolutions and such).

That was until I changed my multi-monitor configuration at home.

Old arrangement: 1280x1024 / 1600x1200* / 1280x1024 (* primary)

New arrangement: 1280x1024 / 2560x1440* / 1600x1200 (* primary)

Then, every time Windows started, shortly after the login screen appeared, all the monitors would jumble themselves up in to something very closely resembling the old arrangement: 1280x1024 / 1280x1024 / 1600x1200* (* primary). It seemed clear something was attempting to put the monitors into the old arrangement based on some saved information, but it wasn't obvious what that something was.

To diagnose the issue, I carefully timed when the monitors first started showing signs of rearrangement after boot and consulted the list of processes and their start times (from "wmic process") to see what was starting at or just before the rearrangement.

That's when I spotted "C:\Program Files\Windows Live\Mesh\wlcrasvc.exe", the Windows Live Mesh remote connections service, starting just seconds before the rearrangement.

So up came Registry Editor (regedit.exe) and a look through HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft until I found HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Live Mesh. Hiding under there, at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Live Mesh\Remote Desktop\DisplayDevices, is a key for each display device on the machine and some monitor arrangement data (x, y, width, height).

I stopped the Windows Live Mesh remote connections service and deleted the entire HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Live Mesh key, since it didn't appear to contain anything except the display devices information. Upon restarting the service, some parts of the key came back but no monitors got rearranged and, upon connecting from another machine, the display devices keys came back, but with my new arrangement. I restarted the service again to be sure, and nothing rearranged itself.

Fixed!

With the release of Open Rails 0.7.0, now seems a good time to document some of the hidden settings and features:

• LoggingFilename (string setting) Specifies a static or generated name for the log file. Defaults to "OpenRailsLog.txt". This is a standard .NET format string, with the following replacements available: (New in 0.7.0)
  1. "Open Rails".
  2. Version number normally, build stamp if compiling yourself.
  3. Version number.
  4. Build stamp.
  5. Current date/time.
• LoggingPath (string setting) Specifies an absolute path to save the log file when running the game. The default is your desktop.
• ScreenshotPath (string setting) Specifies an absolute path to save in-game screenshots (using Print Screen). The default is the "Open Rails" folder in your pictures folder. (New in 0.7.0)
• ShadowAllShapes (boolean setting) Specifies whether to generate shadows from all objects in the game, instead of only those which the route creator has specifically set. This can often make a route more visually pleasing, at a small performance cost.
• VerticalSync (boolean setting) Specifies whether to limit the frame-rate to the monitor's refresh rate.
• ViewingFOV (integer setting) Specifies the field-of-view for the display in degrees, vertically. The default is 45, which is equivalent to 60 horizontally in a 4:3 ratio. Because Open Rails measures this vertically, you'll automatically see more on widescreen resolutions. (New in 0.7.0)
• Print Screen Takes a screenshot and saves it in the ScreenshotPath above in the PNG format. (New in 0.7.0)
• Alt Hold down the "Alt" key and click on a switch to throw it (change which path is taken).
• U Hold down the "U" key and click on a coupling to uncouple part of the player train.
• Control - E Switches between locomotives in the player train.
• Shift - Control - F Switches the direction of the player locomotive. This is a temporary substitute for the lack of MSTS Bin reverse cab support.
• Alt - F1 Dumps the command list with keys to "keyboard.txt" and a high-resolution keyboard image (suitable for printing) to "keyboard.png".
• Control - Alt - Page Up/Page Down/Home Controls the game speed, increasing, reducing and resetting it respectively.
• Control - Alt - F11 Toggles the dispatcher/signalling visualisation, which highlights the selected route ahead of the player and AI trains, with labels for switches and signals.

To edit hidden settings

All Open Rails settings are stored in the Registry, under the key "HKEY_CURRENT_USER\Software\OpenRails\ORTS". The usual disclaimers about editing the registry apply.

• To edit any setting, open Registry Editor (regedit.exe) and navigate the left side to "HKEY_CURRENT_USER\Software\OpenRails\ORTS".
• Some settings will appear already, some have to be created.
• To create a setting, either use the menu "Edit" > "New" or right-click the right side and use the menu "New".
  • String settings should be created as "String Value".
  • Integer and boolean settings should be created as "DWORD (32-bit) Value".
• To edit any setting, double-click its name on the right.
  • Strings can just be edited as-is.
  • Integer settings should be edited with the "Base" option set to "Decimal" in the edit window.
  • Boolean settings should use the value "1" for true and "0" for false.

Microsoft Train Simulator uses a custom file format for its textures and other in-game images - ACE (no, not the data compression format). They are stored as a 2nd-level format inside the Simis file format, which means they can be compressed with zlib's DEFLATE algorithm transparently.

I'm not going to document the format publicly just yet, but instead show a preview of loading ACE images in the next version of Simis Editor:

I have also implemented a DXT1 compression algorithm which, together with the decoder, allows Simis Editor to load and save ACE files in all three of their "formats":

• Uncompressed (A)RGB.
• Zlib compressed (A)RGB.
• DXT1 compressed.

While Simis Editor will not become an image editor in its own right, I will be supporting importing/exporting as standard image types (BMP, PNG, etc.) and the new "ImageFile" command-line tool will provide for easy bulk conversions. Stay tuned for more details.

Download Simis Editor v0.5 and read the documentation. Release highlights:

• New format support: Cameras, GUI, GUI Bitmaps, GUI Screens, Route Forests, Route Gantry Sets, Route Speed Post Sets, Route Tile Definition, Route Tile Definition Low, Route Track Types, Signals.
• Updated format support: Activity, Route Reference, Route Train Path, Shape, Train Consist, Train Engine, Train Wagon, World.
• Bug fixes and general improvements to underlying libraries.
• Thanks to Jeffrey Kraus-Yao for many of the format additions and updates.

I have a Matroska (.mkv) file with the following tracks (data streams):

Tracks : 2
Track 1 : Video
  - Codec : (V_MPEG4/ISO/AVC)
Track 2 : Audio
  - Codec : MPEG Audio 1, 2, 2.5 Layer III (A_MPEG/L3)

Nothing particularly special there; I have the following relevant DirectX Media Objects (DMOs), DirectShow and Media Foundation codecs installed:

• Haali Media Splitter: a DirectShow splitter filter for Matroska files, among other container formats (analogous to the AVI Splitter for .avi and others).
• ffdshow: a DirectShow decoder filter for just about anything, including MPEG-4 Video and MPEG-1 Audio Layer 3 (MP3).
• Windows 7's in-box DMOs decoder filters for MPEG-4 Video and MP3. These can be used by both DirectShow and Media Foundation.

Question: What happens if this Matroska file is played in Windows Media Player or Windows Media Center?

Answer: No video and audio stutters a lot.

Question: Why?

Answer: Both will try to use Media Foundation first and DirectShow second. As Media Foundation has no preferred splitter for Matroska files (either in-box or that I've installed), it hunts for a supported transform (similar to DirectShow's filters) with which to play the file; the MP3 transform duly indicates that it can play the file.

I believe this is because the MP3 decoder ignores the data at the start of the file which it doesn't understand (to allow for ID3 tags) and then picks up the first frame of the audio stream inside the file. The stuttering is most likely it attempting to play back the video frames of data (the two data streams are interleaved within the container).

The solution: Amazingly simple; the only thing that matters is that they're trying to use Media Foundation first, so set one registry key to indicate .mkv files prefer to be handled by DirectShow and it works great in both players.

HKEY_CURRENT_USER\Software\Microsoft\MediaPlayer\Player\Extensions\.mkv
   Runtime    REG_DWORD    0x7

Done.

I've just released the latest version of my Microsoft Train Simulator tools: Simis Editor v0.4 with the usual documentation. Some highlights for this release:

• Open and Save dialogs support full filename filters from BNFs (e.g. "tsection.dat") in addition to extension filters.
• Support for adding new blocks to the tree via context menu with 4 groups of operations:
  • Insert previous siblings.
  • Insert next siblings.
  • Insert before children.
  • Insert after children.
• Problems loading *.bnf files and loading or saving Simis files are all offered for reporting online (via the Feedback class).
• Added a status bar and help text for menu items.

Feedback makes everything better, eventually. Getting or sending feedback is, however, not always simple or usable; users need to be able to bang out simple comments easily, with no forms to fill in, whilst still providing proper context and technical information if the feedback is the result of the application malfunctioning. Feedback should also be anonymous if the user wishes. The Feedback class in the next release of Simis Editor is attempting to do this; here I'm going to outline its user-facing functionality and the back-end implementation.

Entry Points

There are two different ways the feedback process can be started:

• From the user: a "Send Feedback..." menu item under "Help".
• From the application: anywhere in the application that catches exceptions.

While both routes show the same dialog, the latter case collects a load more contextual information to go with the report - most obviously, the exception, but it can also take anything the catch code wants to include.

Instanciation Code

The Feedback class is really simple to use, for both cases:

   try {
       new Feedback().PromptAndSend(ownerForm);
   } catch (SomeException e) {
       new Feedback(e, "sending feedback").PromptAndSend(ownerForm);
   }

The ownerForm is used for showing the dialog modally. The class switches mode based on the arguments: none means "user feedback", Exception (exception) and String (operation) mean "application failure"; there is also a third mode where the caller provides the feedback type, operation and an IDictionary<string, string> of details.

User Dialog

The dialog is mostly the same for the two cases; the biggest change is the "faces" and introductory text. For user feedback, the introduction just explains when to include your e-mail address, as it is entirely optional.

In the application failure case, this dialog is the first thing the user sees when an operation fails, so it must explain that something's gone wrong and then why you should send the feedback at all.

As the purpose of the feedback dialog is to collect as many reports as possible, it attempts to ensure all users (or a maximum of users) are happy to send the reports by allowing the user to view all the data collected for sending. As shown below, this includes the full exception details (obviously) as well as some general system information. It also includes a user ID, which is randomly generated the first time the application intends to send feedback and which is not shared between applications (i.e. two applications that a user has installed that use this feedback system will each send a different user ID).

If the user is happy to send the report and clicks the button, an XML document is constructed, serialised and POSTed to the feedback server. The user is then given a message showing the success or failure of the feedback as a clear completion of the process.

Feedback Format

The feedback is sent as XML to make handing the data as easy as possible. This is an example of an application failure report, but user feedback reports are basically the same - just without the <details>.

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<report version="1.0" uid="ipejGfrUIt5gAZ3Y" time="2010-05-31T22:13:56.4276545+01:00" type="ApplicationFailure" email="">
 <environment>
   <os version="6.1.7600.0">Microsoft Windows NT 6.1.7600.0</os>
   <processor cores="4" />
   <clr bits="64" version="2.0.50727.4927" />
 </environment>
 <application version="0.3.0.0">Simis Editor</application>
 <source file="C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\Simis Editor\Editor.cs" line="185" column="5">SimisEditor.Editor.OpenFile</source>
 <details>C:\Program Files (x86)\Microsoft Games\Train Simulator\ROUTES\JAPAN2\carspawn.dat

> From 0x00000122 - data preceding failure:
>   wnerItem( "Jp1van.s" 6 )
>   CarSpawnerItem( "Jp1van2.s" 6 )
>   )
>  
>  
>
> From 0x000001A2 - data following failure:
>  
>
> > BNF has completed.
> >
> > Available states: .
> > Current rule: <none>.
> > Current state:
> >
> >    at Jgr.Grammar.BnfState.LeaveBlock() in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\JGR.Grammar\BNF.cs:line 175
> >    at Jgr.IO.Parser.SimisReader.ReadToken() in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\JGR.IO.Parser\SimisReader.cs:line 181
>
>    at Jgr.IO.Parser.SimisReader.ReadToken() in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\JGR.IO.Parser\SimisReader.cs:line 196
>    at Jgr.IO.Parser.SimisFile.ReadStream(Stream stream, SimisFormat& simisFormat, SimisStreamFormat& streamFormat, Boolean& streamCompressed, SimisTreeNode& tree) in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\JGR.IO.Parser\SimisFile.cs:line 74
>    at Jgr.IO.Parser.SimisFile..ctor(String fileName, SimisProvider simisProvider) in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\JGR.IO.Parser\SimisFile.cs:line 32

  at Jgr.IO.Parser.SimisFile..ctor(String fileName, SimisProvider simisProvider) in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\JGR.IO.Parser\SimisFile.cs:line 37
  at Jgr.IO.Parser.MutableSimisFile.Read() in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\JGR.IO.Parser\MutableSimisFile.cs:line 28
  at SimisEditor.Editor.OpenFile(String filename) in C:\Users\James\Documents\Visual Studio 2008\Projects\JGR MSTS Editor\Simis Editor\Editor.cs:line 185</details>
 <comments></comments>
</report>

One thing which this does not show is "attachments" - where the code calling the Feedback class specifies arbitrary extra data to include; these are sent as additional details but each with a name: <details name="extra stuff">...</details>.

The Simis file format with the 2nd-level Unicode text and binary Jinx formats are a pretty generic set of formats; they contain an arbitrarily nested tree structure with strings, integer and floating point numbers at any level. To actually interpret and describe the contents, a 3rd level of formats is needed.

As mentioned in both Simis Jinx Unicode Text File Format and Simis Jinx Binary File Format, this 3rd level of formats is identified by a letter and a number - and there are quite a lot of them. To actually define these formats in a useful way, though, we need to use another format - Backus-Naur Form (BNF). The exact format I've used is a variant of the standard Backus-Naur Form derived from the BNF files that shipped with Microsoft Train Simulator itself (in the UTILS\FFEDIT directory).

Train Simulator Backus-Naur Form

The BNF files are text; new lines have no significance; any of ASCII, UTF-8 and UTF-16 character encodings can be used, provided a byte order mark is included to identify UTF-8 and UTF-16. The files are made up of a number of definitions and productions - in any order - and a special termination marker.

Definitions specify a shared or standalone expression. Any other expression can reference it and has their reference expanded to the expression on the right-hand side of the equals ("=").

Productions specify, through the expression on the right of the arrow ("==>"), what is allowed/expected inside the block identified by the name on the left.

The expressions in both definitions and productions contain a space-separated list of items, each of which can be:

• A string literal, e.g. "Activity".
• A pre-defined data type, e.g. :sint. Available data types:
  • uint
  • sint
  • dword
  • string

  Data types can additionally be named, by including a comma and identifier after the type, e.g. :sint,TileX.

• Another production or definition, e.g. :Tr_Activity.

There are three operators allowed within expressions:

• Square brackets, denoting an optional section, e.g. [:Description].
• Curly brackets, denoting a repeatable section (1 or more times), e.g. {:UiD :SidingItem}.
• Pipe symbol, denoting a choice between sections, e.g. :Engine|:Wagon.

The choice operator (pipe) binds tighter than whitespace. Therefore, the expression :foo :bar|:baz means "foo followed by either bar or baz".

The end of an expression is denoted by a period (".").

Comments can be placed anywhere whitespace is allowed and use the common multi-line comment syntax of "/*" to start and "*/" to finish.

Termination of the BNF is indicated by the identifier "EOF". Everything after this is completely ignored.

3rd-level Format BNFs

Here's the current route car spawn.bnf as an example:

/* File format information */
FILE                          = :uint,Count [{:CarSpawnerItem}] .
FILE_NAME                     = "Route Car Spawn" .
FILE_EXT                      = "carspawn.dat" .
FILE_TYPE                     = "v" .
FILE_TYPE_VER                 = "1" .

/* Base types */
CarSpawnerItem                ==> :string :uint .

/* Format types */

EOF                           /* End of file */

All BNFs for the tools are required to have the five definitions shown above, so that the various programs can use them. FILE_TYPE and FILE_TYPE_VER are the letter and number (both as strings) used in all Simis Jinx files. FILE_EXT is either a file extension (e.g. "act") or a filename (e.g. "carspawn.dat") which selects which files can contain this format. FILE_NAME is a name suitable for displaying to the user. FILE is an expression representing the root of the file - the base of all parsing.

Binary Block Type Names

While the BNFs define what is allowed where, there is still one remaining problem for the Simis Jinx Binary format - each block type is identified by a number, not a string. For this, we can turn to some other files included with the original Train Simulator - the files in UTILS\FFEDIT.

• sidn.txt defines a few base IDs, including "core" and "train" (0 and 4 respectively).
• coreids.tok contains a list of all core "tokens" - i.e. block type names - in order of the numerical value.
• appids.tok is a C header which includes forms.hdr and loadstr.hdr with a token defined before and after each inclusion.
• forms.hdr and loadstr.hdr contain lists of all MSTS tokens in numerical order.

To construct the 32bit unsigned number used in the Simis Jinx Binary file format, the base ID and the token ID (from its position) are combined with the base forming the most significant 16bits and the token the least significant 16bits. E.g. the 7th "train" token would be 0x00040007.

Conclusion

Together with the BNFs, the number-block type name mapping completes the picture for loading and saving Simis Jinx files. However, as the BNFs are of my own construction, they are necessarily incomplete and possibly still inaccurate in some areas. This has improved a lot over the past few months, and will continue to do so, providing a good, solid and generic reading and writing capability for most Simis Jinx files.