Silver's Simple Site - Weblog - Tags - Train Simulator

The original Microsoft Train Simulator was a great game at the time, and continues to be good fun (I'll save why it's better than Kuju's newer Rail Simulator for another time). It does, however, not entirely work by default on Windows Vista. There are a couple of issues:

• Resolution changes cause windows to be rearranged.
• On exit, it tries to create C:\FileIn.cns and C:\FileOut.cns and gets stuck in a loop because it can't.

You could work around the latter problem by running it as administrator, but that's clearly a substandard "fix". However, a much better solution which can solve both issues is the Compatibility Administrator in the Microsoft Application Compatibility Toolkit.

The Compatibility Administrator lets you tap in to a powerful collection of fixes already written by Microsoft for making other applications compatible with new versions of Windows. It also lets you view the fixes applied to each and every application in its system AppCompat database.

After installing the Application Compatibility Toolkit, you can launch the Compatibility Administrator as a standard user. For the purposes of this post, I'll create an AppCompat database just for fixing Train Simulator, but you can fix any number of programs with a single database, which simplifies deployment and installation.

1. Since it starts with a new database by default, just rename it by right-clicking and select "Rename". It really doesn't matter what you call it; I called mine "James Ross" in case I ever fix anything else with it.
2. Select the database, select the "Database" menu, "Create New", "Application fix...". This opens the wizard which will configure all the fixes we want for a single application.
3. Enter the details on the first page:
   • Name of the program to be fixed: "Train Simulator" (it's only for display).
   • Name of the vendor for this program: "Microsoft" (also only for display).
   • Program file location: browse to "train.exe" (e.g. "C:\Program Files\Microsoft Games\Train Simulator\train.exe"). This will be used later.

   

4. Now we get to pick from some standard fixes, including the commonly-used version lies. We don't want to select anything here, so ensure "None" is selected for the "Operating System Modes" and move on.

   

5. This is where the fun happens! There is a long list of possible fixes here, many of which can also take a command-line with options to control or limit the code that applies the individual fix. For the relatively simple problems with Train Simulator, we want to use two fixes:

   • CorrectFilePaths

     With this, we can redirect CreateFile API calls (among other file APIs) to actually access somewhere other than where the program passes. We use the following command-line:

     -b "C:\FileIn.cns;%UserAppData%\FileIn.cns" "C:\FileOut.cns;%UserAppData%\FileOut.cns"
     

     The -b (bare) turns off the standard mapping (which covers some system files that have moved over Windows versions), and then we specify two file corrections. Important to note that the double-quotes go outside the semi-colon separated pair of file paths. Also, we make use of one of the replacements (%UserAppData%), but note that these are not environment variables. There is a good weblog post about using CorrectFilePaths as well as some TechNet documentation.

   • ForceTemporaryModeChange

     This one is much simpler, and probably obvious from its name! It simply ensures that any resolution change is flagged as temporary, which prevents the window manager from shuffling things around.

   

   As you may have noticed, the wizard has "Test Run..." buttons on each page of options. You need to be an administrator to use them (as it installs an AppCompat database temporarily to do it), but they can be very handy in some cases.

6. Now we must tell Windows how it is supposed to identify that this set of fixes should be applied to a launching application. The wizard will automatically add the executable selected at the start, and tick some properties. In our case, there's no need for most of them and I unselected everything except PRODUCT_NAME.

   

   You could, if needed, add references to other files and decide which properties from them matter too; the fixes will still only be for the original executable, but it provides finer control over when to apply the fixes.

7. That's the AppCompat database created. You should now save it somewhere you'll remember (and be able to access from your administrator account).
8. Close Compatibility Administrator, and re-open it as administrator.
9. Open your AppCompat database, right-click it and select "Install".
10. Done!

Now, any time you run Train Simulator, the two fixes will be applied and joy will ensue.

Microsoft Train Simulator was made by Kuju Entertainment, formed by a management buyout of the Simis game studio in 1998, according to Wikipedia. Many of the files that form part of the game are in one of a number of formats bearing the "SIMISA" signature. For the purposes of discussing the file formats, I will only label the outer-most format the "Simis file format"; inner formats will be identified separately, later.

The Simis file format is identified a 16 byte or longer signature and header, with the remainder of the file being an inner format/data. The formats can all be identified by reading the first 8 bytes.

Uncompressed Simis File Format

 00000000   53 49 4D 49  53 41 40 40  40 40 40 40  40 40 40 40   SIMISA@@@@@@@@@@

Uncompressed files are simple; 16 fixed bytes ("SIMISA@@@@@@@@@@"), followed by the inner format/data.

Compressed Simis File Format

 00000000   53 49 4D 49  53 41 40 46  .. .. .. ..  40 40 40 40   SIMISA@F....@@@@
 00000010   78 9C                                                x?

Compressed files are slightly more interesting; 8 fixed bytes ("SIMISA@F"), followed by a 4 byte unsigned integer containing the uncompressed size of the remaining data, and another 4 fixed bytes ("@@@@"). The data itself is compressed using zlib's DEFLATE algorithm, which is identified by the two bytes following the 16 bytes header; all subsequent data is the raw DEFLATE stream.

Unicode Text Simis File Format

 00000000   FF FE 53 00  49 00 4D 00  49 00 53 00  41 00 40 00   ÿþS.I.M.I.S.A.@.
 00000010   40 00 40 00  40 00 40 00  40 00 40 00  40 00 40 00   @.@.@.@.@.@.@.@.
 00000020   40 00                                                @.

This format is a variation on uncompressed, but for a specific type of inner format: Unicode text encoded with UTF16-LE. In this situation, two things are done to the header:

• It is encoded as UTF16-LE characters like the inner format/data.
• A UTF16-LE byte order mark is pre-pended.

Inner Formats

Whether the Simis file is compressed or not does not affect the inner formats allowed; any format may be compressed or uncompressed with no differences beyond that compression and the header indicating as much. It is unclear, however, how the Unicode text format would be compressed so, as I have not found any examples within Microsoft Train Simulator, I am considering them distinct at this level.

This is one of the 2nd-level (inner) formats for the Simis file format used by Microsoft Train Simulator.

As I mentioned last time, the 1st-level (outermost) format has an adjusted header when working with this 2nd-level format; in particular, the file starts with a UTF16-LE byte order mark and the header itself is Unicode text encoded as UTF-16LE. Here it is again:

 00000000   FF FE 53 00  49 00 4D 00  49 00 53 00  41 00 40 00   ÿþS.I.M.I.S.A.@.
 00000010   40 00 40 00  40 00 40 00  40 00 40 00  40 00 40 00   @.@.@.@.@.@.@.@.
 00000020   40 00                                                @.

The 2nd-level format identifies itself with its own header, unsurprisingly, starting with the text "JINX0", two values indicating the 3rd-level (inner inner) format, the text "t" indicating the Unicode text variety of this format, some padding (6 underscores) and a newline.

 00000020         4A 00  49 00 4E 00  58 00 30 00  .. .. .. ..     J.I.N.X.0.....
 00000030   74 00 5F 00  5F 00 5F 00  5F 00 5F 00  5F 00 0D 00   t._._._._._._...
 00000040   0A 00                                                ..

Those 4 missing bytes are the two characters (a letter then a number) that identify the 3rd-level format used. As text, the header of these files looks like the following line:

 SIMISA@@@@@@@@@@JINX0..t______

Now follows the actual data...

Simis Jinx files all store data in a tree-like structure, where each tree node has a type and optional name, and is intermixed with values. In other words, each node's children can be both nodes and values, including a mixture. The structure is marked out by parentheses ("(" and ")") for blocks, with values being raw or quoted strings (generally speaking, any value with no whitespace and no significant symbols - quotes, parentheses - can be left unquoted). Let's get straight to an example from Microsoft Train Simulator, the file GLOBAL\gui.txt:

 SIMISA@@@@@@@@@@JINX0I0t______
 
 io_dev ( KEYB 0
     io_map ( T                 "sounddialog"        ALL_UP SHIFT_DOWN )
     io_map ( ESCAPE            "escape"             ALL_UP )
     io_map ( F1                "Help"               ALL_UP )
 )

The header has identified this as containing 3rd-level format "I0"; for now, though, let's focus on the 2nd-level format. There is a single root node of type "io_dev" (and it has no name) which contains:

• The value "KEYB".
• The value "0".
• Three nodes of type "io_map", each containing a selection of values but no further nodes.

This is just a simple example, but the format is pretty easy to read (although a little tricky to parse correctly); the 3rd-level format actually defines which node types and what nesting of them is allowed and which values should be where.

This is the second of the 2nd-level (inner) formats for the Simis file format used by Microsoft Train Simulator.

Unlike the text format, the binary format has a simple binary header - but it should look rather familiar. The binary header has the same 16 characters as the text header, but this time they're encoded as single bytes and one of the characters is notably different: the 8th character is "b" for binary, rather than "t" for text (if there existed a single-byte text format this would have been the only difference with this format in the headers).

 00000010   4A 49 4E 58  30 .. .. 62  5F 5F 5F 5F  5F 5F 0D 0A   JINX0..b______..

Just like in the text format, the 2 missing bytes in the middle are the two characters (a letter then a number) that identify the 3rd-level format used. We will get to those next time.

Note: If the 1st-level format is using compression, this header is the first item within the compressed stream; for simplicity, all my examples will be for uncompressed files.

Now follows the actual data...

As I started talking about last time, Simis Jinx files are basic trees; the binary format is nothing more than an alternative representation of the same data. It is, however, more of a challenge to read and write correctly - something the 3rd-level formats will help deal with.

Each node in the tree has an 8 byte header, consisting of a 4 byte unsigned integer identifying the node's type and a 4 byte unsigned integer specifying the length of the contents. The contents consist of an optional name - 1 byte for length plus UTF16-LE characters - for the enclosing node and the child values and nodes.

There are a number of common types of value included:

• Unsigned integer (4 bytes).
• Signed integer (4 bytes).
• Floating-point number (4 bytes).
• String (2 bytes for length plus UTF16-LE characters).

Let's have a look at an example, GLOBAL\capview.iom, but remember that to correctly parse this I am using the 3rd-level format:

 00000000   53 49 4D 49  53 41 40 40  40 40 40 40  40 40 40 40   SIMISA@@@@@@@@@@

The standard 1st-level header...

 00000010   4A 49 4E 58  30 69 30 62  5F 5F 5F 5F  5F 5F 0D 0A   JINX0i0b______..

2nd-level header indicating a binary version of 3rd-level formal 'i0'.

 00000020   63 00 00 00  64 02 00 00                             c...d...

Node type is 99 (0x63), contents size is 612 bytes (0x264).

 00000020                             00                                 .

Node has no name.

 00000020                                01 00 00  00 00 00 00            .......
 00000030   00                                                   .

Two unsigned integer values: 1 and 0.

 00000030      64 00 00  00 35 00 00  00 00                       d...5....

Node type is 100 (0x64), contents size is 53 bytes (0x35) and there's no name.

 00000030                                   CB 00  01 00                   ....

Unsigned integer value: 65,739 (0x100CB).

 00000030                                                15 00                 ..
 00000040   6E 00 75 00  64 00 67 00  65 00 5F 00  63 00 61 00   n.u.d.g.e._.c.a.
 00000050   62 00 63 00  6F 00 6E 00  74 00 72 00  6F 00 6C 00   b.c.o.n.t.r.o.l.
 00000060   5F 00 6C 00  65 00 66 00  74 00                      _.l.e.f.t.

String value: length of 21 (0x15) plus 21 UTF16-LE characters "nudge_cabcontrol_left".

 00000060                                   00 00  00 00                   ....

Unsigned integer value: 0.

The 1 byte for no name, 8 bytes for two unsigned integer values plus 44 bytes for the string mean the total contents are up to 53 bytes - that means it is the end of this node type 100.

 00000060                                                64 00                 d.
 00000070   00 00 37 00  00 00 00                                ..7....

Node type is 100 (0x64), contents size is 55 bytes (0x37) and there's no name.

Writing the above in text format would give:

 <node type 99> (
     1
     0
     <node type 100> (
         65739
         nudge_cabcontrol_left
         0
     )
     <node type 100> (
 ...

It's clear that we're missing the node type names found in the text files, and there's no indication whether some 4 bytes are a new node, an integer or float - some heuristics can work for this some of the time, I found, but in the end this is what the 3rd-level format is for.

Other 2nd-level formats

I've covered the two main Simis 2nd-level formats, but there are others; most notably, the texture files (.ace) are wrapped in a 1st-level Simis header with their own format inside. I won't be covering these other formats soon, as there are already tools that can handle these files sufficiently for Microsoft Train Simulator's needs, and the 3rd-level Simis formats are more interesting anyway.

The Simis file format with the 2nd-level Unicode text and binary Jinx formats are a pretty generic set of formats; they contain an arbitrarily nested tree structure with strings, integer and floating point numbers at any level. To actually interpret and describe the contents, a 3rd level of formats is needed.

As mentioned in both Simis Jinx Unicode Text File Format and Simis Jinx Binary File Format, this 3rd level of formats is identified by a letter and a number - and there are quite a lot of them. To actually define these formats in a useful way, though, we need to use another format - Backus-Naur Form (BNF). The exact format I've used is a variant of the standard Backus-Naur Form derived from the BNF files that shipped with Microsoft Train Simulator itself (in the UTILS\FFEDIT directory).

Train Simulator Backus-Naur Form

The BNF files are text; new lines have no significance; any of ASCII, UTF-8 and UTF-16 character encodings can be used, provided a byte order mark is included to identify UTF-8 and UTF-16. The files are made up of a number of definitions and productions - in any order - and a special termination marker.

Definitions specify a shared or standalone expression. Any other expression can reference it and has their reference expanded to the expression on the right-hand side of the equals ("=").

Productions specify, through the expression on the right of the arrow ("==>"), what is allowed/expected inside the block identified by the name on the left.

The expressions in both definitions and productions contain a space-separated list of items, each of which can be:

• A string literal, e.g. "Activity".
• A pre-defined data type, e.g. :sint. Available data types:
  • uint
  • sint
  • dword
  • string

  Data types can additionally be named, by including a comma and identifier after the type, e.g. :sint,TileX.

• Another production or definition, e.g. :Tr_Activity.

There are three operators allowed within expressions:

• Square brackets, denoting an optional section, e.g. [:Description].
• Curly brackets, denoting a repeatable section (1 or more times), e.g. {:UiD :SidingItem}.
• Pipe symbol, denoting a choice between sections, e.g. :Engine|:Wagon.

The choice operator (pipe) binds tighter than whitespace. Therefore, the expression :foo :bar|:baz means "foo followed by either bar or baz".

The end of an expression is denoted by a period (".").

Comments can be placed anywhere whitespace is allowed and use the common multi-line comment syntax of "/*" to start and "*/" to finish.

Termination of the BNF is indicated by the identifier "EOF". Everything after this is completely ignored.

3rd-level Format BNFs

Here's the current route car spawn.bnf as an example:

/* File format information */
FILE                          = :uint,Count [{:CarSpawnerItem}] .
FILE_NAME                     = "Route Car Spawn" .
FILE_EXT                      = "carspawn.dat" .
FILE_TYPE                     = "v" .
FILE_TYPE_VER                 = "1" .

/* Base types */
CarSpawnerItem                ==> :string :uint .

/* Format types */

EOF                           /* End of file */

All BNFs for the tools are required to have the five definitions shown above, so that the various programs can use them. FILE_TYPE and FILE_TYPE_VER are the letter and number (both as strings) used in all Simis Jinx files. FILE_EXT is either a file extension (e.g. "act") or a filename (e.g. "carspawn.dat") which selects which files can contain this format. FILE_NAME is a name suitable for displaying to the user. FILE is an expression representing the root of the file - the base of all parsing.

Binary Block Type Names

While the BNFs define what is allowed where, there is still one remaining problem for the Simis Jinx Binary format - each block type is identified by a number, not a string. For this, we can turn to some other files included with the original Train Simulator - the files in UTILS\FFEDIT.

• sidn.txt defines a few base IDs, including "core" and "train" (0 and 4 respectively).
• coreids.tok contains a list of all core "tokens" - i.e. block type names - in order of the numerical value.
• appids.tok is a C header which includes forms.hdr and loadstr.hdr with a token defined before and after each inclusion.
• forms.hdr and loadstr.hdr contain lists of all MSTS tokens in numerical order.

To construct the 32bit unsigned number used in the Simis Jinx Binary file format, the base ID and the token ID (from its position) are combined with the base forming the most significant 16bits and the token the least significant 16bits. E.g. the 7th "train" token would be 0x00040007.

Conclusion

Together with the BNFs, the number-block type name mapping completes the picture for loading and saving Simis Jinx files. However, as the BNFs are of my own construction, they are necessarily incomplete and possibly still inaccurate in some areas. This has improved a lot over the past few months, and will continue to do so, providing a good, solid and generic reading and writing capability for most Simis Jinx files.

I've just released the latest version of my Microsoft Train Simulator tools: Simis Editor v0.4 with the usual documentation. Some highlights for this release:

• Open and Save dialogs support full filename filters from BNFs (e.g. "tsection.dat") in addition to extension filters.
• Support for adding new blocks to the tree via context menu with 4 groups of operations:
  • Insert previous siblings.
  • Insert next siblings.
  • Insert before children.
  • Insert after children.
• Problems loading *.bnf files and loading or saving Simis files are all offered for reporting online (via the Feedback class).
• Added a status bar and help text for menu items.

Download Simis Editor v0.5 and read the documentation. Release highlights:

• New format support: Cameras, GUI, GUI Bitmaps, GUI Screens, Route Forests, Route Gantry Sets, Route Speed Post Sets, Route Tile Definition, Route Tile Definition Low, Route Track Types, Signals.
• Updated format support: Activity, Route Reference, Route Train Path, Shape, Train Consist, Train Engine, Train Wagon, World.
• Bug fixes and general improvements to underlying libraries.
• Thanks to Jeffrey Kraus-Yao for many of the format additions and updates.

Microsoft Train Simulator uses a custom file format for its textures and other in-game images - ACE (no, not the data compression format). They are stored as a 2nd-level format inside the Simis file format, which means they can be compressed with zlib's DEFLATE algorithm transparently.

I'm not going to document the format publicly just yet, but instead show a preview of loading ACE images in the next version of Simis Editor:

I have also implemented a DXT1 compression algorithm which, together with the decoder, allows Simis Editor to load and save ACE files in all three of their "formats":

• Uncompressed (A)RGB.
• Zlib compressed (A)RGB.
• DXT1 compressed.

While Simis Editor will not become an image editor in its own right, I will be supporting importing/exporting as standard image types (BMP, PNG, etc.) and the new "ImageFile" command-line tool will provide for easy bulk conversions. Stay tuned for more details.

With the release of Open Rails 0.7.0, now seems a good time to document some of the hidden settings and features:

• LoggingFilename (string setting) Specifies a static or generated name for the log file. Defaults to "OpenRailsLog.txt". This is a standard .NET format string, with the following replacements available: (New in 0.7.0)
  1. "Open Rails".
  2. Version number normally, build stamp if compiling yourself.
  3. Version number.
  4. Build stamp.
  5. Current date/time.
• LoggingPath (string setting) Specifies an absolute path to save the log file when running the game. The default is your desktop.
• ScreenshotPath (string setting) Specifies an absolute path to save in-game screenshots (using Print Screen). The default is the "Open Rails" folder in your pictures folder. (New in 0.7.0)
• ShadowAllShapes (boolean setting) Specifies whether to generate shadows from all objects in the game, instead of only those which the route creator has specifically set. This can often make a route more visually pleasing, at a small performance cost.
• VerticalSync (boolean setting) Specifies whether to limit the frame-rate to the monitor's refresh rate.
• ViewingFOV (integer setting) Specifies the field-of-view for the display in degrees, vertically. The default is 45, which is equivalent to 60 horizontally in a 4:3 ratio. Because Open Rails measures this vertically, you'll automatically see more on widescreen resolutions. (New in 0.7.0)
• Print Screen Takes a screenshot and saves it in the ScreenshotPath above in the PNG format. (New in 0.7.0)
• Alt Hold down the "Alt" key and click on a switch to throw it (change which path is taken).
• U Hold down the "U" key and click on a coupling to uncouple part of the player train.
• Control - E Switches between locomotives in the player train.
• Shift - Control - F Switches the direction of the player locomotive. This is a temporary substitute for the lack of MSTS Bin reverse cab support.
• Alt - F1 Dumps the command list with keys to "keyboard.txt" and a high-resolution keyboard image (suitable for printing) to "keyboard.png".
• Control - Alt - Page Up/Page Down/Home Controls the game speed, increasing, reducing and resetting it respectively.
• Control - Alt - F11 Toggles the dispatcher/signalling visualisation, which highlights the selected route ahead of the player and AI trains, with labels for switches and signals.

To edit hidden settings

All Open Rails settings are stored in the Registry, under the key "HKEY_CURRENT_USER\Software\OpenRails\ORTS". The usual disclaimers about editing the registry apply.

• To edit any setting, open Registry Editor (regedit.exe) and navigate the left side to "HKEY_CURRENT_USER\Software\OpenRails\ORTS".
• Some settings will appear already, some have to be created.
• To create a setting, either use the menu "Edit" > "New" or right-click the right side and use the menu "New".
  • String settings should be created as "String Value".
  • Integer and boolean settings should be created as "DWORD (32-bit) Value".
• To edit any setting, double-click its name on the right.
  • Strings can just be edited as-is.
  • Integer settings should be edited with the "Base" option set to "Decimal" in the edit window.
  • Boolean settings should use the value "1" for true and "0" for false.