.MAP How To Extract Raw Pages

[AMD 885]

Assets (models, textures, sounds, etc.) in .map files are stored in what we informally call the "raw table." This is a special section of a map file which is reserved for only storing asset data (in other words, it's completely separate from the tag system) and it is split into several "pages." Each "page" in the table contains a set of compressed resources, and by extracting these pages, you can get at the raw asset data in a file. The intent of this tutorial is to show how these pages can be extracted in order to spark research on extracting assets such as models, textures, and sounds.

 

Note, however, that the raw asset data stored in these pages is stored in an extremely bare format which your standard model or image editor will not be able to load. Simply extracting these pages is not enough to extract models or textures, because the data in the pages also has to be parsed and converted back into a standard asset format first. This is beyond the scope of this tutorial, however, mostly because the specifics of converting the data back are not entirely known yet.

 

So, let's begin. In this tutorial, I'll show you how to extract the model data for an elite.

 

Part 1: The Tools

 

In order to extract raw pages from a map file, you will need the following tools:

• A map to extract an asset from, along with campaign.map, shared.map, and mainmenu.map
• Release version of Assembly (includes the necessary Reach plugins)
• A hex editor (I'll be using HxD here, but any decent editor will do)
• A "deflate" tool I made to decompress the pages (download, virus scan)

Part 2: Finding Asset Info in [zone]

 

Now that your tools are set up, open up Assembly and load the map which you want to extract the asset from. (In this example, I'll just use forge_halo.map.) Now, the first step to extracting an asset is to find its index in the "Tag Resources" reflexive in zone. So, we'll start by loading up zone in Assembly. The tag should be called "there they are all standing in a row" (oh Bungie).

 

 

The next thing you'll want to do is look for the "Tag Resources" reflexive. You can do that by using the search feature or just by scrolling down far enough. It should look like this:

 

 

What we need to do now is look for the entry in this reflexive which holds asset info for the the tag you want to extract the resources of. Hold up though - there are 6784 entries in this reflexive. You would have to be pretty bored to search through this thing by hand.

 

Of course, there's an easier way. Resource tags store the index of the entry in this chunk that refers to them. While not all plugins have this mapped out, mode does (and I believe Xerax has a snd! plugin which has this mapped out for sounds if you ask nicely). What you'll need to do is open up the resource tag you want to extract and search for a field called "Asset Index". Use Assembly's search feature to find it.

 

Here's the asset index field for elite.mode:

 

 

What this tells us is that entry 4764 in the "Tag Resources" reflexive in zone holds information about the model. Let's go there and make sure.

 

 

Success! As you can see, the "parent tag" field shows a reference to the elite model tag. Now we can use the values in this entry to determine some info about the asset. So far, so good.

 

Part 3: Finding Page Info in [play]

 

However, we're not done yet. Despite what some of those fields are named, this doesn't tell us much about where the page is located. To do that, we need to look in [play].

 

First, we need to find the raw table segment containing the resource page(s). We can do this by looking at the "[play] Segment Index" field in the reflexive in [zone]. It's at the bottom of the screenshot right up there. This value is an index into a reflexive in [play] called, well, "Segments". So, first, let's open up [play].

 

 

Now use Assembly's search feature or just scroll down to the bottom of the tag to get to the "Segments" reflexive. This reflexive tells us which raw page(s) the asset data can be found in. Go to the entry in this reflexive whose index is value of the "[play] Segment Index" field in [zone]. Since the segment index for the Elite model is 4584, we'll go there:

 

 

Cool. What this information tells us is that the elite model is located on page 13 and starts at offset 1134592 in that page. If "Optional Page Index" and "Optional Segment Offset" aren't -1, then it means that the resource data is split across two pages. If this is the case with your asset, we believe (although we're not 100% sure yet) that you will need to extract and decompress both pages and then concatenate them (simply put, you need to put the second page at the end of the first). Models usually only use one page, so this will be pretty easy.

 

So, where is page 13? To figure that out, we need to look at the "Raw Pages" reflexive two reflexives up. As you might have guessed, the page index field in the "Segments" reflexive tells us the entry index that we need to look at in "Raw Pages". So, let's go to entry 13 in the "Raw Pages" reflexive.

 

 

Nice.

 

Now, in order to extract the page, we need to know four things about it, all of which this reflexive will give us information about:

1. Which map file the page is in
2. Where the page is in that map file
3. Is the page compressed?
4. How large the compressed page data is

Points 2-4 come straight out of fields in this reflexive. The "block offset" field will tell us the offset of the page within the map's raw table, the "compressed block size" field will tell us the size of the compressed page, and we can determine if the page is compressed or not by comparing the "compressed block size" and "uncompressed block size" fields (if they're the same, then the page is not compressed).

 

So, we know the following things about this page:

• It's at offset 12898304 in the map's raw table
• The page is compressed, because the two sizes are different
• The compressed size is 1829654

The only thing left to determine now is which map file the page is located in. We can do this by looking at the "Shared Cashe Index" (yeah, I know, it's spelled wrong) field. If the value of this field is -1, then we're done and the page is located in the current map file. If it isn't however, which is the case here, then we need to look in a shared map file. To determine which file exactly it's located in, we use this value as an index into the "Shared Cashes" (also spelled wrong) reflexive near the top of the tag. Since the "Shared Cashe Index" field is 0, we just look at the first entry.

 

 

And there we have it! The page is located at offset 12898304 in mainmenu.map's raw table, it's compressed, and has a compressed size of 1829654. Awesome.

 

Part 4: Extraction

 

We're almost there! Now all we have to do is extract the page.

 

First, we need to open up the map that the page is located in and get its raw table offset. Assembly tells us this value in the "Map Information" tab.

 

 

As you can see, the raw table offset for mainmenu.map is 0x12E000. You can double-click on the value to copy it.

 

Next, we need to add the "Block Offset" field in play to this. Pull up Windows Calculator (or whatever you like to use for doing math) and do 0x12E000 + 12898304 (remember that the second value is in decimal). If you're following along with me and you did that correctly, you should get 0xD7B000. This is the file offset of the page!

 

So, open up the map containing the page (mainmenu.map in this case) in your hex editor and go to that offset. If you did this correctly, you should probably see some padding before the data (although there technically doesn't always have to be any).

 

Now, we need to select the page and save it to a file. Your hex editor should probably have some sort of "select block" feature in it which you can use to do this. The number of bytes you should select is the "compressed block size" value in play.

 

 

Again, if you did this correctly, you should probably see some padding after the data.

 

 

Copy this and save it to a file (I'll just put it on my desktop and call it "page13_compressed.bin").

 

Finally, since the page is compressed (you can skip this if it's not), we need to decompress the page. Remember that deflate utility I had you download earlier? Open it up. Browse for the file you just saved, and then browse for a file to save it to. Click the "Deflate" button. If everything goes well, you should get a "Done!" message.

 

 

Aaaaand we're all done! All you need to do now is open up the decompressed file in your hex editor and go to the offset specified by the "Required Segment Offset" field in the "Segments" reflexive in play (because the page will usually contain more than just the asset we want). In this case, it's 1134592. Note: chances are that there won't be any padding before the data this time.

 

 

Enjoy your raw asset data!

 

 

A note, though: the size of this data isn't stored anywhere to our knowledge. You have to just interpret the data based upon info in the original tag.

 

Anyway, I hope you learned something from this, and I hope that you'll be able to use this to figure out how to extract or even inject assets. If you have any questions, let me know.

 

Also, future updates of Assembly will probably be able to automate this process for you.

 

Happy modding!

[ddxcb 14]

I get this

 

System.IO.InvalidDataException: Block length does not match with its complement.

 

when trying to deflate.

[iBotPeaches 12]

I get this

 

System.IO.InvalidDataException: Block length does not match with its complement.

 

when trying to deflate.

Must of grabbed an invalid block then, as it worked fine for me. Unless you discovered another type of compression and there different, but I doubt that. So just double check you got everything

[AMD 885]

I get this

System.IO.InvalidDataException: Block length does not match with its complement.

when trying to deflate.

Just checking, you're not trying to extract from H4 are you? H4 uses a different compression scheme (LZX?) which I haven't been able to figure out yet.

Otherwise check your offsets.

Also, this is unrelated, but if Assembly crashes when you try to open the shared map to get the raw table offset, the raw table offset is stored at offset 0x470 in the map (in Reach at least) and you can just get it with your hex editor.

[Lord Zedd 1,771]

H4 uses a different compression scheme (LZX?)

Not supported by xbdecompress' LZX in the SDK.

[JJIJR 666]

How do i get the raw table offset for campaign.map if assembly doesn't open it?

[ddxcb 14]

How do i get the raw table offset for campaign.map if assembly doesn't open it?

 

what amd said, "the raw table offset is stored at offset 0x470 in the map (in Reach at least) and you can just get it with your hex editor."

 

also @amd, no there was something wrong with my net framework. It works now.

[JJIJR 666]

what amd said, "the raw table offset is stored at offset 0x470 in the map (in Reach at least) and you can just get it with your hex editor."

 

also @amd, no there was something wrong with my net framework. It works now.

my bad, didn't see that

[M0RPH10U5 0]

what amd said, "the raw table offset is stored at offset 0x470 in the map (in Reach at least) and you can just get it with your hex editor."

also @amd, no there was something wrong with my net framework. It works now.

How did you fix this, because I'm having the same issue, where it says:

System.IO.InvalidDataException: Block length does not match with its complement.

Edit:

I'm on Windows 8, but I can use a virtual machine to do this if I have to.

Edited April 23, 2013 by M0RPH10U5