.MAP Referencing Shared Assets In A New Map [Tag Injection]

[Lord Zedd 1,771]

Note: This process has since been automated in Assembly, so unless you want to learn how to do it manually you can skip this tut.

 

Alright, so here's the tutorial a lot of you have been waiting for.
Going to also be honest with you in saying that there aren't many "treasures" to be pulled from shared.map, so there may not be as many practical multiplayer applications of this tutorial as one would like until loading from campaign.map in MP can be figured out.

It should also be noted that this tutorial should work on any Halo game running on the 3rd Generation Blam Engine, though may require some tweaks depending on ZONE and PLAY layout differences.

What you will need:

• Assembly (with Updated Plugins, posted in conjunction with this tutorial)
• Time
• Hex Editor
• Time
• Patience
• Time
• MapExpand - Powered by Blamite (Not the original one)
• Also, Time
• Turning on Invisibles in Assembly is also necessary. You don't want to accidentally miss something like Gamecheat did, now would you?

Outline:

•  
• Deciding on the asset(s) to add.
• Destination Map Planning.
• Rearranging ZONE and PLAY data based on plans.
• Filling in new chunks. - PLAY
• Filling in new chunks and data. - ZONE
• Filling in new tag data and the like.

1. Deciding On The Asset

When deciding on an asset, you must consider the destination map you plan on using and what kind of map it is/its limits.

Multiplayer Maps:
mainmenu.map - YES
shared.map - YES
campaign.map - NO

Eeeeeverything Else:
mainmenu.map - YES
shared.map - YES
campaign.map - YES

As you can see, and as is mentioned at the beginning of this post, Multiplayer maps are at a bit of a disadvantage in that they cannot (currently?) load from campaign.map, where admittedly a lot of the "goods" are.
Even further, a map like Forge World already contains a lot of what shared.map has to offer, having all the standard vehicles and weapons inside.

Aside from these things, shared.map is (currently) known to contain the following extras:
Longsword
Phantom

So yeah, not that interesting of a list, though there very well could be more out there. It just requires some looking into.

How would you go about looking into it, you ask? Why, AMD's Extraction Tutorial, of course! Parts 2 and 3, specifically, will help you navigate to an asset's Raw Page to check the Shared Cache Index (typo's been fixed, lol).
In fact, just read through his entire post. It really helps in understanding what is going on in this tutorial, plus it is less I have to explain here )

Another thing to account for is how intricate the object itself is tag-wise.
Thumb through the tags that make up your object and ask yourself how much of that you can tolerate fixing/adding manually, because you'll be linked to My Tag Duplication Tutorial for the final step, and fixing the tag data between maps is the hardest part of this tutorial, in my opinion.
So go ahead and pre-emptively read through that post too, if you already haven't.

The above information should help you decide on what asset(s) to add. If you just got scared off, don't worry, as Assembly should be able to automate this entire procedure in the future.

2. Planning Out Your Destination Map

With your asset(s) picked out, we need to grab the sizes of various blocks of data in ZONE and PLAY. In conjunction with MapExpand we will need to plan out just where these blocks are going to be duplicated so we can safely add new chunks to them. The following things need to be sized up:

Raw Pages Reflexive [play]
Segments Reflexive [play]
Tag Resources Reflexive [zone]
Global Zoneset's Required Raw Pool [zone]
Global Zoneset's Required Tag Pool [zone]
Information DataRef [zone]

-For reflexives, you can get the size by expanding the Information dropdown and multiplying the Count and Entry Size values together, then for the sake of this tutorial, converting the result into Hexadecimal.

-For the dataref, you can right click the name and choose "View Value As..." From there you can scroll until you find the UInt32 example. That will be your size. Convert that to Hexadecimal as well.

*If you are on Windows 7 or above, you can convert to hexadecimal with the Windows Calculator by setting it to "Programmer" mode via View>Programmer or the hotkey Alt+3.
It automatically starts in Decimal so paste in your number and click the Hex radio button to change it to Hexadecimal.

With the sizes, it is time to figure out just how to lay it all out.

These are the final sizes for a clean forge_halo.map, which will be used for example: (leading 0s added to be tidy)
x0005EC48 Raw Pages
x00019AB0 Segments
x0006A000 Tag Resources
x00000350 Required Raw Pool
x000005F4 Required Tag Pool
x001BF42C Information

So in forge_halo's case, you can get away with only expanding enough to house the new information dataref, relocating the others to information's original location with generous spacing for further expansion. This will leave you their original locations to use for tag data, and another block at the end of the old information.

Time to start editing the map. We will start by running MapExpand on it to get the space we need.
Copy the map to MapExpand's directory and then run the program from the command prompt. You can either navigate your way there, or if you have the MapExpand folder on your Desktop, you can hold Shift and Right Click to get a slightly different context menu that includes an "Open Command Window Here" option.

MapExpand works via the following command:

mapexpand mapname.map #

mapname.map is the filename of your map, and # is the number of new data "pages" you want to add in decimal.
Each "page" is x00010000 bytes long, making your input a multiple of that.
Since I am sticking to my above plan, I only want to expand enough to fit a new information dataref.
It's current size is just under x1C (28 decimal) pages. Using x1C will not allow for much expansion, so I'm going to go 1 more with x1D (29 decimal).

So, for this tutorial, the map is forge_halo.map and the number of pages will be 29. After running MapExpand you'll want to make note of a couple things before you close the window:


Before we move however we need to grab the original locations of everything for both future reference and so we can move everything in one go. I didn't ask to grab these before because using MapExpand would make all the file offsets incorrect (Memory Offset would not change though)

To be thorough, we will grab both the file and memory offsets of everything.

For the reflexives, you can expand the Information dropdown to get the memory offset, and right click the FIRST VALUE of the FIRST CHUNK (CHUNK 0) and choose View Value As to get the file offset, which will be at the top of the new window.


For the dataref you need to do some conversion. Right click the dataref and View Value As again like you did for the size, but now scroll down to the Raw Examples. At 24 characters in/12 bytes, the following 8 characters/4 bytes are your memory offset. Note that down.


In the main Assembly window, Click "Map Information" and double click the "Map Magic" value. Note that down too.


Now do the following math:
Dataref Memory Offset - Map Magic
The result will be your file offset. Note that down.

These are the final offsets for a forge_halo.map expanded by 29 pages, which will be used for example:
FILE (MEMORY)
x05EC1220 (xBE0D0220) Raw Pages
x05F3E230 (xBE14D230) Segments
x05FF701C (xBE20601C) Tag Resources
x0606101C (xBE45E370) Required Raw Pool
x06061ACC (xBE45F370) Required Tag Pool
x0613CC80 (xBE34BC80) Information

3. Rearranging

First thing to do is get that dataref relocated. So open your expanded map in a hex editor (Do I even have to tell you to make backups?) and jump to the file offset for the Information data.

From here, use your editor's select block tool to select the entire dataref via the size you got. Copy and paste into a new document for safe keeping.


With the data safe in a new file, jump to the file offset mapexpand gave you and select another block there of the same size. Go to your new file, select all, copy it again, then go back to the map tab while everything is still selected and paste.

Due to the large amount of data being pasted in, your hex editor may say something about the undo-buffer; just hit OK and continue on.

At this point be sure to note down the new information offsets, close out the extra file you made to store the data, then save the map.

Let's do it again. But first, let's 0-out the original information data so we don't get confused. Head back to that offset, select the block again, but this time use the fill tool to fill the selection with 0s. Again ignoring any potential buffer messages.


And if you want to be extra careful, you can type something over the last few bytes that you just filled, like so:


Save again and jump to the Raw Pages file offset. And as you did with information data, select the entire block using your size, and paste into another file.

In the map tab jump to your newly-blank old information offset and again, select the size of the data you just copied. Go back to your storage file and copy everything, then paste over your map selection. Not even gonna mention undo buffer messages from here on out, you know what to do with them.

But unlike the information data, we are going to add an extra step here.

Upon pasting your data, your cursor should be at the end of the data. If so, then open up the jump/goto dialog and change the starting point to the current position. Now, the value you choose is up to you, and you should also be conscious of the other 2 blocks of data you will be adding in. For this example, x10000 is more than generous and will work fine.


Note exactly where you landed, or for tidiness you can jump to the beginning of the current row.


Noted? Good, because that's where we will be putting the next block of data, using the exact same procedure you've done twice now. Yay!

Jump to the Segments file offset. (FYI you are totally still in current position mode instead of from the beginning, so change that back before you embarrass yourself)

Copy the size, paste in a new file, jump to that Offset you just noted down before, select size, and paste over.

Jump forward from the end to another point. (I'm using x10000 again) And note that location, or the beginning of that row, whatever.

Again! Go to Tag Resources offset, select and copy, paste to new file, go to the fresh offset, select and paste. Jump ahead another whatever (x10000 again for me) and note down the point/beginning of the row.

Again! Go to Raw Pool offset, select and copy, paste to new file, go to the fresh offset, select and paste. Being much smaller than the others, we are going to jump forward a smaller amount (x1000 here). For my example I am jumping x1000 from where this data starts, rather than from the end of it like we've been doing. After doing so, note down the offset.

Once More! Go to Tag Pool offset, select and copy, paste to new file, go to the fresh offset, select and paste. This is also getting the same jump treatment as Raw Pool, so do that and note the offset. (I'm adding x1000 from the start again)

From this offset onward, this is free space to use for whatever. Note it down as such. Give another save and let's 0 out the original locations of those reflexives so they can be free space, too. We also need to math up the memory offset for those relocated ones.

Jump to the original pages offset, select the size, and fill with 0s.

Jump to the original segments offset, select the size, and fill with 0s.

Jump to the original resources offset, select the size, and fill with 0s.

Jump to the original raw pool offset, select the size, and fill with 0s.

Jump to the original tag pool offset, select the size, and fill with 0s.

Save again. You can now close the hex editor for now if you want, though we will be returning to it.

3 and a half. Fixing the mess you just made.

This is how our offsets look now:
name file offset (memory offset), size
new info x5CF1000 (xBDF00000)
new pages x613CC80 (xBE34BC80)
new segments x61AB8C0 (?)
new resources x61D5370 (?)
new raw x624F370 (?)
new tag x6250370 (?)
free space 1 x6251370 (?), ?
free space 2 x5EC1220 (xBE0D0220), x5EC48
free space 3 x5F3E230 (xBE14D230), x19AB0
free space 4 x5FF701C (xBE20601C), x6A000

There are a few ? that need filling in, so let's begin finding those.
For the memory offsets, those are simple. Remember how you got the dataref file offset? Now do that in reverse. That is;
File Offset + Map Magic = Memory Offset.

Do that for the first 3 ?s and...
name file offset (memory offset), size
new info x5CF1000 (xBDF00000)
new pages x613CC80 (xBE34BC80)
new segments x61AB8C0 (xBE3BA8C0)
new resources x61D5370 (xBE3E4370)
new raw x624F370 (xBE45E370)
new tag x6250370 (xBE45F370)
free space 1 x6251370 (xBE460370), ?
free space 2 x5EC1220 (xBE0D0220), x5EC48
free space 3 x5F3E230 (xBE14D230), x19AB0
free space 4 x5FF701C (xBE20601C), x6A000

This leaves us with finding the remainder of the old information data. We can figure that out with our notes, as such:

free space offset - beginning offset of old info = starting location of free space.
Now,
old info size - starting location of free space = size of free space.

Which gives us...
name file offset (memory offset), size
new info x5CF1000 (xBDF00000)
new pages x613CC80 (xBE34BC80)
new segments x61AB8C0 (xBE3BA8C0)
new resources x61D5370 (xBE3E4370)
new raw x624F370 (xBE45E370)
new tag x6250370 (xBE45F370)
free space 1 x6251370 (xBE460370), xAAD3C
free space 2 x5EC1220 (xBE0D0220), x5EC48
free space 3 x5F3E230 (xBE14D230), x19AB0
free space 4 x5FF701C (xBE20601C), x6A000

Great!

Now lets open up the map in Assembly so we can assign these new memory offsets.

Starting with PLAY, scroll down to Raw Pages and notice that everything is 0s now. To fix, Expand the Information dropdown and paste in your relocated offset. Upon clicking off the textbox everything should go back to normal.

Repeat for Segments and hit save. Open up ZONE.

Do the same for Tag Resources, Global Zoneset's Required Raw and Tag Pools, then the datare... Oops, Assembly cannot edit sizes and offsets of datarefs directly as of this writing, so back to the hex editor we go!

But first, hit save to save the update the other reflexives, then right click that dataref name and view value as. This time you are after the file offset you see at the top. This is the file offset for the actual dataref pointer. Note it down and close the map from Assembly.

Open the map in your hex editor and jump to your newly found file offset.
A Dataref is laid out as follows:


What we want to edit right now is that memory offset. Select the old, and paste the new one over it. Save and close, then check it in Assembly.

Now everything is safely relocated so you can...

4. Fill in new PLAY chunks

Now that everything is re-arranged and ready to have chunks added, we are going to do just that. This guide is going to become more disconnected depending on what you are adding, so I'll just go over this one single asset, and you can repeat as needed, k?

So let's get the Longsword model into our example forge_halo.

Starting with Raw Pages, expand the Information dropdown and add 1 to the Count value. Navigate to your new, blank chunk at the end. Note down the chunk while you are at it.

You are going to fill in your new Raw Pages chunk to match the one from the source map 100%. No exceptions. Then save.

Now add 1 to Segment's Count. Set the Required Page Index to the chunk you just made in Raw Pages, and use the same Required Segment Offset as the source map.
If your asset is a bitmap/uses the Optional values, you will need to add 1 more to your Raw Pages count and match the source's Optional page 100% as well, then set the Optional Index to that extra chunk. Save once more and PLAY is filled out! Note your new Segment chunk and head over to ZONE.

5. Fill in new ZONE chunks

Scroll down to Tag Resources and match everything from the original EXCEPT "Fixup Information Offset" and "Optional Fixup Information Offset" (Unless the Optional one is -1, which is okay to copy.) Since we haven't added any tags, leave the tag reference null, too.
This includes the 2 Fixup Reflexives. To do those, let's take one of our free space memory offsets and give it to "Resource Fixups" and set the count to match the source map. Now fill in each chunk. If there are 10+ chunks, you could also: (In spoiler)

In the source map, you can get the file offset for it, and do the math for total size like before. Then in a hex editor, select the size, and copy over an equal amount of space in one of your free space file offsets.

After the Resource Fixups is filled in, get the size of it using Count * Entry Size again and convert the answer to hex. Add that result to the memory offset you used, and give it to "Resource Definition Fixups" and follow the same procedure. Then get the size, convert to hex, add to the used memory offset, and note it down. Feel free to add both sizes to the file offset too. Save.

Time for the pools.

As was recently (finally) figured out, turns out pools work by assigning each asset and tag 1 bit, and grouping them in groups of 32 for the reflexive. We are editing the Global Zoneset so that our new content will be always loaded no matter what, making it easier to access ingame. We also have to do this anyway or all this new stuff won't even load.
So first we need to figure out if we even need to add any chunks or if we can get away with using existing extras due to the 32 bit groupings.

To find this out for the Required Raw Pool, take the total number of chunks for "Tag References", which will be in parenthesis after the #-# in the dropdown, and divide by 32. All decimal. It may help to switch out of Programmer mode so Windows Calculator doesn't round up/down on you.

So basically, if the result on the calculator is GREATER than the Raw Pool's chunk count by any amount, then you need to add one chunk.
If the result is by any amount LESS than the chunk count, add a chunk anyway to be safe

Once you have added your chunk, to be lazy, tick everything in both your new chunk and the one before it. Everything. Now you are safe from touching this again for 32 Tag Resource chunks! Neat!

For the Required Tag Pool, we are going to have to open our map in a hex editor again and go to the offset x2B4. The next 4 bytes make up the tag count. So select them copy them into Windows Calculator or whatever to view the bytes as decimal. In Hex Workshop you just select them;


Like Raw Pools above, divide that value by 32 and act appropriately. Same thing. Save again.

One last beast to tame in ZONE, which is the Fixup Information. This again cannot be modified in Assembly so make sure all is saved and close your map out of Assembly. Open your source map in Assembly if it wasn't already open, and go to the original Tag Resources chunk for your asset, then note down the Information Offset, Information Length, and Optional Offset (if applicable). Then scroll down to the source map's Fixup Information dataref and grab the file offset for its data. You remember how from the beginning of this tut, don't you?

Once grabbed, open both your source map and destination map in the hex editor. Take the offset(s) you got from the Tag Resources chunk and convert them into hex. Add the first to the dataref's offset and in the SOURCE map, go to the resulting offset. Select the length you got (Convert to hex, too) and copy to a new file. If there's an optional offset, take the converted offset from earlier and add THAT to the dataref offset this time and go to it. Copy the same length and paste in another new file. Go to your destination map.

Add the destination map's dataref size to its offset and jump to the result to quickly get to the end. Where you landed, select the length of the first information data you copied a minute ago, go to the first file you made that has the data, select all, copy, and paste over your selection in the destination. If an Optional exists, repeat the same from the end of that paste with the second new file and save.

Jump to the dataref location that we were at earlier to fix the memory offset so we can fix the size now.


As you can see, you should be right on the size, so take your hex-converted Information Length, and add it to the size. Note down the result. If Optional, add it again, noting the result again. Save once more and close the hex editor.

Open the destination map into Assembly again and go to your new Tag Resources chunk. Now we can fill in our offsets. The main Information offset will be equal to the original Information size, so convert it back to decimal and paste it in. For Optional, use the first size you noted when added the data a second ago. If no optional, the size you noted will be where your next data will be for future assets. If optional, the second noted size will be that offset. Save a final time and get ready...

6. Filling in dat tag data.

...For me to refer you to My Tag Duplication Tutorial! It was written with Ascension in mind, but if you've made it this far, you should be savvy enough to translate.
Sorry for this dickish last step, but seeing as I promised this tutorial almost a week ago, I figured everything above is the real injection portions. If you have any questions about translating the Duplication tutorial, post them in THIS thread.

Some things to note on your tag data journey:

• StringIDs will likely be wrong after porting. If you find your destination map lacks a particular string, don't fret. Find a similar substitute and remember to use that. Alternatively, you could just leave the stringIDS as they are after porting as anything else you port will match where necessary, which is what StringIDs are all about anyway.
• Some things CAN be winged, such as ignoring model markers and only porting ones you need to save time. Use it to your advantage.
• Don't forget to update the [zone] Asset Index value to point to the right Tag Resources chunk, and fix that tag reference in the chunk I told you to leave before!

And that's it!
Please don't hesitate to ask whatever you need to down below. I'll be happy to help.

[Guest Gamecheat]

i am the first person to do this

[D3FEKT 25]

i am the first person to do this

[vonkova 66]

That is one very long tutorial lol. Good job on it mate.

[AMD 885]

Awesome work.

And as Zedd said, Assembly will be able to automate this to the point where you only have to click a few buttons, so if you can't understand this tutorial or don't have enough time, you won't be left out. This can only get more complicated too as we figure out how to inject non-shared resources or resources from campaign.map.

EDIT: Something I'm not sure you mentioned is that the information data ref in zone HAS to be aligned on a 16-byte boundary (i.e. its offset in hex must end with a 0) or else it'll cause problems with the BSP and impostor models. It worked out anyway in this case but it's something that needs to be paid attention to.

[0xdeafcafe 226]

Great job

[JJIJR 666]

Awesome!!!!!!!!

[misterbanshee 51]

my head is going to blow up :/ u people are too smart for me by far lol also campaign.map into mp maps would be insane

[JJIJR 666]

my head is going to blow up :/ u people are too smart for me by far lol also campaign.map into mp maps would be insane

just follow it step by step and you should be fine.

[D3FEKT 25]

my head is going to blow up :/ u people are too smart for me by far lol also campaign.map into mp maps would be insane

i spent 2 hours reading it constantly and it's easy now since it's stuck in mah head.

[masterchafe 3]

this is awesome! I am glad you had to time to share it zedd

[misterbanshee 51]

massive head ache :/ but it was a good tut best one ive seen in a while

[Akarias 250]

My god, hex editing !

 

I just spent the last 1 and a half hours doing this for the first time. Following the exact same method of putting a long sword into forge world, it kinda worked. Model was disfigured but usable!

 

I would post some images but my falcon jtag just died on me, setting up a new one now. 

[misterbanshee 51]

is it possible to get sp stuff into mp maps yet ? or would that never be possible ?

[AMD 885]

is it possible to get sp stuff into mp maps yet ? or would that never be possible ?

It isn't possible yet but I wouldn't say it never will be.

[misterbanshee 51]

did u get stuff from shared.map into mp maps maybe if u get sp tags into shared.map we could get it into mp map just a idea

[JJIJR 666]

did u get stuff from shared.map into mp maps maybe if u get sp tags into shared.map we could get it into mp map just a idea

That's even harder. We'll probably learn to use campaign.map in the shared cashes before that.

[misterbanshee 51]

yea that would be making it harder to have to get a tag into one then _ take it from that one and add it to another lol