Adventures in the Blam! Engine

Script Expressions: electric boogaloo eiditon

If you manage to actually read this and apply it to something give yourself a pat on the back you incredible bastard.

Writing this as a blog as it applies to all halo games.

PAGE INDEX (Get the pun? you will if you manage to read this all):

1. Expressions

2. Manually navigating expressions

3. Creating a script from scratch

1. Expressions

<--- an example of a blank, invalid script expression. Note my plugins are different to the main build of assembly.

A quick run down:

Expressions are located in the scenario tag of your desired map. Use Ctrl+F to search for things quickly. The Scripts & Globals are also located here. These entries point to a starting expression for the script. 

Script String Data is also located in the scenario, but its not necessary for normal usage. But if you want any decompiler to be able to read quoted text "sounds\xboxchaos\yeet", you will need to add them to this table and set the String Table Offset in the expression to match this. Again, not required. 

An expression is x18 (24 Decimal) in length, use this to your advantage for calculating bits and bobs when editing large amounts of expressions as raw hex. 

A salt is for all intensive purposes just another number that you will need to take into account when writing a datum. (You don't really need to worry about this being unique, just leave any you write as 0. Makes life easier)

A index is the position of an expression in the expressions block. (The little number that you see in the top right hand side of a block)

A datum is the salt+index. If we think of a datum as a C#  we would write it like so:

Quote

        public DatumIndex(ushort salt, ushort index)
        {
            Salt = salt;
            Index = index;
        }

        public uint Value
        {
            get { return (uint) ((Salt << 16) | Index); }
        }

(SOURCE: Blamite\Blam\Scripting\ScriptExpression.cs)

('Value' being the both the salt and index represented as a unsigned 32 bit integer) 

Next I will explain the structure of an expression, as you see in the picture above:

• Salt: one half of the Datum. It's base value will be 58226 + the index of the expression (As per any scripts compiled by Bungie). You can leave it as 0 for anything you write manually.
• Operation Code: this defines the overall action of the expression, such as the function code to be used when the Expression Type is set as Function. (See the scripting xmls for a list)
• Return Type: Does this expression return anything to the expression that came before it? This will be void most of the time unless its required.
• Expression Type: Defines the expression type from a few options, you will be using 'Group' and 'Expression' most of the time.
• -
• 'Group' expressions represent a pair of parenthesis '( )' along with any expressions that will be placed inside them '( abc )' & any expressions that will be placed outside '( abc ) def'.
• 'Expression' expressions represent everything else other than references to other scripts/globals/parameters. You will be using this a good 80% of the time.
• -
• Next Expression Datum (Salt & Index): For your convenience you will see the NextExpressionDatum as two separate values. The next expression to be looked at after the current one. (This can be confusing when dealing with Group expressions, explained later)
• String Table Offset: a simple offset that is created by the script compiler. It stores the original text that was used to create this expression. (The node/atom depending on what you like to call it) 
• Expression Value/Data: this is important and is used any time an opcode/group expression requires extra data. This data can be simple numbers, stringIDs and even a Datum for Group expressions!
• Line number: do I really have to explain this? Well actually yes. Unless you are using the 'scripting' build of assembly from my GitHub this will always need to have a value of at least 1 or the decompiler does not show it. In Bungie's compiled scripts any expressions with a line number of 0 are implied  expressions, this can make navigating a set expressions manually without my build of assembly a huge headache as there will be more expressions than you expect to actually find. This is where people always get confused and give up. Examples of implied expressions are things like 'begin' actually existing at the start of every compiled expression. 

2. Manually navigating expressions, following the expression tree by hand

Now that you know what an expression is made of, we can finally learn how to follow an expression tree just by staring at them in assembly (or if you are sadistic as hell, in a hex editor). I recommend that you use my 'scripting' build of assembly for the purposes of learning and not having to take implied expressions into account. After you have got the hang of things you will be able to read through implied expressions and realise where they do and don't exist.

First find a script to actually follow, I will be using 'objective_3_clear' from citadel in Halo 3.

 <--- The comment at the top is turned on using the 'Show Extra Information' button inside the options drop down of the Scripts page.

 <--- If you don't see this, you are not on the 'dev' or 'scripting' build.

So let's go to our scenario tag, jump to expressions and goto the index specified by the comment. Alternatively goto the 'Scripts' block in the scenario tag and find your script there. It will have the Datum for your script.

You can click on the block inside in the top right and type the number to instantly jump straight to it just like you can in windows explorer with file names. Very useful for maps with 40,000+ expresions... Yikes.

So let's take a look at the first expression and decipher what it does... I'm fucking lost? The Next Expression index is 65535? What?

Hol' up fam. Dont wet your panties just yet. This is a group expression. Remember that a group has both expressions on the inside and potentially on the outside.

 <--- 65535 / xFFFF , simple means nothing. Meaning there is nothing outside of this group. This is true, look at your script.

Let's ignore anything other than the expressions in our script, the other pair of parenthesis is not actually an expression.

 <--- What we are interested in.

So yeah, having xFFFF on the Next Expression Index is correct but where is the over index for the 'begin' function expression? Its in the expression value! see where we have 4 8bit integers? Between x10 and x13? That's our Datum and where the next expression to be placed inside the parenthesis is located. But we only see -61, 35, 95 & -81. Those make no sense? Well they are actually correct, but due to our plugins we are reading them the wrong way.

Think back to what a Datum actually is: Salt+Index and both are unsigned 16bit integers. We can right click on the first int8 at x10  and click 'view value as...' to read from that offset as something else without having to edit our plugins. So let's do that now.

 <--- just like so, scroll down to the unsigned 16bit values... it reads two of them one after another, very useful for us!

So we can see that at x10 there is a uint16 with a value of 49955. That's the salt of the datum! You should be able to guess the second one at x12, that's our index that we need to go to in the expressions block. 

Here is a visual example in terms of indexes as to what we just read: ( 24495 ) 65535, So that's a group with an expression inside it that is located at 24495 and there is no expressions to be placed outside thanks to 65535 (xFFFF) meaning there are no more expressions to be used outside of this group. Neat. I hope you are still with me on this and not lost.

A general rule of thumb for Bungie compiled scripts is that the contents of a group are always placed directly after the group expression, with anything outside that group then coming after the expressions that were inside the group.

With this logic now in your head it should be obvious what comes next at 24495, a function with the opcode for begin. Just so happens the opcode for begin is 0. We are actually heading UP the index rather than down, this adds to the confusion. Let's take a look...

 <--- Yup, Expression with function_name and an opcode of 0. Yeetie.

Okay, this is straight forward. Let's follow the Next Expression Index. We will find another group expression at this index.

 <--- We started at 1, then went to 2 and now we are at 3 as per the next expression index of 2.

Here is what we find at 24485, the third expression of our script.

 <--- we have both a Next Expression index and a Datum in the value. Meaning there is expressions inside and outside the parenthesis. 

So this where things can be really confusing, where do you find the print expression? At the Next Expression Index or at the datum that is in the expression value? Where is the next line specified?

The next line in this case is specified by the Next Expression Index, this means we have a group expression that looks at another group expression and so fourth. This is how we navigate lines. Every new line has a group expression. 

 <--- Well because we want the expression INSIDE the parenthesis we follow the datum in the expression value, so view value as like we did before.

 <--- we can see that the print expression is at 24486, let's go to it....

 <--- here we are, a function with the opcode of 27. Print.

Before we go any further, we need to talk about functions. Most functions require an argument of some kind to actually be useful. If we take a look at our script xml file we can see that print requires and argument to work correctly.

 <--- While print is nulled in the release version of the game, this is not relevant for our discussion right now.

We can see the required argument is a string, so with this in mind we know that the next expression after our print function is going to be something string related. Let's take a look now by going to 24487.

 <--- Take note of the return type and opcode, both are set to string hopefully for obvious reasons. 

Note the Next Expression Index is now 65535 (xFFFF) and this means there are no more expressions. This is the end of the contents for the group expression as a result and thus the end of this line.

For strings the BlamScript engine will read the expression value section for a string table offset. Now when this script was compiled it placed the string for it at 42597. So if we read the value as a Uint32 we should see the same. 

 <--- indeed we see the same string offset in the value section

So you by now have likely caught on to the fact that the value section of an expression can really be anything as required and its meaning will change depending on the context. For example an AI expression can actually specify individual AI Units, Groups etc all using the same expression & just the values change to tell the script engine what to look for. SnipeStyle/AMD(?) did a lot of research on this many years ago for Halo Reach and deserves a mention for this. 

So the line is finished. Let's recap where we are in the index as we are going to go BACK to our line's group expression so we can find the next line. Savvy people will already know where to find the next group expression, but otherwise keep reading.

 <--- back at 3, we read the NEI for the next line

So we navigate to 24488 like so and are greeted by another group expression as expected.

 <--- we are now at the second print line

 <--- considering that we just manually read the same expressions, we are actually going to skip reading this line. We know what it is.

 <--- Let's look at the next line of the script by following the group expression 6's NEI. It will be yet another group at 24491.

  <--- LAST LINE OF THE SCRIPT YO.

So this is the last line of the script, we can tell this because the group for this line has its NEI set to 65535 (xFFFF). The BlamScript engine will see this as the script being finished, but let's continue by investigating the contents  of this group. Thus we must do our typical 'view value as...' or if you are savvy you will know that it is on the follow index. Let's take a look now.

 <--- a function with opcode 1100, objectives_finish_up_to. It requires a long (32bit integer) to work correctly.

So this function requires a long to work correctly, so with this knowledge the next expression specified by the NEI should be a long. Let's take a look...

 <--- Piece of cake, if we read the value as a Int32 from x10 we get 2. Lovely. never take what your plugins say for face value. 

 <---So that's it. We can see 2 down there just like we see in our script! 

Done. That's the end of the script!

Feel free to add extra lines to your plugins for making ready Values easier, such as adding two uint16 readers at x10 and x12 for Group Salt & Indexes. I leave it to you to decided how you want to go about this. Personally I use a hex editor with a data inspector and calculator as it's just faster for me. 

So that's pretty much it for reading scripts by hand, as long as you follow the rules of how group expressions work you will be perfectly fine.

Have a go at navigating a complex looking script and changing numbers. This is good practice you.

Enjoy reading nested expressions, it's a hell scape. Thanks LISP.

 <--- You should be able to mentally know where the group expressions are, have a think about it because you will be writing this manually soon.

 <--- This makes me hard just thinking about it. oof.

3. Making a script from scratch

So now that you can read scripts its about time you started writing them, that's what you came for right? Well bad news buddy if you just skipped everything that I wrote there you are a total jerk. Go read it. I wont answer any questions that you ask if you are not able to prove you can read expressions yourself, otherwise whats the point? If you cant read them you cannot understand how to write them.

I'm going to assume you are using a map with no scripts or you have removed all the existing scripts. I leave the allocation of Datum Indexes to your own discretion. I'm going to use Valhalla in Halo 3 as it has no scripts by default. 

Start by adding a block to the scripts block in the scenario tag. Give it a name and set the script type to Startup and the Return Type as void. We are not returning anything and this is normally used for more advanced scripts, such as a script that continuously returns a list of all objects that are inside a trigger volume to be used by another script. You can make some pretty advanced stuff. But for now we are sticking to the basics and doing a basic script that toggles the gravity between high and low every few seconds. We will do this using the physics_set_gravity opcode and the sleep opcode. 

So let's review the script before we begin:

 <--- We will set the gravity to 30%, wait 3 seconds, set the gravity to 130% for a nice slam dunk and then wait 3 seconds. The script will be set to continuous so it will repeat after it ends.

Your mind should already be racing, you already see exactly how many expressions you need and what they need to be set too. 14 Total. (Or I'm just really fucking weird because I can. I hope you can too.)

I will number them out below:

1. Group, NEI: Datum(65535,65535), Value: Datum(0,2)
2. Function Begin, NEI: Datum(0,3)
3. Group, NEI: Datum(0,6), Value: Datum(0,4)
4. Function physics_set_gravity, NEI: Datum(0,5)
5. Function Real, NEI: Datum(65535,65535), Value: Real(Float) @ 0.3
6. Group, NEI: Datum(0,9), Value: Datum(0,7)
7. Function sleep, NEI: Datum(0,8)
8. Function Short, NEI: Datum(65535,65535), Value: Short @ 90
9. Group, NEI: Datum(0,12), Value: Datum(0,10)
10. Function physics_set_gravity, NEI: Datum(0,11)
11. Function Real, NEI: Datum(65535,65535), Value: Real(Float) @ 1.3
12. Group, NEI: Datum(65535,65535), Value: Datum(0,13)
13. Function sleep, NEI: Datum(0,14)
14. Function Short, NEI: Datum(65535,65535), Value: Short @ 90

Note that '12.' is the last group expressions and as a result has a NEI of 65535 (xFFFF) effectively ending the script. Now I would write out an even longer section of this blog but instead I will show you a video of the process. 

[NOT DONE YET COME BACK LATER FAM]

Creating custom maps, the old school way.

A quick overview on how custom maps were made back in the Halo 2 Xbox days, applied to Halo 3.

This isn't a tutorial just a glance over.

We will be going through the process of creating a custom map but in the old school way of mulling out BSP geometry and using any objects that we can salvage from the single player campaign maps. So the first thing we need is some objects to actually use. Back in the days of Halo 2 we would use machines from the campaign as custom geometry and the basis for our custom map. So image a map from Halo 3, remove all its native geometry and you suddenly have a blank canvas that you can populate. It's important to note the bounds of the BSP and use the largest possible map and also take into account the maps physics size on the disc, as I want to have quite a lot of single player assets injected, namely quite a lot of AI related tags.

An example of using this method is shown in the platforms map that I showed a few months ago, the platforms are actually machines from 110_hc.map.

Ditto, in engine. Note missing pieces.

Something to take into account is lighting, broken lighting can be resolved by using fake light maps. Replace the existing bitmaps with blank ones, experiment for the best outcome.  

Plan things out. You can use a program such as 3DSMax for this along with Adjutant's import script to get all the objects you may need into a scene and pieced together. Set your unit setup to 100 inches and you will have one to one coordinates just like in engine. By far the best maps to use my opinion would be those from the DLC maps as they tend to contain all the weapons and vehicles that you may want. Don't forget about file sizes! The last thing you want is to be half way through a maps creation and run out of space from being a idiot and injecting too much garbage... Yikes.

Don't forget you can always scale and object's render model and physics model to make it larger or smaller! Lots of things can be built doing this.

So you have a target map and some objects to use for the basis of your maps geometry, lovely. Lets remove all the existing BSP geometry so we can get down to populating our map with new geometry. Take a look at your map's SBSP tag, before you can go about clearing it out you need to understand how the map is constructed. The bulk of a map is created from two things: Clusters and Instanced Geometry. Clusters are typically used for the displaced terrain of a map, think of the sand dunes on shrine. Instanced Geometry tends to be all the more complex and convex pieces of geometry, the multiple pieces of the tombs on shrine and the scattered parts of the crashed phantom are all instanced pieces of geometry. To give you a better idea I was working on a map called Thunderdome until I reached space issues (Hint Hint) and ended up moving away from it. Thunder dome is actually made from a single player BSP injected into sandbox and given a nice back drop with a enlarged scenery piece to fill up the backdrop. But on this BSP is a big ugly ass half of a boat that takes up a lot of space, I actually removed it from the BSP. How you ask? Instanced geometry.

This ugly thing.

Poof. No more ugly boat.

If you extract a BSP using Adjutant and import it to 3DSMax you can see all the various pieces of Instanced Geometry and various Clusters that actually make the map. Following this information you can selectively remove the geometry from the map just by scaling it down to 0. This also removes the collisions thankfully so no need to panic about dealing with MOPPs. With clusters things are slightly different and we will talk about them later. But as a quick experiment you can find a piece of Instanced Geometry in 3DSMax and then goto its block in Assembly, make a few changes and finally poke or save. Load up the map and observe the changes you have made. So with this logic in your head you can simply set the scale of all instanced geometry in a map to 0 as a means of removing almost all of its geometry, after that you can either manually remove the clusters or just set their rmtr bitmaps to invisible ones. You will now have a big empty map. Remove any extra spawned items like vehicles from their relevant blocks in the SCNR tag. Now its starting to feel really empty indeed.

Shrine with all its Instanced Geometry removed.

You can now get onto the hard part, building a new map from only things that you have injected or salvaged. Machines... objects... anything that has proper collisions will work. Add in some spawn points and you are on your way. Set a custom sky dome up add some weather effects and now you have something truly unique. This is how I used to do things years ago, there was very little information on forums combined with good old dial up I had to figure most of this out on my own back in the days of Halo 2. It's kind of stuck when since then so I thought I'd write out my process here for all of you to enjoy or mock. Yikes.

No one seems to really have done this for any of the modern games, so why not give it a go? make some stuff!