Modding Some Observations On The Console Game Files

[selyb 117]

I was just thinking about the lack of support for console versions of Skyrim files and decided to share some things I have learned.

ESM files

Mod files (plugin files) are collections of records which are divided into fields of data.

• Skyrim.esm for the 360/PS3 is about 100MB larger than the PC version. From what I have seen with a hex editor, it looks like it has an extra REFR field for nearly every record.
  
• Skyrim.esm for 360 and PS3 are identical and the data is stored in Big-Endian
  
• The console esms also contain many XCGD fields which I'm convinced have to do with the ground. If we use any mods that change the ground but don't contain XCGD entries, the changes don't stick. I noticed this while using the PC Hearthfire.esm.
  
• The official DLC esms are stored in Little-Endian but are still much larger than the PC versions with extra REFR and XCGD fields.
  
• TES5Edit will open and edit/clean these official DLCs but it complains about the XCGD entries
  
• Update.esm is identical on all platforms.
  
• The Xbox360 will load and use the PC Skyrim.esm but it loads incredibly slow and constantly pauses in-game.
  

BSA files

BSA files are archives of files much like zip or rar.

• The format for bsa archives for PC or console is identical in every way except half of the filename hash is stored in Big-Endian.
  
• PS3 Skyrim requires the Xbox360 option in it's archives.
  
• Files in the bsa are sorted by its hash so to convert from PC to console would not only require byte-swapping the hash but also reordering the files.
  
• If you choose to compress an archive it will use zlib compression.
  
• If you choose Xmem Codec with compression, the files will individually be compressed using the Xmem codec from XNA v4.0
  
• If you choose Xmem compression and you have tiny files (seq files for example) Archive.exe will crash with no error
  

Textures

There are others who know quite a bit more than I do about the different types of textures and if anyone wants to chime in, I'll add their input to this list.

• PC uses DDS (Direct Draw Surface) textures while consoles use a custom DDX format.
  
• Textures that end with '_n.dds' are called normal maps and these are used by the game to give 'texture' to object (bumps on rocks or crevices in furniture)
  
• Xbox360 DDX files
  • completely different from PS3 DDX files.
    
  • always have a header of 0x4C bytes
    
  • The last 12 bytes are 3 UInt32s which indicate the sizes of up to 3 compressed textures.
    
  • The first 4 bytes of each compressed texture is a UInt32 indicating the size of the uncompressed texture.
    
  • Each compressed texture is compressed using Xmem from XNA v4.0

  [*]PS3 DDX textures

  • have a header that looks much like a standard DDS file
    
  • like the Xbox360 version, will contain 1, 2, or 3 compressed DDS textures
    
  • each compressed texture is compressed using zlib

Meshes

Meshes are 3D objects. I know little about them but here is what I do know.

• You can use the creation kit to import an object created in blender etc to Skyrim's version of Gamebryo's nif file.
  
• The console version of these are very similar in format but much or all of the data is stored in Big-Endian.
  
• Consoles will happily use PC meshes but our consoles need some different data to handle scaling. Scaling is where a mesh is resized to fit different body sizes. Scaling happens on body clothing/armors and arm/hand clothing/armors.
  
• Until someone finishes Nebby85's work on reversing the console nif format or someone leaks a tool to convert them, we will never have custom armor or anything else that scales (some custom new animals for example)

Edited December 12, 2012 by selyb

[Haxxzor1 4]

Thanks for this, nice to have it all laid out like that. I was trying to research some of nebby85's work with Nifskope and found this

this is my first time using python so there is probably a 'cleaner' way, but this lil program builds morph offset data, which is basically the difference between the FP32 vertices data from a _0.nif and _1.nif which is then converted to an FP16 and stored as a separate file which can then manually be imported to the _0.nif file under the NiTriShape block

    import struct
    import sys

    def loop(count,file_offset):		   # iterate through data
	    count += 1
	    offset = file_offset+(count-1)*4
	    # find and read int32
	    data_0.seek(offset,0)
	    FP32_0 = data_0.read(4)
	    data_1.seek(offset,0)
	    FP32_1 = data_1.read(4)
	    # gets vertices difference
	    f0 = struct.unpack('f', FP32_0)
	    f1 = struct.unpack('f', FP32_1)
	    f = f0[0]-f1[0]
	    # converts from FP32 to FP16
	    i = struct.pack('f', f)
	    I = struct.unpack('I', i)
	    h = FloatToHalf(I[0])
	    FP16 = struct.pack('H', h)
	    # writes to file
	    data_BMO.write(FP16)
	    if count < (vertices*3):
		    loop(count,file_offset)

    def FloatToHalf(i):
	    s = int((i >> 16) & 0x00008000)				  # sign
	    e = int((i >> 23) & 0x000000ff) - (127 - 15)	 # exponent
	    f = int(i & 0x007fffff)						  # fraction
	    # need to handle NaNs and Inf?
	    if e <= 0:
		    if e < -10:
			    if s:								    # handle -0.0
				   return int(0x8000)
			    else:
				   return int(0)

		    f = (f | 0x00800000) >> (1 - e)
		    return int(s | f >> 13)
	    elif e == 0xff - (127 - 15):
		    if f == 0:								   # Inf
			    return int(s | 0x7c00)
		    else:									    # NAN
			    f >>= 13
			    return int(s | 0x7c00 | f | (f == 0))

	    else:
		    if e > 30:								   # Overflow
			    return int(s | 0x7c00)
		    return int(s | (e << 10) | (f >> 13))
    if __name__ == "__main__":
	    vertices = input('Number of Vertices: ')
	    file_offset = input('File Offset to Start of Vertices: ')
	    count = 0

	    nif_0 = raw_input('File and Path of _0.nif: ')
	    nif_1 = list(nif_0)					 # assumes same path as _0.nif
	    nif_1[-5] = '1'
	    nif_1 = ''.join(nif_1)
	    BMO = list(nif_0)
	    BMO[-5] = 'B'; BMO.insert(-4, 'MO')
	    BMO = ''.join(BMO)

	    sys.setrecursionlimit(vertices*3+10)    # maximises recursion limit

	    data_0 = open(nif_0, "rb")
	    data_1 = open(nif_1, "rb")
	    data_BMO = open(BMO, "wb")			  #overwrites file if already exsists
	    data_BMO.close
	    data_BMO = open(BMO, "ab")
	    # writes Header
	    Header = '\x47\x61\x6D\x65\x62\x72\x79\x6F\x20\x46\x69\x6C\x65\x20\x46\x6F\x72\x6D\x61\x74\x2C\x20\x56\x65\x72\x73\x69\x6F\x6E\x20\x32\x30\x2E\x32\x2E\x30\x2E\x37\x0A\x07\x00\x02\x14\x01\x0C\x00\x00\x00\x01\x00\x00\x00\x53\x00\x00\x00\x08\x4E\x65\x62\x62\x79\x38\x35\x00\x01\x00\x1E\x20\x50\x45\x20\x53\x6B\x69\x6E\x6E\x65\x64\x20\x47\x65\x6F\x6D\x65\x74\x72\x79\x20\x28\x44\x69\x73\x6D\x65\x6D\x29\x00\x01\x00\x1B\x00\x00\x00\x42\x53\x42\x6F\x64\x79\x4D\x6F\x72\x70\x68\x4F\x66\x66\x73\x65\x74\x73\x45\x78\x74\x72\x61\x44\x61\x74\x61\x00\x00'
	    Block_Size = struct.pack('I', (vertices*3*2+8))
	    Num_Vertices = struct.pack('I', vertices)
	    data_BMO.write(Header)
	    data_BMO.write(Block_Size)
	    data_BMO.write('\x01\x00\x00\x00\x03\x00\x00\x00\x03\x00\x00\x00\x42\x4D\x4F\x00\x00\x00\x00\x01\x00\x00\x00')
	    data_BMO.write(Num_Vertices)
	    loop(count,file_offset)
	    # writes footer
	    data_BMO.write('\x01\x00\x00\x00\x00\x00\x00\x00')
	    data_BMO.close()

I'm not too good with programming but here is the original thread where I found it. I remember seeing a script or something he made as well that allows Nifskope to read M$ meshes but can't seem to find it now.

[selyb 117]

I remember seeing a script or something he made as well that allows Nifskope to read M$ meshes but can't seem to find it now.

It's on the nifscope forums.

[Haxxzor1 4]

So I was obsessing over getting armor mods to work (again) and found this on the TES Nexus Wiki:

Skyrim has introduced a new feature for characters: the weight slider. You can set the weight of characters or NPCs from 0% (the weight slider is completely to the left) to 100% (the slider is completely to the right). The changes in the shape of the character are managed using 2 mesh files for each armour/piece of clothing (this also applies to the meshes that contain parts of the body):

• Whith 0% weight, the game uses the _0.nif
  
• With 100% weight, the game uses the _1.nif
  
• With any intermediate weight, the game combines the _0.nif and the _1.nif
  

An example of this is shown in picture Skyrim armour/clothing meshes and character weight.

This means that you will have to create two meshes for each of the armours/pieces of cloth instead of only one. In this tutorial we are going to create first the _0.nif armour and once it is finished we will adapt it to the _1.nif body

The _1 and _0 variants aren't the only nifs that have scaling issues. Other dynamic meshes such as custom animals or bows sometimes mess up. I have a converted weapon pack with a handful of added bows and some custom meshes work where as others don't. What I was getting at was, using the proper tools, could you somehow work with the an armor mesh and remove scaling functionality or perhaps make a character with weight set to min or max to test?

[pcaquercia 13]

I don't think that's entirely true, if i remember correctly some mods only have one of the meshes and they do work in game, if they scale or not i don't know

from this tutorial: http://wiki.tesnexus.com/index.php/Creating_an_armour_for_Skyrim._Part_2#Create_the_1.nif_armour_mesh

If you didn't do anything that would cause vertices to be out of order on the _1 mesh, but the game is still not properly processing the average weight mesh, make sure the trishapes you added are in the same order for the _0 and _1 mesh. Skyrim averages the meshes together based on their node's order in Nifskope, not based on their text name. (For example: You have a Trishape named "ribs" and under it a Trishape named "chest" on your _0 armour. On your _1 armour you have the same trishapes in the opposite order: "chest" on top with "ribs" under it. Skyrim will merge the "ribs" Trishape from the _0 armour with the "chest" Trishape from the _1 armour, which will result badly.

perhaps our meshes problems have something to do with that, attaching the shapes to the wrong nodes? Just a wild guess though

[selyb 117]

This is not something I have really dug into, but from what I read and understood of Nebby85's posts on NifTools, the main problem we have is the numbers are stored in Big-Endian. It is possible that if we could convert these numbers we could have working meshes. If someone with enough motivation and knowledge could compile niftools, we could have it do the byteswaps before saving the mesh and with this we should be able to open a mesh, save as, and boom: xbox 360 mesh.

Edit: This actually kinda makes sense. Someone on Xbox360iso mentioned that the meshes look like the points are indide out. When you byteswap numbers it often makes small numbers very large and vice versa.

Edited December 23, 2012 by selyb

[Haxxzor1 4]

This is not something I have really dug into, but from what I read and understood of Nebby85's posts on NifTools, the main problem we have is the numbers are stored in Big-Endian. It is possible that if we could convert these numbers we could have working meshes. If someone with enough motivation and knowledge could compile niftools, we could have it do the byteswaps before saving the mesh and with this we should be able to open a mesh, save as, and boom: xbox 360 mesh.

Edit: This actually kinda makes sense. Someone on Xbox360iso mentioned that the meshes look like the points are indide out. When you byteswap numbers it often makes small numbers very large and vice versa.

Yeah, I've scoured the internet for Nebby85 in hopes to find some answers. Ha. I think it ended with him asking how to get NifSkope to write in big Endian. I wish it were as straightforward as paying someone to rewrite NifSkope. Anyone know python? lol

[rankstar 0]

Hi all,

 

Anyone has a structure file for nif files in order to try a program that read a nif file and swap bytes from big to little endian?

 

Something like this:

 

 

Firtst 4 digits - Header

Next 8 digits - Offsets

....

 

 

Thanks

[selyb 117]

I could easily write a script to swap bytes. The problem I have is that many nif files are compressed and I can't script the compression. I think it would be best to modify niftools to read PC nif files and save xbox360 files.