The Gremmar Warden

Nice work, and yeah this forum seems pretty dead

Thanks!
And yep, there are some more updates. First, someone informed me that the images are using a common PS2 file format, so it turns out there was no need to reverse engineer them after all. Instead, I turned to the model formats (probably a known format as well but I didn't know that at the time, and the only PS2 model ripping methods I see when googling it involve using a PS2 emulator). I stumbled across this reddit post that gave me a clue on where and how the vertices were stored. Starting with this knowledge, I was able to work out how most of the model data is stored (details in spoiler)

Spoiler

The model is stored as a series of vertex groups that look something like this in binary:

00 00 00 00  00 00 00 00  00 00 00 00  04 00 00 14

00 80 04 60  07 00 00 00  41 00 00 00  00 40 1E 30

00 C0 1E 30  05 01 00 01  00 00 00 20  40 40 40 40

04 80 07 78  98 AB 3F C0  BF BD 55 3F  D4 94 BF BE

EC C8 3D C0  FA A0 61 3F  59 05 D3 BE  C8 71 5A C0

42 12 62 3F  58 83 A8 BE  F7 4D 3F C0  19 92 71 3F

9A 8C BE BE  A9 D1 59 C0  4C B1 6A 3F  24 02 9F BE

FB 16 3E C0  CE C8 65 3F  28 2C 99 BE  6E 1A 59 C0

6A 35 65 3F  18 43 8C BE  05 80 07 7E  B5 E1 9F 00

D6 F8 89 00  B0 F3 9F 00  AC F3 A3 00  C4 6D 17 00

C6 06 70 00  CD 0A 73 00  06 C0 07 6E  80 80 80 80

80 80 80 80  80 80 80 80  80 80 80 80  80 80 80 80

80 80 80 80  80 80 80 80  07 80 07 64  18 1E D2 BE

1C D3 0D 3E  E2 1A D4 BE  5C 40 17 3E  54 36 D4 BE

80 1E 69 3C  AA C6 D6 BE  0C D7 0B 3E  88 A7 D5 BE

E0 A6 8D 3C  08 CD D4 BE  2C BD 15 3E  58 BC D4 BE

00 1F AA 3C  04 04 00 01  00 00 00 00  00 00 00 00

 

And here indented for clarity:

 

The dark green bytes here always start a vertex group's data. The red one shows the number of vertices in the group. This number is repeated several times throughout the data block as shown by the darker "07"s below.

00 00 00 00  00 00 00 00  00 00 00 00  04 00 00 14

00 80 04 60  07 00 00 00  41 00 00 00  00 40 1E 30 

00 C0 1E 30  05 01 00 01  00 00 00 20  40 40 40 40

 

04 80 07 78  -- The yellow bytes signify the beginning of the vertex data.

 

Each of these groups of 12 bytes is a vertex in (x,y,z) format, with each coordinate represented by a 4-byte float.

98 AB 3F C0  BF BD 55 3F  D4 94 BF BE  

EC C8 3D C0  FA A0 61 3F  59 05 D3 BE  

C8 71 5A C0  42 12 62 3F  58 83 A8 BE  

F7 4D 3F C0  19 92 71 3F  9A 8C BE BE  

A9 D1 59 C0  4C B1 6A 3F  24 02 9F BE

FB 16 3E C0  CE C8 65 3F  28 2C 99 BE  

6E 1A 59 C0  6A 35 65 3F  18 43 8C BE  

 

I am not sure what these bytes represent, seeing as there are only 4 bytes corresponding to each of the 7 vertices in this group. They do seem unique to each different vertex, however.

05 80 07 7E  B5 E1 9F 00  D6 F8 89 00  B0 F3 9F 00 

AC F3 A3 00  C4 6D 17 00  C6 06 70 00  CD 0A 73 00

 

I don't know what these are either, except that once again there are 4 bytes per vertex.

06 C0 07 6E  80 80 80 80  80 80 80 80  80 80 80 80 

80 80 80 80  80 80 80 80  80 80 80 80  80 80 80 80

 

 

07 80 07 64  -- These bytes show the beginning of texture mapping data.

 

Each set of 2 dwords (4 bytes) represents the (x,y) mapping of a vertex to the texture, each coordinate taking the form of a 4-byte float.

18 1E D2 BE  1C D3 0D 3E  

E2 1A D4 BE  5C 40 17 3E  

54 36 D4 BE  80 1E 69 3C  

AA C6 D6 BE  0C D7 0B 3E  

88 A7 D5 BE  E0 A6 8D 3C  

08 CD D4 BE  2C BD 15 3E  

58 BC D4 BE  00 1F AA 3C  

 

The green bytes always signify the end of a vertex group's data.

04 04 00 01  00 00 00 00  00 00 00 00

 

Although the vertex data can be clearly seen this way, there does not seem to be any explicit edge or face data in the files. Instead, the game does something else that I find both simple and ingenious:

 

It turns out, each group of vertices forms an elongated "strip" shape, like this:

 

But crucially, the vertex order causes them to form a zigzag:

 

And all it takes to form a triangle from two lines of the zigzag is to connect them together!

 

The same applies for the other vertices:

 

In other words, the vertex order allows each trio of adjacently numbered vertices to form a face!

Knowing how the vertex and face data works, it's easy to convert this into an .obj file format for any 3D model viewer to use.

And now having access to the model data, me and a few other people in a NiGHTS discord channel discovered that there is actually a HUGE amount of unused content. This is just one of many unused, likely prototype models in the game:

We put some of this content up on TCRF, and someone else offered add the rest of the model renders there. 

Some updates:

1. Actually, images can be rectangular, and many of them are.
2. Some images are encoded as 4-bit bitmaps, using only 4 bits for each pixel. This limits the color palette to 16 colors. They also appear to be scrambled in a different way than the 8-bit bitmaps.
3. After some trial and error, I think I've managed to figure out the meaning of certain header bytes. Here's an example:

These two blocks of bytes are the header data for one image. I believe the first one refers to the length and formatting of the pixel data itself, and the second one defines the length and formatting of the palette. However, some files (notably some starting with DITEM) have an odd number of header "blocks." I'm not sure what this means.

I'm not completely sure, but this blue byte seems to be connected with whether or not you need to double the width and height values to get the true size of the image. If 14, don't double, and if 00, do double. 

06 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00

04 00 00 00 00 00 00 10 0E 00 00 00 00 00 00 00

00 00 00 00 66 04 02 14 50 00 00 00 00 00 00 00         

00 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00

08 00 00 00 10 00 00 00 52 00 00 00 00 00 00 00        pink = width of image in pixels

00 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00        green = height of image in pixels 

04 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00

04 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00

Both the width and height values appear to be 4-byte integers, and are little endian (bytes must be read backwards). Thus, width here is 0x00000008 = 8, and height is 0x00000010 = 16.

This byte is connected with the type of palette. If 02, the colors are 16-bit, otherwise they are 24-bit.

06 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00

04 00 00 00 00 00 00 10 0E 00 00 00 00 00 00 00

00 00 00 00 BA 04 01 02 50 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 51 00 00 00 00 00 00 00

08 00 00 00 02 00 00 00 52 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 53 00 00 00 00 00 00 00

02 00 00 00 00 00 00 08 00 00 00 00 00 00 00 00

02 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00

These 2 bytes always appear to have the same value as each other, and seem to say whether or not an image is 4-bit (probably has something to do with the size of the palette). If 02, the pixels are 4-bit, otherwise they are 8-bit.