JJ2 File Format Specifications
 

Edit: I just slipped this tileset extractor on top so people can see it.

Well, I was working on a file-format entry for the ERE, but the formatting never came out the way I wanted it. Seeing as I'm in somewhat of a rush, I've decided to dump my article here, in the hopes that someone can move it to ERE or some other public article.

Zlib Compression in JJ2
This article will give you a crash course to understanding zlib, how it can be compressed / decompressed and how it is implemented in JJ2. You need either zlib.lib and zlib.h if you’re programming for C++, or zlib.dll if you’re programming for Visual Basic. Armed with this knowledge alone, it is possible to decrypt JJ2 files into their raw formats, and then figure out the formats from there.

Declaring zlib functions
Just about every programming language requires you to declare a function before you can use it. Notes about declares: The compress function will not be used at all, and is only provided for closure. This is because it is superseded by compress2, which allows compression at higher levels. Also, if you do not care about the integrity of your files, you can ignore the crc32 function and skip all crc checks. Without further ado:

Declares in C++
Declares in VB
Don’t forget to add "Private"s in front of them if you’re going to use them in your forms.
How compression is used in JJ2 files
As a bare minimum, all zlib-compressed files in JJ2 files should at least provide the compressed and uncompressed sizes in its header. For example, consider an example zlib section of a file which goes:

0000 0010 0020 0030 0040

44 00 00 00 4F 00 00 00 78 DA E3 60 80 00 17 73 69 30 ED F0 53 98 41 E0 97 30 03 23 90 DD 92 53 C6 78 E7 87 30 C3 7B 31 47 06 35 89 03 FC 20 F9 04 0A DC DB 3C 07 B0 5A 01 55 39 06 95 04 25 86 2B 40 1A 0C 9E 30 00 00 2B 02 0E A6

D...O...X..`...^ %0..S.A..0.#...S .x..0.{1G.5... . ....<..Z.U9...%. +@...0...Q.[

In this self-made example, the first four bytes (which reads 68 in hex) represents the size of the compressed stream (the input data). The next four bytes (which reads 79 in hex) represents the size of the uncompressed stream (the output data). Armed with that data, we now know that we will be decompressing a 68-byte buffer into a 79-byte one. Note that zlib streams starts with a "78 DA" when compressed at the maximum level, and this can be very useful for skimming through large files to find out where a zlib stream begins or how many zlib streams there are.

In case you are already confused, a zlib stream is basically an array of bytes which contains compressed data, and is very unfriendly to the human eye. Fortunately, there is a very simple function which can be used to decompress a zlib stream into a buffer. This function is uncompress.

Using uncompress in C++
For this short snippet we will assume that you have the above "example" file stored as "tutorial.dat". This assumes you are writing a console program and have already #included anything necessary.

That wasn’t too bad was it? Once you can understand the concepts of malloc and uncompress, you can decompress just about every file format in JJ2. Don’t forget to free what you malloc though. Note that we pass a pointer to the second argument of uncompress, which will return with the uncompressed size of the buffer. Assuming you already had the right size in the first place, this value will not change. Also, the return value of uncompress determines whether the operation was a success or not. However, to keep things simple, we assume that our zlib stream is valid, and will not error-handle it. Note: I would usually prefer to use gzread(look it up on google), but chose uncompress because it is compatible with VB.

Using uncompress in VB
Here we use the same assumptions: a file tutorial.dat with containing that binary data.

The VB function actually looks slightly harder than the C++ version. One thing to note is that ReDim uses "x-1" bytes when you want to allocate x bytes, because the array goes from buffer(0) to buffer(x-1).

Compressing files
Overall, uncompressing shouldn’t be too difficult, since you already know both the exact compressed AND decompressed size. Trying to compress data however, is more difficult, because you don’t know how big the new zlib stream will be. However, the zlib manual states that the compressed size of a buffer will be at most (1.01 * size + 12) bytes big, regardless of compression level. Here we will use compress2, using a compression level of 9 (maximum).

Using compress2 in C++
For this short snippet we will assume that the in buffer already contains a string called "Let us...". We will compress this into zlib and turn it into a file that is similar to JJ2's file formats.

Again, if you are implementing this, remember to #include, and don’t forget to free what you malloc. If you are fussy and want your out buffer to be the right size, use realloc to adjust it.

Using compress2 in VB
Same thing as above, string to zlib, write to testcompress.dat.

Again, the Visual Basic function looks slightly harder than C++. (inserts rant about how everyone using VB should start using VC++) Only thing to note is the use of ReDim Preserve. Remember how the UncompressedSize value changes after going through compress2? We adjust the buffer so that it fits exactly the compressed stream, and prevent any problems with Put #1, , outBuffer.

CRC Checking

Finally we get to the crc32 stage. All (or most) of the JJ2 files contain an internal (and some even external) CRC check to verify that the file is not corrupted in any way. Seeing how most programmers are lazy, they will probably skip the CRC mechanism and just assume that file is valid, but it will be included here just for completeness anyway.

Declare Function crc32 Lib "zlib.dll" (ByVal crc As Long, ByRef buffer As Any, ByVal bufferLen As Long) As Long

or, in C++ format,

uLong crc32 (uLong crc, const Bytef *buf, uInt len);

As you can see, the crc32 function takes 3 arguments. The first one is a previous crc value, in case we do the CRC check in chunks. For simplicity, we will always use CRC on the whole buffer at once, which means we leave CRC as 0. buf/buffer is the buffer, and len/bufferLen is the length of the buffer. If you’ve fully understood the uncompress and compress2 functions then there is no need for an example here, since the buffer and length arguments are used here in exactly the same way as before.

Final notes/other stuff
None of the above code has been tested. They were jotted down directly on notepad and have never gone through any stage of compilation. If there are any bugs within them, report them here. The posts below will give info regarding specific file format headers, but this information alone should already be adventurous enough. ;D

The J2A File Format
Well, it should probably be called the anims.j2a file format, since that's the only file I'm aware of that uses the j2a extension.

The ALIB Header
This is not a C/C++ structure, but I will be using a similar format to define the ALIB header.

Right after the ALIB header, which is not all that confusing, comes the first of many sets. However, to be more technical, we'll call them ANIM sub-files. In addition, these ANIM sub-files each contain 4 sub-sub-files, which are, in order, Animation Info, Frame Info, Image Data, Sample Data. For simplicity, we will think of them as buffers as call them Data1 to Data4.

Each ANIM sub-file starts with a 44-byte header like this:
After that you should see your familiar zlib stream that starts with "78 DA". As soon as that stream ends the next one begins, until the end of Data4's stream. That's where the next set starts and the next ANIM header appears.

Data1 - Animation Info
Assuming you have already uncompressed this, you will get a buffer of (UData1) bytes, which also happens to be (8 * AnimationCount). This is because each animation info is only 8 bytes long:
Should be pretty straightforward, no explanation required.

Data2 - Frame Info
Same as above, uncompress, and discover that UData2 = 24 * FrameCount. Each frame info is 24 bytes long:
What the relatives mean is that the hotspotx and hotspoty value are almost always negative values, (there are notable exceptions of course). I will try to include an example at the end of everything.

Data3 - Image Data
By now uncompressing it should be a breeze. But how do you use the info contained in here? JJ2 uses its own format (or maybe one that already exists, but I don't recognise), which is similar to RLE. The first four bytes contain width and height, except that the width has some flag that is sometimes set on its most significant bit. I have no idea what that is, so we'll skip it.

After those 4 bytes, try to read the image data as codebytes. A codebyte < 128 (0x80) means to skip that many pixels. A codebyte > 128 (0x80) means to read and set that many pixels from the buffer onto the image. A codebyte = 128 (0x80) means to skip to the next row. Note that the overall image data is aligned to 8 bytes, so there can be up to 7 bytes of unused bytes between the current frame's image data and the next one.

You probably already realised that the image is 8-bits, so get a palette from some random level to start with. Anyway, the mask data is also in this Data3, and is 2-bit (black or white). I have yet to check whether this is clipping mask or transparency mask (more likely the latter), but anyway the mask data is aligned to 4 bytes.

Data4 - Sample Data
I have never been good with sound. In short, I haven't even attempted to find out what could possibly lie in here.

End of anims.j2a
The next post should include an example of reading a single frame in anims.j2a. Unless I'm very lazy or out of time, in which case I'll jump to J2T and J2L.

Example of using anims.j2a
Just a very basic example, we'll use the first frame of the first animation of the first set for this tutorial. This assumes that you have already decompressed the first ANIM sub-file into Data1, Data2 and Data3 buffers.

Peeking into Data1 - Animation Info
Okay, we will just look at the first animation (8 bytes):
The first "07 00" means it has 7 frames in the animation (first 7 * 24 bytes in Data2) and the "0A 00" stands for 10 something (I assumed FPS). Straightforward.

Peeking into Data2 - Frame Info
We shall just look at the first frame defined in this buffer (in this case, the first frame of the first animation), 24 bytes long:
I should probably have done this in a table, but I'm just too lazy:

The first "11 00" means a width of 17 pixels, "10 00" a height of 16 pixels. The "00 00" and "00 00" after it would refer to the coldspot's x and y, except a zero means that none is defined.

The "F7 FF" which converts to -9 in a signed short, means that the frame starts drawing 9 pixels to the left of the hotspot. Similarly the "F5 FF" means the frame starts drawing 11 pixels above.

The next four zeros obey the same rule as the coldspot.

The four zeros after that indicate the starting address of the image within Data3 (image data). Note that this address is aligned to 8 bytes (always a multiple of 8).

Finally the last four zeros indicate the starting address (in Data3) of the mask for this frame. This one is aligned to bytes. If the mask address is given as -1 (FF FF FF FF), then there is no mask given for it.

Peeking into Data3 - Image Data
Yay, we've come to the really fun part. It is slightly technical but not very difficult. If you are already familiar with RLE this concept should be a breeze. Remember we had an image address of 00000000, so we will read address 0 of Data3:
As mentioned before the first 4 bytes of the image data for the frame is the width and height (again), except that the most significant bit of the width is set. This could possibly a flag, or something else, I haven't quite researched this yet. So after the 4 bytes, the image begins, as a series of codebytes. Refer to previous post(s) for more details, but you could imagine it as being rearranged like this:
And you'll end up with something looking like this:

[IMG]http://www.files.bz/files/5589/j2aPALimage.png[/IMG]

Note: I included a palette for convenience. But basically, those few bytes we just went through would have drawn the first 6 rows of the "flame" :). Also, I'm not a HTML expert or anything, so some stuff could not appear the way I wanted to. Anyway, it would be great if anyone could help convert this into an ERE-readable format, since I now hate Textile. Oh yeah, and if anyone wondered what happened to Data4 (Sample Data), I have no idea how to use it. So figure it out yourself :P. This ends the J2A section of the file formats, very dull stuff. So I'll probably be making the J2T and J2L versions much more condensed (even though they can be much more complicated than this).

The J2L File Format
I had never really looked into the header with that much detail, so some parts here were stolen from another post, at least for the LEVL Header. This is mostly for stuff I've never thought about, like "Hide level in Home Cooked List" or passworded levels.

The LEVL Header
Ok, without further ado let's rewrite that as a more comprehensible struct:
The header is followed by 4 zlib streams, and by now decompressing should be a piece of cake. Data1 contains general level data, Data2 contains events, Data3 is the dictionary of "8-byte" tile groups (4 tiles per group), and Data4 contains "words", which it takes from the dictionary and outputs as proper level data. Confusing so far? Data3 and Data4 will take a while, so we'll go through the easier ones first (1 and 2 are in order anyway =/).

Data1 - General Level Data
Like everything else I've never had a need for research, I haven't looked into this buffer much. Someone else should probably update this. I've only looked at a few 1.23 files, and I think this has size 33517 (0x82ED) for the official levels, or 34029 (0x84ED) for levels saved in JCS.
I'd assume the rest of Data1 contains individual layer headers such as the width, the height, the pitch, (auto)x-speed, (auto)y-speed, and those other flags you see in JCS. The most important thing you'll need to look for here is the layer4 width and height, since that's the only one I've ever used anyway.

Try poking into the buffer at offset 0x20FB and you'll read something like:
The rest remains a mystery :(. I'd assume (quite sure actually) that animating tiles would be somewhere in here, which explains it consuming a large space with mostly zeros.

Data2 - The Event Map
Each event is a long (4 bytes long), so this buffer should be (Layer4Width * Layer4Height * 4) bytes long. So, reading an event from a coordinate in the level shouldn't be too difficult at all. The hard(er) part would be to parse the events. But since each event has a different set of arguments, I'll leave this as an exercise for the reader.

Data3 - The Dictionary
This is probably an unusual way of defining a buffer, but I use dictionary because it contains a lot of "words", which cannot be used individually but have to be stringed together by Data4 to create something meaningful.

The size of this buffer is (WordCount * 8), but I'm not sure if the number of words is actually defined in the file, so I use UData3 instead. Each word contains 4 "Tiles", and each Tile is a short which corresponds to that tile index in the J2T declared. If a Tile has its 0x1000 (4096) bit set, it is a flipped tile. There are probably other flags for translucent or caption, but I haven't gone into detail with these. As an example, battle1.j2l has a dictionary that begins like this:
This probably doesn't make much sense, but it defines the first 4 words in the dictionary:
word[0] = {0, 0, 0, 0} <-- an empty tile group, word[0] must always be this
word[1] = {0, 0, 81, 15} <-- contains two empty tiles, then two bricks
word[2] = {15, 15, 15, 15} <-- 4 bricks
word[3] = {15, 15, 15, 14} <-- 4 bricks, the 4th one using a different tile

Note that because it can use shorts, there can be a maximum of 65535 words (in theory). Each word is unique, so it usually a LOT less words, if similar tilegroups are used all over the level. This concept should not be difficult if you are already familiar with LWZ.

Data4 - Taking words and stringing them together
Okay... so that's not really a proper name for Data4, but it describes what it does. Words are only defined for LayersThatHaveAnyTiles (refer to data2). For example, battle1.j2l again: the first layer that is "defined" is layer 3, which is 128x128. The width is 128, so that means it reads 32 words per row (remember there are 4 tiles per word). Note that it rounds up, so a 124 width would read 31 words per row, but 125 would be 32.

Slight correction: this part uses info from the LayerWidthRelated defined above. Or at least that's my assumption, because layer 8 pretends it has width 60 rather than the 30 we see in JCS? I don't really get it lol.

Enough random chitchat ;), so the first 32*128 words belong to layer 3, the next 32*128 words to layer 4, and the last 15*12 words to layer 8. (15 because its 60\4, dunno why it didn't use the 30 to get 8 words per row. Could have something to do with tile width/height, or I'm just completely wrong.)

Since each word is 2 bytes long, we should have 8372 words in Data4, thus a buffer size 16744. So basically, this is just a sort of "mapping" done which copies and paste those "words" in the dictionary onto a larger screen.

The J2T File Format
My time is slowly running out, so we'll cut to the chase.
Data1 - Tileset Info
For the files I checked so far, this buffer always has a size of 27652 bytes for v1.23, or 107524 bytes for TSF. This is due to it supporting up to 4096 tiles instead of the original 1024.
First question that comes to your mind is: what are those address thingies? I should probably have called them offsets, but they basically refer to the address in their corresponding buffers from which to start reading.

In specific, ImageAddress reads from Data2, TMaskAddress reads from Data3, MaskAddress and FMaskAddress reads from Data4, and you should probably memorise this to avoid confusion later on. If you are planning to make a tileset extractor of some sort, you would probably need only to use ImageAddress and MaskAddress.

Data2 - Image Address
Each image is 1024 bytes long, uncompressed. Each byte represents a palette index, so you have a 32*32 picture just by reading each byte as a pixel. Note that each image here is unique, so for example if your tileset has a blank space in say, tile #42, then ImageAddress[42] will be equal to zero (offset zero in the Data2 buffer).

Data3 - Transparency Mask
Similar to Data2, except each mask is 128 bytes. This is because the mask is only 1 bit-per-pixel (black or white). If you are trying to extract mask information into a picture for the first time, expect some problems with bit orders; I can't remember whether it reads MSB-first or LSB-first.

Data4 - Clipping Mask
Same notes as Data3.

Other notes
Remember that Data2 is not equivalent to the actual tileset graphics as seen in JCS, and Data4 is not the mask either. Data2 simply contains an image of each unique tile in the set, likewise with Data4. You must read Data1 if you want to find out exactly what image and mask a particular tile# has.

EDIT: Very minor edit, don't ask.

Post-release notes

We've only covered 3 file formats (J2A, J2T, J2L) in the series. If you're adventurous enough, you could use the same methods to hack the other JJ2 file formats. This is sort of like a parting gift from me, and I hope you will have more use for it than I did. I really hope that when I get back here (maybe 6 months?) there will be some useful programs and new ERE entries containing this information :D. Well, have fun!

I love you. Now I just need to finish my C++ course and I WILL use this.

I haven't a CLUE what this is all about but DANG you rock! =D

...Neobeo = awesome. That's all I have to say for now.

OMG 7-double post!

Once again, awesome work...

I can't wait to get back to working with this stuff.. (I didn't know J2L except for the header and first part of Data1, or a fair bit of J2A data2)

I'll try to find some more useful code/documentation I made to share later..

one thing I guess I'll have to double check is where you have:

(in J2T) char Unknown[256]; //I suspect this has something to do with transparency; haven't looked at it yet

I have a longer thing written down (entirely possible I had expanded it based on the bits), but I think it flagged tiles that either had some transparency and/or some pure white I think (probably a bug or white used to indicate transparent or somesuch..)..

a PHP J2T decoder and a couple comments I gave someone..:
it's slow and poorly made (requires GD, only does image [no mask], doesn't really do anything cleanly), but it's more of a proof of concept that you can do stuff in non-C/C++/VB/whatever...

*Watches as Neobeo bursts through the wall* Aww, yeah!

Oops, that should have said Unknown[MAX_TILES]; .. fixed it :D.

I have never learnt any PHP beyond "hello world", but it sure looks like great stuff (it actually looks a lot like C/C++) ;). I didn't really have much time to look through the structures you defined, but I guess FastBlit would be a good way of putting it ;D.

I've never actually used most of the information in J2T either, I just did some last-minute research to provide closure for my article. So I'll be really glad if people will fill in the missing information, publish it in ERE, all in time for my "first return visit".

That's pure gobbledegook to me, but I'm hoping it can be put to some practical purpose.

If you don't understand the purpose of this, it basically means that these are the methods used to read and write to JJ2 data files. Ergo, it means that this is how we can edit crap.

Awesome. I didn't have the attention span to actually read all of that.

If you need a fully featured example, I've released an open-source Tileset Extractor which supports both v1.23 and TSF. I've never had any use for it, but others might, and besides J2T is the easiest format to understand, yada yada etc...

:) This is good, very good, extremly good. I've just started learning C++ a bit ago iam not very advanced, but i can't wait until i've learned enough to use this.
EDIT:
This probably deserves a sticky!

Neobeo, you own dude! I would say keep up the good work but unfortunately that wouldn't be very easy considering where you are off to :\

Neobeo: Well written and very thorough, great work man!

jam u help me wif the dll now k? i got source^^

Awesome. Don't understand much yet, but I'm on the way. ;)

Article this! :D

:) A tileset previewer should be added to J2O

"[...]it's slow and poorly made (requires GD[...]"

it's possible, but I don't really see anyone rushing to implement it anytime soon...

School keeps me too busy right now... maybe late May?

Sorry about that Ice M A N.

I agree :D
(download Section)

Slow ?
Sorry for the C# code. This will perform a check, if the name given up is not in the database, the (slow) funtion will do his work, set the url in database. and return the string to the function. or else.. the file will just seach up in the database and find the location of the ( allready ) renderd image!

J2O is written in PHP, foo.

But I'm sure Bobby/Frank can write this themselves. Just wanted to correct Marijn ;)

yes yes... I suggested that obvious solution as early as the code was made... it's just a matter of getting around to doing it, taking into account the tileset would need to be readjusted when reuploaded, what if a pack includes a tileset already "generated" (could use a combination filesize name and/or hash in the J2T file) etc.. (plus does J2O's host have GD enabled?)

We're in the same boat. Nothing to be sorry about.

It seems a lot more practical for the uploader to include screenshots, though. The tileset itself only shows so much, while a screenshot can be a lot more informative.

Here's an idea: What about something that could take a scaled screenshot of an entire level (or rather, all the layers where the X and Y speeds equaled 1) with perhaps all of the visible events? I know Neobeo practically made something like this already, but I think it would have its uses.

Also (I and Bob spoke about that before) you would have issues with wether uploaders from the past actually wanted screenshots of their levels on J2O. Issue would be wether to only work with permission (= most old tilesets won't have a preview) or not. J2C posted screenies from tilesets but that wasn't retrospective I think.

A few potential issues (at least with using this idea on J2O):
*neobeo (apparently?) didn't find where the X and Y speeds are (although it should be trivial to find that.. I just don't have JCS/JJ2 so it makes it difficult).. and it would also need to handle layers that tile, etc...
*once J2A edits start coming out, the visible events will change (we could do JCS-style labels, but it wouldn't really be useful in a scaled image...)... even using the standard anims would somewhat difficult, requireing prodding around in anims.j2a.. actually.. no.. we could just preprocess it and get the first frame of each event and just use those PNGs or whatever to composite the image..
*I'd guess it'd be amazingly slow in PHP (which begs the question does the J2O server allow for PHP to run an external C/whatever program somehow..?)
*Should we worry about CliffyB style hidden messages in levels (a la JJ1)
*What about retroactive appication to the current uploads (as fquist was talking about)

Of course I was planning on doing something like what you said anyway (a complete composite view of levels) once school is out (i.e. this summer) and perhaps starting a 2D game compatible with JJ2 files... an "OpenJazz2" if you will..

And then again, as you said, just having the uploader provide a preview image would be more practical...

Perhaps there could be a check box type thing when you upload a level that asks you if you want there to be a screenshot.

<OpenJazz23

A few thoughts on tileset previews for J2O.

The images should be kept lightweight, as in small in file size. The purpose of a preview is just to let the user get a quick glance at what the tileset looks like without having to download it and all. Therefore, the quality of the preview image can be dropped down greatly in order to save on file sizes (and download times). I'm thinking scaling the image down to 50%, and save it as a low-quality JPEG so that the compression can be way up.

See if you can use a compiled (c?) version of this conversion program on the server as PHP or other scripting languages are relatively slow. Then just invoke the compiled program from the site script.

The images should obviously be cached. If there's a concern about size, delete preview images that haven't been viewed in over a month or so. Generate preview images on demand.

I just was referring to new uploads where the creater wants to include a screenshot. However, I'm not against having others upload preview shots for older levels and tilesets. I don't think it's important though (plus I'm getting off-topic).

Also, I didn't mean to imply my entire-level screenshot idea would be done dynamically on J2O. It would just be a program that people may choose to use.

Neobeo you should also make a 1.24 to 1.23 J2L converter.
The JCS patch I released a while ago really sux bad.

Well ok, this is a prototype from 3 weeks ago, haven't worked on it since then. Made in Visual C++, it only reads layer 4 and represents the players as dots. (I probably stole the concept/idea from Overlord's VCR) The actual purpose of it was to test out its ability to parse J2L/J2T from within memory rather than reading player's positions. Also, while I'm able to read events, I found it impractical to include them.

If you see a chatlog in any of them, that was made in VB, and has been abandoned in place of a C++ one. So...
http://www.files.bz/files/5589/vcrtest1.png
http://www.files.bz/files/5589/vcrtest2.png
http://www.files.bz/files/5589/vcrtest3.png
http://www.files.bz/files/5589/vcrtest4.png
http://www.files.bz/files/5589/vcrchat1.png
http://www.files.bz/files/5589/vcrchat2.png
http://www.files.bz/files/5589/vcrchat3.png
http://www.files.bz/files/5589/vcrchat4.png
(Coincidence? I just noticed today that all the vcrchat screenshots have BlurredD in them)

About anyone discussing tileset screenshots, (and if you haven't noticed), I uploaded an opensource tileset extractor. As for OpenJazz2, it sounds great and good luck with it. I actually tried making a Jazz2 implementation once, more of an ASM to C++ decompilation rather than a re-creation of the game engine, and haven't really gotten that far. Finally, the 1.24 to 1.23 J2L converter should work, except I have never made a 1.24 level and therefore have no idea what the differences are.