Fate XP3 Files

I'm painfully new here as well as to this whole doujinshi/hentai/visual novel, or whatever the heck you wanna call it, scene.

I like to extend a special thank you to Ed from insani for pointing me in the general direction of getting started, quoting programs that I'd find useful and well as some hints and a link to this forum. I need some more help.

I've been a fansubber for several years and a friend of mine (that works with me in the group I'm involved with) is working with a fansub group that's subbing the Fate/stay night anime. With this prospect, I asked if they also might want to translate the game as well. It would be good for them and it would also give me practice in order to learn how this all works, but I've run into another snag...

I have no idea how to write a repacker for the Fate XP3 files so I can pack the translated scripts back into it. I've looked online for tutorials on how to write packers, but I couldn't find anything beneficial to my cause in a language I have tried, and am still trying, to learn during this whole game translation process: Python or C++ (preferrably Python which is a language I'm more familiar with than C++, but either is acceptable).

What goes into writing a packer? How does the XP3 compression format work? How do you figure out how a file is compressed in order to write a packer/unpacker? Is there an easy way to learn these things?

I'm not asking for explicit instructions or a step by step tutorial, but maybe an example, a link to some place that would be helpful in learning, or the source to another, similar packer so I can get an idea of how one works...

Before Ed at insani replied to my e-mail about getting started, I've asked others before him and I've been flamed and called a stupid idiot, so I've not had a very good track record with asking questions about this subject (and am ultimately afraid of asking such questions, but with the abuse I've sustained I've stopped caring at this point).

On top of it all I'm a slow learner and it takes time for me to understand this stuff... I hope someone here can help. And thank you.

[Haeleth]

Welcome to the island of relative sanity that is Hæleþes gemot! Fear not, you won't be flamed here: that's one of the few things I absolutely don't tolerate. (That said, I'll ask you not to go into any detail on the subject of subbing the Fate anime here, as it's licensed - but talking about translating the game is fine.)

As far as handling packers in general is concerned, I can give you some pointers, and doubtless others here know a fair bit too. But the one person I know who's actually written a packer for the Fate XP3 files is Ed himself. I'm a bit surprised if he hasn't even given you any information on the format, let alone source code - did you not happen to mention to him that it's Fate you're interested in?

Thank you for your welcome! It's good to be here. :)

Firstly, I apologize about the Fate anime. I hadn't heard it had been licensed (and it went fast, too) otherwise I would have kept it out of the post entirely. :)

... And I lost my sanity after the moment I decided to take up subbing. :P

No, I never mentioned Fate as my main objective to him. At the time I asked I wasn't sure what game I would like to try first, and Fate only came after I discovered I could actually get anywhere with it. Otherwise it would be me just playing around without any sort of solid, pysical goal in mind.

I also wanted to take a more general approach and see what I could do on my own learning the ways and actions of how to do what on anything I decided to undertake. Now that I realize how deep I'm in it, I'll definitely mention to him that's what I'm interested in.

But overall I would like to learn as much from everyone here as possible about the whole business. :)

[roxfan]

EX3 are just pictures, to get the scripts you need the .p packer/unpacker (decp/encp) which seem to be GPL, but you probably have them if you contacted Ed. By the way, there is a group that already started working on Fate, though I'm not sure how big that secret is so I won't say who.

As for how to reverse a random file format, there are generally two approaches:
1) Look at the files and try to figure out the format by making some observations. This works best if you have something that can produce such files (e.g. an archiver or compiler).
2) disassemble/debug the program that reads the file and try to figure out what it does.

An example of the first way is described here. While it looks nice and pretty easy, this approach is mostly useless for Japanese games. Japanese LOVE inventing new encryption/compression algorithms for every new engine. They're usually not very advanced (often to the point of being lame) but still different from your stock zlib/lzw stuff. So in most cases you'll have to go the second route.
One thing that helps here is that Japanese gamers are pretty quick in figuring out the format/algos and you can find unpackers/susie plugins for about any more or less known game. This doesn't eliminate the problem of putting it back, but might give some starting points.

[roxfan]

Originally Posted by Haeleth

But the one person I know who's actually written a packer for the Fate XP3 files is Ed himself.

Weird, my copy of bmptoex3.cpp has this:

BMPTOEX3 - Melty Blood bmp -> ex3 converter

  (c) Copyright 2003-2005 Theo Berkau(cwx@softhome.net)

Though this is for Melty Blood... maybe Fate used a different algorithm?

[Haeleth]

EX3 != XP3. :P


Executive summary of how I approach hacking a game archive follows. This was written while roxfan was posting, so it's partly redundant.

First, work out the general archive format. Most archives start (or end) with a table of contents that gives the names of the files, where they are in the archive, and how big they are. This part can almost always be done just by opening the file and looking at it - I have yet to come across a case where the table of contents was compressed or encrypted in any way.

AVG32 has a nice detailed header that's easy enough to work out:

struct PaclibArchive {
    char identifier[16]; // "PACL" and 12 null bytes
    int file_count;
    char padding[12];    // null bytes
    struct {
        char filename[16];
        int offset;             // position of file in archive
        int compressed_size;
        int uncompressed_size;
        int compressed;         // 1 if compressed, 0 otherwise
    } table_of_contents[];  // there are "filecount" elements in this array
    // data follows
};

In RealLive, meanwhile, files are known only by numbers, not by names, so the header is very simple:

struct Seen_txt {
    struct {
        int offset;  // position of file in archive
        int size;    // compressed size (all files are compressed)
    } table_of_contents[10000];
    unsigned char data[];
};

Generally, if files are compressed at all, they'll be compressed separately - if you know Unix conventions, then think of a tar of gzipped files, rather than a gzipped tarball. The idea is to make it as efficient as possible to extract a random file from the archive at runtime.

Now, as for how to determine how - or if - the files are compressed, this can be trickier. If you can't see any text in the files, then there are three options: either they're compressed, or encrypted, or both. The easiest way to find out which it is is usually to google for it! Often a Japanese hacker will already have written a program to extract text and images from a popular game. (They generally don't write programs to reinsert modified content, though.) If Google fails, you either make a lucky guess (which is what I did with Majiro), or you load the game up in a debugger and watch how it accesses files. I've never seen encryption that's more complicated than simple XOR stuff (in ONE, everything's just XORed with 2, while in RealLive and Majiro there's a fixed key), and I've never seen compression that's more complicated than LZ77 (and Majiro doesn't use any compression at all). But that's not to say it doesn't exist.

In the case of Fate/stay night, this part is actually very simple. The data is "encrypted" trivially by flipping a few bits, and then it's compressed using the standard zlib library - so you don't need to write your own compression/decompression routine. :)

[roxfan]

Originally Posted by Haeleth

EX3 != XP3. :P

Oops, silly me.

I've never seen encryption that's more complicated than simple XOR stuff (in ONE, everything's just XORed with 2, while in RealLive and Majiro there's a fixed key), and I've never seen compression that's more complicated than LZ77 (and Majiro doesn't use any compression at all). But that's not to say it doesn't exist.

I guess I'm luckier than you since I've seen both :) Wind DVD and Haruoto use Blowfish with a separate key per resource type (filenames are encrypted too, btw) and the Melty EX3 format uses a homegrown LZMW variation (with a tiny 256-byte dictionary so the compression is not that great).

[GreatSaintLouis]

Do you really need a specialized repacking tool for Fate script files? Fate/stay night runs on the KAG3/Kirikiri scripting language (common knowledge, I know, but I'm just reiterating), but I thought the KAG3 toolkit included utilities for packing .ks script and various resource files into the .xp3 format used for releases?

Of course, I've just tinkered a bit with a doujin title that doesn't share Fate's encryption, as well as just done some work on my own, so it's possible that something completely different was used for creating the .xp3 files in Fate--or I could be completely off the mark with this 'information' entirely. If so, please disregard this post.

[zalas]

Sure, they have the packers, but not the unpackers. Furthermore, you'd still need something to scramble/unscramble the files to make it compatible with Fate/stay night.

[GreatSaintLouis]

Yeah, I remember the fun I had trying to get that unpacker to work with your help, hehe.

I wasn't sure if the KAG3 toolset would scramble the files or not, I just knew that you could output a data.xp3 file from it. Sounds like Choco Ed's tools are still needed anyways.

[zalas]

I think he should set up a sourceforge project page for that or something... I have a bunch of diffs I'd like to contribute ~_~

[Danj]

There's a susie32 plugin for Fate/stay night and Fate/hollow ataraxia's XP3 files, so you could use that for unpacking. Not sure what you'd use for packing though. Also for images you can use the susie32 plugin together with a program called Linar, which will let you browse them with thumbnails and stuff, so you don't have to extract them all to find the one you want.