Disassembling SEEN.TXT

[howwouldIknowthat]

How do I disassemble Seen.txt that is for version 1.5.0.4 (~9 mb)???? kprl says it is not a bytecode file...
I also came across some other applicable versions (~7,7 mb), when I try to Disassemble it, rldev gives out: Error: expected [$\<] in get_expr_term, found 0X20 near 0X000665
-is there any solution for that???
I alsolooked it up on Baka-Tsuki, there is one instruction that I dont quite get:

I have also found the XOR-key for CLANNAD FV which is 0xAF2FFB6BAF3077178748FE2C681AB9F0. In order to disassemble make sure to specify the version, 1.5.0.4. Example:
kprl.exe --target-version=1.5.0.4 -y AF2FFB6BAF3077178748FE2C681AB9F0 -e utf-8 -d SEEN.txt

there is no -y option in RLdev although I updated it (according to the patch) and Rldev has it in its program code... So im completely lost...
If anyone has some info on that please share it with me...

Clannad FV (and Clannad ME, and Little Busters!) have an extra level of encryption that older games didn't have. The version of rlDev on Haeleth's website is old and doesn't know about the extra encryption, which is why kprl doesn't have the -y option and complained about the error.

To use that patch, you'll need to re-compile rlDev after applying the patch to its source code... but the better method is to get the pre-patched code from the SVN repository of the previous Clannad translation team, because the SVN version is newer. Compiling rlDev involves OCaml 3.09 (it must be 3.09 -- the newer versions won't work). I'm not sure if you have OCaml already; let me know if you'd like more details.

The patched rlDev will be able to use the XOR key to decrypt SEEN.TXT.

BTW, the original Japanese SEEN.TXT is ~9MB. The 7.7MB version is probably an English version with debugging symbols removed -- you don't want to use that one, as kprl can't disassemble a non-Japanese SEEN properly (most of the text will be left in the .ke files, instead of getting pulled into the .utf files)

Out of curiosity, are you from KeySF?

[Richard 23]

Originally Posted by howwouldIknowthat

How do I disassemble Seen.txt that is for version 1.5.0.4 (~9 mb)???? kprl says it is not a bytecode file...
I also came across some other applicable versions (~7,7 mb), when I try to Disassemble it, rldev gives out: Error: expected [$\<] in get_expr_term, found 0X20 near 0X000665
-is there any solution for that???
I alsolooked it up on Baka-Tsuki, there is one instruction that I dont quite get:

I have also found the XOR-key for CLANNAD FV which is 0xAF2FFB6BAF3077178748FE2C681AB9F0. In order to disassemble make sure to specify the version, 1.5.0.4. Example:
kprl.exe --target-version=1.5.0.4 -y AF2FFB6BAF3077178748FE2C681AB9F0 -e utf-8 -d SEEN.txt

there is no -y option in RLdev although I updated it (according to the patch) and Rldev has it in its program code... So im completely lost...
If anyone has some info on that please share it with me...

rldev hasn't really been officially updated since about 2006. Since then several RealLive games have been released and additional functionality represented by new bytecodes have been added to the game interpreter. Also RealLive appears to have been retired as VisualArt's primary game engine as of 2010, replaced by SiglusEngine, which kindly separates bytecode data from localisable text and resource data, which should make game translation much easier for unauthorized translation groups. Huzzah!

Someone who isn't me (SWIM) has been providing RealLive hacking assistance to the #fluffy translation group which has necessitated updating rldev to handle opcodes introduced between 2006 and the end of RealLive's lifecycle (the present?). So my updated disassembler and compiler may be able to handle it.

SWIM tried to humanise some of the cryptic and unhelpful error messages in the disassembler and added some fuzziness to the disassembler so that it can attempt to disassemble files containing unknown bytecodes in the future.

SWIM also added an extensible game.nfo file to which custom xor keys can be added for specific games.

Excerpt from game.info:

Originally Posted by game.info

game CLAN "Clannad" by Key
	for RealLive 1.2.3.5
	
game CFV "Clannad Full Voice" 
	by Key inherits CLAN
	for RealLive 1.5.0.4

	using key from 256 for 257:
		0xaf 0x2f 0xfb 0x6b 0xaf 0x30 0x77 0x17
		0x87 0x48 0xfe 0x2c 0x68 0x1a 0xb9 0xf0

Although I suspect that isn't really necessary. It seems very likely even to me that game specific encryption keys are quite possibly embedded in the game archive (SEEN.TXT). Some had previously theorized that the custom key was included in the executable.

But it has been reported that SEEN.TXT files encrypted using different keys were successfully opened and executed by multiple versions of RealLive. When there is time and motivation perhaps that same person who isn't me will dream of reverse engineering that pesky annoyance and and factoring it out.

(A newer annoyance in early instances of SiglusEngine would exit to shell if a simple, and unremarkable Am I Running in Japan? test failed: the test was checking one of the standard system dll files in Windows\system32 to see if the version resource had a Japanese language flag set. Lame and lazy!)

rldev 1.45 hasn't been publickly released anywhere as far as I can tell, but perhaps I can help you out with your disassembly problem if you'd like. ;-)

tl;dr?

R23

[Richard 23]

Originally Posted by Polarem

Clannad FV (and Clannad ME, and Little Busters!) have an extra level of encryption that older games didn't have. The version of rlDev on Haeleth's website is old and doesn't know about the extra encryption, which is why kprl doesn't have the -y option and complained about the error.

Done your homework, eh?

Originally Posted by Polarem

To use that patch, you'll need to re-compile rlDev after applying the patch to its source code... but the better method is to get the pre-patched code from the SVN repository of the previous Clannad translation team, because the SVN version is newer. Compiling rlDev involves OCaml 3.09 (it must be 3.09 -- the newer versions won't work). I'm not sure if you have OCaml already; let me know if you'd like more details.

It's a royal alchemical pain in the ass. Don't install OCaml or GODI or hassle through the twisty turny path to line up all the prerequisites if you don't have to! And if you're used to imperative programming languages and haven't already embraced or heaped scorn upon the functional programming idiom... FOR GOD'S SALE JUST DON"T DO IT!!!

Originally Posted by Polarem

The patched rlDev will be able to use the XOR key to decrypt SEEN.TXT.

Assuming Clannad FV doesn't have previously encountered bytecodes like Little Busters et al.

Originally Posted by Polarem

BTW, the original Japanese SEEN.TXT is ~9MB. The 7.7MB version is probably an English version with debugging symbols removed -- you don't want to use that one, as kprl can't disassemble a non-Japanese SEEN properly (most of the text will be left in the .ke files, instead of getting pulled into the .utf files)

I think that's a consequence of "kepago" script files that have been compiled for use with rlBabel actually. Normally RealLive script files (.org actually, .ke is a haeleth invention) text is included inline Anything that doesn't look like a function call or built-in command is interpreted as text to be sent to the window, rendered using Japanese lineation (no linebreaks since it DrawString isn't used).

When compiled for rlBabel, resource strings are assigned to a string variable and sent to rlBabel via a dll function call. Since the disassembler assumes that string variable assignments are for dynamic text assembly (building strings programmatically from snippets of substrings), such operations are not exported to the resource file, another Haeleth [tm] invention.

The reason for passing the text to rlBabelized in string variables and going through some rather intensive overhead is probably because the boss didn't find a convenient method of intercepting the standard textout calls, otherwise a game compiled for rlBabel wouldn't die without it, as seems to be the case. I'm hoping there's a non-trivial way to hook the standard textout functions which would reduce the compexity and necessity for rlBabel specific compilation. But time just sails by.

Originally Posted by Polarem

Out of curiosity, are you from KeySF?

Writing a book? Lol!

R23

[howwouldIknowthat]

Guys I really appreciate your help, but as you must have already noticed im not an english speaker and more to it - not a programmer so most of your discussion here is really double dutch for me... I visited SVN repository and rewrote necessary files but still it would not give me anything, so could you please explain me in a simlpler language (step by step if possible) what i should do to disassemble SEEN.txt v1504????????????????????
Its just that the original tl team managed to dis\assemble it somehow, right, so i just want to know which steps i should follow...
BTW if someone has a disassembler (preferably rldev version) that is able to do that please send it here: alexushka2@hotmail.com
many thanx
PS KeySF? never heard of it...

[Richard 23]

Originally Posted by howwouldIknowthat

Guys I really appreciate your help, but as you must have already noticed im not an english speaker and more to it - not a programmer so most of your discussion here is really double dutch for me...

I'm sorry that you're having trouble with a seen file and that the error message is about as helpful as a page torn out of a book in someone else's language. And no, I didn't notice that you are also trying to get help using a frustrating language too. If you feel like you're being picked on or I'm trying not to help you because I'm cruel, it's all in your head. So I apologize. I do know the feeling though as most of the info and hacking on RealLive is done in Japanese and in Japanese forums... I've got a big pile of documentation that's all in Japanese so getting to the good stuff is slow and presents an agony of choice.

So I apologize for making you uncomfortable. I'm not unhelpful on purpose, okay?

Originally Posted by howwouldIknowthat

I visited SVN repository and rewrote necessary files but still it would not give me anything, so could you please explain me in a simlpler language (step by step if possible) what i should do to disassemble SEEN.txt v1504????????????????????

That's just it, don't you see? You've already done exactly the steps that you would have suggested for you. You already know what you're doing. You don't need anyone to step by step you, you've shown that you can walk and think and reason out the process but something isn't working. Welcome to the club!!!

I know what it's like to work out a couple of those keys and the technique is far from my specialty and not at all fun when you aren't sure how to do it. The hours spent can become agonizing. Rldev is a great tool for what it can do and, as I've been tinkering with it for longer than I want to admit, it's an admirable and complex package of software. It is very primitive and the error messages are often quite horrible -- in what little they say. One wonders if they should dig into the code to find out what it's trying to say, or just put the 45 into one's mouth and pull the trigger. There is no right answer to that either. D-:

Most people wouldn't have gotten that far, you've done quite well -- that's exactly what I would have done. You maybe think you're doing something wrong that I can see from over here. No. It sounds like you're already a RealLive expert if you know about the extra encryption key and -y, something maybe a dozen people have even attempted. You're already further along than those reading your request for help.

I don't have the ABSOLUTE ANSWER for you, but I can offer some informed guesses based on past experience.

--------------------------------------------

It sounds like you're getting an error very early in the disassembly, and if memory is correct, you're going to see an error due to the xor scrambling (encryption) within about the first 1024 bytes if the data wasn't properly decoded before attempting to disassemble.

SEEN.TXT is obviously not a text file, but it is COMPRESSED. Obviously you wouldn't open a zip or LZ77 compressed file with a hex editor and try to piece together the words in the scrambled data. If you get a binary file as an email attachment -- it's been encoded to survive the trip through servers, but unless you decode the email attachment it's not going to be readable. The same goes for the SEEN.TXT file you're trying to get to unzip and decode -- and it's been encoded at least twice and LZ77 compressed once. Unless all that is reversed without a single mistake it's going to look like pencil shavings to a human as well as to kprl, the decompiler.

I didn't work with the Clannad team, but I have seen their work and a few notes here and there. I don't think they faced new and unknown bytecode like we over at #fluffy did, so you aren't going to be needing anything that isn't already publicly available, you just have to get it working... and sure it's frustrating as hell. We've all been there, I suspect.

I don't recall if the last version at svn.haeleth.net is even the last one he made minor changes to. The Clannad project may have it. That it even worked, assuming it did (I have the original Clannad not FV) would have been a really lucky coincidence. That the second round of encryption was based on a simple 16 byte key that is repeated rather than something a little longer and less trivial to work out.

The -y KEY option broke about as soon as it was hastily thrown in a non-release hack fork of rldev. Little Busters EX uses a longer key which one could not simply paste into a command line. That was a whole lot of no fun to sweat over hoping against hope that the key would reveal itself. I remember that being less than pleasant.

Originally Posted by howwouldIknowthat

I visited SVN repository and rewrote necessary files but still it would not give me anything, so could you please explain me in a simlpler language (step by step if possible) what i should do to disassemble SEEN.txt v1504????????????????????

A simpler language wouldn't tell you more than you know. A crayon drawing won't help you. I could act it out with finger puppets, but the finger puppets haven't gone as far as you and without making mistakes that you avoided.

Originally Posted by howwouldIknowthat

Its just that the original tl team managed to dis\assemble it somehow, right, so i just want to know which steps i should follow...

You don't actually know that for sure do you? Someone may have resorted to cheating or adapting the original clannad script. I was crazy enough to write a voice patch for Air SE so desperate was I to play a little bit of my game with a few badly translated lines and those great voices. The translation was never going to happen.

A person desperate enough can use a debugger to dump a decoded and decompressed data file from the running application and then separate it from the rest of the dump image. That's also one of the steps one might take in trying to hammer out an unknown key. They may have gotten crass to do it or ExtractData or some other lesser known tool on a Japanese site, or maybe they got a disassembled copy from someone who somehow extracted it who died shortly after handing it over. You really don't know.

Maybe the key you've got is a character short or has a typo. I can come up with more if you want, but since you already know what you're doing, anything further is speculation. Good work so far.

Originally Posted by howwouldIknowthat

BTW if someone has a disassembler (preferably rldev version) that is able to do that please send it here: alexushka2@hotmail.com
many thanx

I'll probably email you today, I might be able to kick it in the gut and make it spill for you, but I won't know what steps I will take until I try. Once you've tried all the right things, that's when the less sane and more superstitious arts come into play, and most people don't like to admit what they have to go through sometimes to get past even the slightest amount of obfuscation.

If it's any consolation, I could send you a RealLive hacker badge, because you've rushed in where most know better than to tread.

Originally Posted by howwouldIknowthat

PS KeySF? never heard of it...

I think he's referring to KeysFanClub, a Japanese forum / hacks archive / specialized knowledge (in a foreign language for me). I spent some time there in wonderment thinking I'd found a bit of the holy grail, but I didn't know my way around and am not so stupid to think that I'm going to walk around in the dark and stumble upon a free-for-the-taking-home Aya Hirano or KomariMax (MAI WAIFUU OMG KAWAII ^__^) if I bang into walls long enough. I may have to go back there and stagger around again though. Fanatics!

If I can help you with it, I will, but my hands are tired busy and full, from punching around rlBabel and arranging a three version one base translation workflow in place while hacking rldev enough that it can retire with RealLive soon, and with the ability to tear down and rebuild games from an entire decade of that game engine. Heh. I'm going to die.

Say too much, say too little. I never seem to get it right. Phooey.

[Richard 23]

Someone suggests to me I should be concise and make with the answer and I come back with an even longer wall of text.... Heh, I'm too easy to manipulate.

I've got to resist the urge to answer questions when the person asking them clearly already has the answers at hand. It doesn't leave much chance for something revelatory to be revealed and nobody's going to throw a party at the end.

Ironically named, now that it sinks in, howwouldIknowthat? So you ARE toying with me. D-8

You surely already know that haeleth's last official version is before the second key was slapped onto a mod or fork and before mods with Little Busters or Clannad keys made brief appearances. So yeah, no -y KEY, but then that wasn't really so much a useful feature being added as it was a quick hack to meet a moment where everything broke that has since passed.

Rldev has been quite tempermental since its creator saw his chance and bolted, running over the hills and far, far away.

Originally Posted by dev.haeleth.net

Known problems and workarounds

The version of RealLive used for Little Busters! utilises additional encryption underneath the compression. This is not handled by the current release of kprl.

It also places a file in the pointer table of SEEN.TXT, causing it to fail strict validation, so kprl can't extract files from it. But extracting them by hand is trivial (can you say "Perl one-liner"?), so I'm not too worried about that.

rlBabel does not work with RealLive version 1.4 and up.

If kprl or rlc segfault on startup, try specifying the interpreter version manually with the -f option.

There are numerous bugs and inadequacies... (AND ON)

I suspect the assembla repository where Clannad's work is archived probably has a Clannad key in it or maybe the FV key, dunno. Other than that a search engine will probably exhaust itself on rldev and RealLive links pretty quickly since it's such a specialized interest.

Other than that, it looks like you're up to speed. Maybe you can get an already disassembled set in a zip somewhere. You may not need to do it yourself for now, and you might have better luck getting it to cooperate if you finder a bigger and more menacing hammer.

Now back to abusing my battered fork and daring to check my email -- did rlBabel and the test build work at a location across the globe or did something go wrong that requires more violence in order get it to approach beta? I love it and I hate it. Which is worse?

Anyway if it turns out that I can't help you and you can't get it to work you might shoot velocity7 or one of the other successful and experienced veterans from the Clannad project.

v7 is friendly enough and it seems like he's got a whole lot of info about all sorts of stuff crammed into his head. maybe go see a man about a waffle. ;-)

Well, don't be desperate, here are some hints for you.

1. The encryption of reallive games is done in two parts, masking some bytes the scenarios, and embed the key in the reallive executable and applying an out-of-box protection layer. So you can't find the key in the SEEN.TXT, and good luck trying to debug the copyright protected exe. :P

2. The key length is the same for all these games. Of course each game use a different key.

3. The main weakness of the protection scheme is that the encryption is too simple. As you may already know from the rldev source, it is just XOR.
Given another fact that reallive is a widely used engine by all companies (or workshops) under VisualArts, and not all released games are encrypted -- so the key is actually very easily recoverable by doing the following:

a. Get another unprotected game released around the same period
b. Extract a unencrypted utility scenario (these scenarios provides standard in-game services, and are game independent, and likely to be reused across multiple games)
c. Extract the encrypted utility scenario (usually having the same name/index) from your game
d. XOR them, now you get the key.

Note that since the protected is added out-of-box, the actual reallive engines of these encrypted games are exactly the same as their unencrypted counter parts. So it means that after you decrypt the scenarios, you can re-pack them as an unencrypted SEEN.TXT, and run it on an unprotected reallive.exe with the same or higher version.

This is about as much information as I can give as general knowledge concerns. To do the actual "hacking" and keep it legit, you have to buy an authentic copy and do it yourself. :D

[Richard 23]

And KeysFanClub shows up in the virtual flesh. Lol.

Your third item has some really good advice. I don't recall precisely what my procedure was when the key was coaxed from the Little Busters! EX DVD but it sounds like it was a variant on what you described.

I haven't seen enough seens to notice common contents in any two scenario files across games, but I'm not going to dispute what you've said. I might peer into the SDK though and see if a common item intended for reuse is among the sample project. So thanks for the hint. I enjoy the occasional scavenger hunt~ ;-)

Going back to point one, you have confirmed one of my suspicions I'd almost decided to discard. At any rate I hadn't had the time to confirm or refute my theories.

Originally Posted by KeysFanClub

1. The encryption of reallive games is done in two parts, masking some bytes the scenarios, and embed the key in the reallive executable and applying an out-of-box protection layer. So you can't find the key in the SEEN.TXT, and good luck trying to debug the copyright protected exe. :P

I will need to retrace my steps, but I am pretty sure that an encrypted scenario archive (the original untouched seen.txt) from Little Busters! EX (RealLive 1.5.2.4) was cheerfully opened and executed by the executable from the original Little Busters! (RealLive 1.4.8.8). I am dyslexic however, so it may have been the reverse. Up until that point I suspected that the game-specific key was in the executable itself, which would require a custom exe, although the modification would be minor. But when the scenario archive from one game opened fine using the executable from another, where each game has its own game-specific key, I thought it would be wise to rethink that and look back to the archive file itself... maybe an unused slot holds the key... it's not that long....

I have not seen an encrypted executable as far as I can tell. The only obfuscation I've come across in in 1.5.x.x where the imports table has been moved, which seemed an odd choice of tampering to me.

As to the encryption not being very severe, which I'd also noticed and was thankful for, I have a little theory. There's a whole range of nasty things that can be done to obfuscate and "protect" software which can be brought to bear in an escalating war between corporations and tinkerers. In the case of VisualArt's, I'd like to think that that the level of paranoia that you find in some shops is mostly if not entirely absent. I'm sure that they are aware of localisation efforts that occasionally form, but as these are amateurs who freely give their time for the love of the work and not profit seeking groups looking to make money off of somebody else's property, they are secure enough to tolerate it. So any protection placed on any given game is token protection, more for appearances for the client's sake and it also serves as a test of the localisation group's skill and dedication. You don't want quitters working on a translation of your game, even if they aren't charging for it. A crappy job even without authorization can potentially lose one a little face. :-)

If they really wanted to, they could make things really hard on anyone who would dare be curious about the game engine they designed, but they don't. I suspect the reason is a benevelant one. I'm sure the designers are proud of their product, it would be a shame to deface it with keep out signs and to cloak it in paranoid defense schemes. And every programmer has been a tinkerer on some level at some point in their lives. It goes with the territory. An incurious programmer is not an inventive one who finds new paradigms.

The text of my speech concludes.

Most importantly VisualArt's demonstrated what it's really made of when they unveiled SiglusEngine, which from an outsider's point of view appears to be the company's future crown jewel as RealLive is sent out to pasture for its well-deserved retirement after a decade of service. RealLive is surely better known at home than abroad, at least among game players and game designers, assuming "game" is the correct word to describe visual novels. I'm sure that many of the amateur groups at home who join the throng of crowds at comiket and offer doujin games of their own with unique character designs and scenarios have at one time or another wanted to try out RealLive, not that they could dream of affording the licensing (I haven't seen a figure, but I have heard it isn't cheap and the engine is a premium one). But now they can. Repackaged as RealLiveMax, the entire development kit with tools and documentation quite possibly never seen by anyone not already operating under a signed agreement with VisualArt's is available for doujin groups to use as they see fit a price of ZERO yen and there are no restrictions worth mentioning as far as I could tell.

That's incredibly generous and was entirely unnecessary from a business point of view. They could have simply chucked it in a hole, ending all support with it, and moved on to SiglusEngine, doubling the fee in celebration of 10 years of AVG. The move really surprised me; a corporation wouldn't make that kind of decision alone -- someone in charge did. That I applaud. I don't think it can be overstated how unlikely it is that such a thing would happen. It makes me want to find out more about the company. I'd never seen any official VisualArt's materials leaked online anywhere prior to that nor have I since. So they run a tight ship.

I suspect KeysFanClub, if that's your real name (LOL just kidding), may know much more than I do on the matter and could correct what I got wrong with regards to RealLiveMax, as I am not a native speaker of Japanese. Actually I couldn't even play one on TV, well a non-native speaker I mean. Ok, ok, I don't have a grasp of Japanese suited for casual conversation and I wouldn't be able to read a manga without cheating a bit.

I'd actually like to know a bit more about what was behind the decision to release the SDK, how it's been received. You don't find that kind of information in the slashdot.org news. Even novelnews.net doesn't provide that level of specialization. ;-)

Originally Posted by KeysFanClub

2. The key length is the same for all these games. Of course each game use a different key.

That may be. In the few games where I have some familiarity with the key in question, the same 16 bytes is repeated a few times, while in Little Busters! EX, I suppose one might say the key is a little longer and doesn't repeat as much. LOL!

Curiously the key says to me "this is only a sample of one of the milder things we might be forced to do if you don't behave so be good ok?" because after whatever effort it took to put the scheme into place and to test it and make sure it didn't break anything etc, the key is only applied to a sliver of the bytecode data and it doesn't intrude into the header section at all. So to say that each scenario is treated to another layer of encryption is a bit misleading. It overstates the matter. The game accidentally put its elbow down onto a bit of xor but don't worry it'll come out in the wash. So I can't help but think that someone at VisualArt's is amused or pleased that there are people out there who don't even exist as far as marketing is concerned yet spend vast amounts of time working with their product in an unofficial manner for zero money and 200% love for the product.[/QUOTE]

Originally Posted by KeysFanClub

This is about as much information as I can give as general knowledge concerns. To do the actual "hacking" and keep it legit, you have to buy an authentic copy and do it yourself. :D

Regulars here, as you know, can recite this principle and more importantly they understand its meaning and why it exists. They believe in it and they more than adhere to it. The occasional unregistered visitor may feign ignorance but one would have to have been hatched yesterday to have heard something similar. Rare is it that someone can put it so well and not make it sound like their holding a large bat with nails in it or wearing a golden crown as they proclaim their own superiority and lord it over all who hear the decree from above!

I may have some silliness in there but it's not aimed at you, it's for the folks who can't wait to wield it as a weapon and demonstrate to all how perfect and righteous they are. Because we all know why people do that. And it's not because they're holy. LOL.

I shall be putting my list of questions together and hunting you down for a friendly question and answer session. You cannot escape.

But just one question which cannot wait. Please. It's important and everyone will want to know.

Out of curiosity, are you from KeySF? You guys didn't really mail me a membership card did you? :-(

If they really wanted to, they could make things really hard on anyone who would dare be curious about the game engine they designed, but they don't. I suspect the reason is a benevelant one.

No, I think the reason was that they never though that their game would gain world popularity, so no protection mechanism was designed as a part of the engine. Later they decided to add some, but it was too late -- advanced ciphers can't be integrated in the engine without significant rewrite, and those superficial protection sucked.

But, things have change now. The latest SiglusEngine ships with a complicated text scrambling algorithm fused right into the game engine. The strength of the encryption is at least a couple of levels harder than the ones used in Reallive. And, I am enjoying it right now... :P

[Richard 23]

Originally Posted by KeySF

No, I think the reason was that they never though that their game would gain world popularity, so no protection mechanism was designed as a part of the engine. Later they decided to add some, but it was too late -- advanced ciphers can't be integrated in the engine without significant rewrite, and those superficial protection sucked.

Well, I was being a little optimistic and hopeful, maybe just a teeny bit. ;-)

Originally Posted by KeySF

But, things have change now. The latest SiglusEngine ships with a complicated text scrambling algorithm fused right into the game engine. The strength of the encryption is at least a couple of levels harder than the ones used in Reallive. And, I am enjoying it right now... :P

I only had time to tinker with the demo a bit, and that was pretty much wide open. One thing I did appreciate was that the script (bytecode) and text are separate unlike before where the text was inline. With the old format you pretty much needed a pretty clear understanding of the bytecode structure in order to fully disassemble it and then put it back together adjusting any pointers whose targets have moved due to localisation.

At least in the format found in the initial Rewrite demo, the bytecode could be ignored entirely if the goal was simply translation. I have a perl library I was working on that could handle most of the parsing, extraction and reassembly.

But the configuration file was certainly obfuscated. And there was one of those "am I running in Japan?" checks, IIRC. I'm not sure if it was Kud Wafter or Rewrite that looked at a system dll's version resource for a Japanese language flag to determine whether to proceed or bail.

So has anyone deobfuscated it completely yet? It's the challenge that gets people working on it in the first place. You don't keep bees away by waving flowers at 'em!

Now you've got me curious to revisit an old friend. Thanks! :-D

[Minagi]

Originally Posted by Richard 23

But the configuration file was certainly obfuscated. And there was one of those "am I running in Japan?" checks, IIRC. I'm not sure if it was Kud Wafter or Rewrite that looked at a system dll's version resource for a Japanese language flag to determine whether to proceed or bail.

So has anyone deobfuscated it completely yet? It's the challenge that gets people working on it in the first place. You don't keep bees away by waving flowers at 'em!

Now you've got me curious to revisit an old friend. Thanks! :-D

I thought I gave you the info on that when you talked to me on IRC a while ago but I guess not. Back when Rewrite's first trial came out I wrote a full tool set for it and it was going to be used for the Rewrite trial translation that was going on.

The encryption for the Gameexe.dat is as follows:
The header for the Gameexe.dat file is 1 dword for the compressed length and 1 dword for the decomperssed length, followed by the data section.

If the compressed length is not 0, XOR compressedLen bytes of the data section using this key:

Spoiler

0xC0, 0xA1, 0x8A, 0x52, 0xE9, 0x8D, 0x16, 0x22, 0x8B, 0xF7, 0xCE, 0xCC, 0xF8, 0x7A, 0x56, 0xAF
//0x7A, 0x6D, 0xAB, 0xA7, 0xEA, 0x24, 0x87, 0xD7, 0x27, 0x91, 0x38, 0xB5, 0x10, 0xBA, 0x92, 0xFE // for mare mare mare SP

If the compressed length is 0, XOR filesize-8 bytes (should be the entire data section) using the following key and grab the first 4 bytes of the decrypted data for the compressed length and the next 4 bytes after that for the decompressed length:

Spoiler

0xD8, 0x29, 0xB9, 0x16, 0x3D, 0x1A, 0x76, 0xD0, 0x87, 0x9B, 0x2D, 0x0C, 0x7B, 0xD1, 0xA9, 0x19,
0x22, 0x9F, 0x91, 0x73, 0x6A, 0x35, 0xB1, 0x7E, 0xD1, 0xB5, 0xE7, 0xE6, 0xD5, 0xF5, 0x06, 0xD6,
0xBA, 0xBF, 0xF3, 0x45, 0x3F, 0xF1, 0x61, 0xDD, 0x4C, 0x67, 0x6A, 0x6F, 0x74, 0xEC, 0x7A, 0x6F,
0x26, 0x74, 0x0E, 0xDB, 0x27, 0x4C, 0xA5, 0xF1, 0x0E, 0x2D, 0x70, 0xC4, 0x40, 0x5D, 0x4F, 0xDA,
0x9E, 0xC5, 0x49, 0x7B, 0xBD, 0xE8, 0xDF, 0xEE, 0xCA, 0xF4, 0x92, 0xDE, 0xE4, 0x76, 0x10, 0xDD,
0x2A, 0x52, 0xDC, 0x73, 0x4E, 0x54, 0x8C, 0x30, 0x3D, 0x9A, 0xB2, 0x9B, 0xB8, 0x93, 0x29, 0x55,
0xFA, 0x7A, 0xC9, 0xDA, 0x10, 0x97, 0xE5, 0xB6, 0x23, 0x02, 0xDD, 0x38, 0x4C, 0x9B, 0x1F, 0x9A,
0xD5, 0x49, 0xE9, 0x34, 0x0F, 0x28, 0x2D, 0x1B, 0x52, 0x39, 0x5C, 0x36, 0x89, 0x56, 0xA7, 0x96,
0x14, 0xBE, 0x2E, 0xC5, 0x3E, 0x08, 0x5F, 0x47, 0xA9, 0xDF, 0x88, 0x9F, 0xD4, 0xCC, 0x69, 0x1F,
0x30, 0x9F, 0xE7, 0xCD, 0x80, 0x45, 0xF3, 0xE7, 0x2A, 0x1D, 0x16, 0xB2, 0xF1, 0x54, 0xC8, 0x6C,
0x2B, 0x0D, 0xD4, 0x65, 0xF7, 0xE3, 0x36, 0xD4, 0xA5, 0x3B, 0xD1, 0x79, 0x4C, 0x54, 0xF0, 0x2A,
0xB4, 0xB2, 0x56, 0x45, 0x2E, 0xAB, 0x7B, 0x88, 0xC5, 0xFA, 0x74, 0xAD, 0x03, 0xB8, 0x9E, 0xD5,
0xF5, 0x6F, 0xDC, 0xFA, 0x44, 0x49, 0x31, 0xF6, 0x83, 0x32, 0xFF, 0xC2, 0xB1, 0xE9, 0xE1, 0x98,
0x3D, 0x6F, 0x31, 0x0D, 0xAC, 0xB1, 0x08, 0x83, 0x9D, 0x0D, 0x10, 0xD1, 0x41, 0xF9, 0x00, 0xBA,
0x1A, 0xCF, 0x13, 0x71, 0xE4, 0x86, 0x21, 0x2F, 0x23, 0x65, 0xC3, 0x45, 0xA0, 0xC3, 0x92, 0x48,
0x9D, 0xEA, 0xDD, 0x31, 0x2C, 0xE9, 0xE2, 0x10, 0x22, 0xAA, 0xE1, 0xAD, 0x2C, 0xC4, 0x2D, 0x7F

Once it's decrypted you can decompress it with this:

Spoiler

void DecompressData(unsigned char *inbuffer, unsigned char *outbuffer, int decomplen)
{
	void *endbuf = outbuffer+decomplen;
	int totaldecomp = 0;
	
	while(totaldecomp < decomplen)
	{
		short loop = 8, len = *inbuffer++;
	
		for(; loop>0 && totaldecomp < decomplen; loop--, len>>=1, *inbuffer++)
		{			
			if(len&1)
			{				
				*outbuffer++ = *inbuffer;
				totaldecomp++;
			}
			else
			{
				unsigned short data = *(unsigned short*)inbuffer++;
				int templen = 0;
				
				templen = (data&0x0f)+2; // ecx
				data >>= 4; // eax
				
				totaldecomp += templen;
				while(templen-->0)
				{
					*outbuffer++ = *(outbuffer-data);
				}
			}
		}
	}
}

Kinda meh code but it works. The decompression function also works for the Scene.pck compression.

For completion's sake, here's the key to decrypt the scripts:

Spoiler

0x70, 0xF8, 0xA6, 0xB0, 0xA1, 0xA5, 0x28, 0x4F, 0xB5, 0x2F, 0x48, 0xFA, 0xE1, 0xE9, 0x4B, 0xDE,
0xB7, 0x4F, 0x62, 0x95, 0x8B, 0xE0, 0x03, 0x80, 0xE7, 0xCF, 0x0F, 0x6B, 0x92, 0x01, 0xEB, 0xF8, 
0xA2, 0x88, 0xCE, 0x63, 0x04, 0x38, 0xD2, 0x6D, 0x8C, 0xD2, 0x88, 0x76, 0xA7, 0x92, 0x71, 0x8F, 
0x4E, 0xB6, 0x8D, 0x01, 0x79, 0x88, 0x83, 0x0A, 0xF9, 0xE9, 0x2C, 0xDB, 0x67, 0xDB, 0x91, 0x14, 
0xD5, 0x9A, 0x4E, 0x79, 0x17, 0x23, 0x08, 0x96, 0x0E, 0x1D, 0x15, 0xF9, 0xA5, 0xA0, 0x6F, 0x58, 
0x17, 0xC8, 0xA9, 0x46, 0xDA, 0x22, 0xFF, 0xFD, 0x87, 0x12, 0x42, 0xFB, 0xA9, 0xB8, 0x67, 0x6C, 
0x91, 0x67, 0x64, 0xF9, 0xD1, 0x1E, 0xE4, 0x50, 0x64, 0x6F, 0xF2, 0x0B, 0xDE, 0x40, 0xE7, 0x47, 
0xF1, 0x03, 0xCC, 0x2A, 0xAD, 0x7F, 0x34, 0x21, 0xA0, 0x64, 0x26, 0x98, 0x6C, 0xED, 0x69, 0xF4, 
0xB5, 0x23, 0x08, 0x6E, 0x7D, 0x92, 0xF6, 0xEB, 0x93, 0xF0, 0x7A, 0x89, 0x5E, 0xF9, 0xF8, 0x7A, 
0xAF, 0xE8, 0xA9, 0x48, 0xC2, 0xAC, 0x11, 0x6B, 0x2B, 0x33, 0xA7, 0x40, 0x0D, 0xDC, 0x7D, 0xA7, 
0x5B, 0xCF, 0xC8, 0x31, 0xD1, 0x77, 0x52, 0x8D, 0x82, 0xAC, 0x41, 0xB8, 0x73, 0xA5, 0x4F, 0x26, 
0x7C, 0x0F, 0x39, 0xDA, 0x5B, 0x37, 0x4A, 0xDE, 0xA4, 0x49, 0x0B, 0x7C, 0x17, 0xA3, 0x43, 0xAE, 
0x77, 0x06, 0x64, 0x73, 0xC0, 0x43, 0xA3, 0x18, 0x5A, 0x0F, 0x9F, 0x02, 0x4C, 0x7E, 0x8B, 0x01, 
0x9F, 0x2D, 0xAE, 0x72, 0x54, 0x13, 0xFF, 0x96, 0xAE, 0x0B, 0x34, 0x58, 0xCF, 0xE3, 0x00, 0x78, 
0xBE, 0xE3, 0xF5, 0x61, 0xE4, 0x87, 0x7C, 0xFC, 0x80, 0xAF, 0xC4, 0x8D, 0x46, 0x3A, 0x5D, 0xD0, 
0x36, 0xBC, 0xE5, 0x60, 0x77, 0x68, 0x08, 0x4F, 0xBB, 0xAB, 0xE2, 0x78, 0x07, 0xE8, 0x73, 0xBF

I can't remember what the Japan check was on Kud Wafter but the one in the SiglusEngine games are a complete joke and can be bypassed if you change the function to a return 1. The entire function handles the checking so that one change fixes it.

Edit: All of this works on Rewrite trial v1.00 but it seems that they did do something to the text in v2.00.
Edit 2: Looked into the new encryption on the scripts. It's surprisingly stupidly simple. To decrypt the string, calculate the XOR key by using lineNumber * 0x7087.

Yeap, figured that out a couple of days ago...

When I found out the prime number (along with another couple of prime numbers embedded in the code) I naturally connected it to RSA-like encryptions, and thought "aha, it is finally time for them to use a harder cipher"... but apparently it was just LCG...

But, this new cipher is definitely a step up. They could easily make decryption *a lot* harder, by using the byte-offset of the string reference as key -- then you will have to enjoy parsing the entire script (correctly, of course).

Wow, I disappear for a week and this place becomes a hacker's treasure trove... :D You guys rock!

Originally Posted by Richard 23

I think he's referring to KeysFanClub

I was referring to a Spanish group. They contacted me on the same day that this thread started, so I thought the two were connected.

Originally Posted by KeysFanClub

3. The main weakness of the protection scheme is that the encryption is too simple. As you may already know from the rldev source, it is just XOR.
Given another fact that reallive is a widely used engine by all companies (or workshops) under VisualArts, and not all released games are encrypted -- so the key is actually very easily recoverable by doing the following:

a. Get another unprotected game released around the same period
b. Extract a unencrypted utility scenario (these scenarios provides standard in-game services, and are game independent, and likely to be reused across multiple games)
c. Extract the encrypted utility scenario (usually having the same name/index) from your game
d. XOR them, now you get the key.

Thanks for the useful hint. I haven't done any cryptography before, so I'm keen to try... What's a RealLive game that was released last year? (I want to decrypt Clannad ME)

I suspect that poor howwouldIknowthat still hasn't gotten his answer yet though. Here's the short answer: The original TL team...
1) Installed Cygwin
2) Used Cygwin to install OCaml
3) Used OCaml to build rlDev

I'll look for a way to replicate that over the next few days.

[Pencil]

EDIT: Moved on to a newer problem, so I'm making my own thread.