gemot encubed  

Go Back   gemot encubed > Gemot > Technical Issues

Technical Issues For bug reports, problem solving, and help running Japanese software.

Reply
 
Thread Tools Display Modes
  #1  
Old 2011-04-26, 09:24
howwouldIknowthat howwouldIknowthat is offline
Visitor
 
Join Date: Apr 2011
Posts: 2
Question Disassembling SEEN.TXT

How do I disassemble Seen.txt that is for version 1.5.0.4 (~9 mb)???? kprl says it is not a bytecode file...
I also came across some other applicable versions (~7,7 mb), when I try to Disassemble it, rldev gives out: Error: expected [$\<] in get_expr_term, found 0X20 near 0X000665
-is there any solution for that???
I alsolooked it up on Baka-Tsuki, there is one instruction that I dont quite get:

I have also found the XOR-key for CLANNAD FV which is 0xAF2FFB6BAF3077178748FE2C681AB9F0. In order to disassemble make sure to specify the version, 1.5.0.4. Example:
kprl.exe --target-version=1.5.0.4 -y AF2FFB6BAF3077178748FE2C681AB9F0 -e utf-8 -d SEEN.txt

there is no -y option in RLdev although I updated it (according to the patch) and Rldev has it in its program code... So im completely lost...
If anyone has some info on that please share it with me...

Last edited by howwouldIknowthat; 2011-04-27 at 11:13.
Reply With Quote
  #2  
Old 2011-05-03, 07:48
Polarem
Guest
 
Posts: n/a
Default

Clannad FV (and Clannad ME, and Little Busters!) have an extra level of encryption that older games didn't have. The version of rlDev on Haeleth's website is old and doesn't know about the extra encryption, which is why kprl doesn't have the -y option and complained about the error.

To use that patch, you'll need to re-compile rlDev after applying the patch to its source code... but the better method is to get the pre-patched code from the SVN repository of the previous Clannad translation team, because the SVN version is newer. Compiling rlDev involves OCaml 3.09 (it must be 3.09 -- the newer versions won't work). I'm not sure if you have OCaml already; let me know if you'd like more details.

The patched rlDev will be able to use the XOR key to decrypt SEEN.TXT.

BTW, the original Japanese SEEN.TXT is ~9MB. The 7.7MB version is probably an English version with debugging symbols removed -- you don't want to use that one, as kprl can't disassemble a non-Japanese SEEN properly (most of the text will be left in the .ke files, instead of getting pulled into the .utf files)

Out of curiosity, are you from KeySF?
Reply With Quote
  #3  
Old 2011-05-04, 08:00
Richard 23's Avatar
Richard 23 Richard 23 is offline
Regular
 
Join Date: Jun 2006
Location: Washington, USA
Posts: 67
Default youneverknowdoyou

Quote:
Originally Posted by howwouldIknowthat View Post
How do I disassemble Seen.txt that is for version 1.5.0.4 (~9 mb)???? kprl says it is not a bytecode file...
I also came across some other applicable versions (~7,7 mb), when I try to Disassemble it, rldev gives out: Error: expected [$\<] in get_expr_term, found 0X20 near 0X000665
-is there any solution for that???
I alsolooked it up on Baka-Tsuki, there is one instruction that I dont quite get:

I have also found the XOR-key for CLANNAD FV which is 0xAF2FFB6BAF3077178748FE2C681AB9F0. In order to disassemble make sure to specify the version, 1.5.0.4. Example:
kprl.exe --target-version=1.5.0.4 -y AF2FFB6BAF3077178748FE2C681AB9F0 -e utf-8 -d SEEN.txt

there is no -y option in RLdev although I updated it (according to the patch) and Rldev has it in its program code... So im completely lost...
If anyone has some info on that please share it with me...
rldev hasn't really been officially updated since about 2006. Since then several RealLive games have been released and additional functionality represented by new bytecodes have been added to the game interpreter. Also RealLive appears to have been retired as VisualArt's primary game engine as of 2010, replaced by SiglusEngine, which kindly separates bytecode data from localisable text and resource data, which should make game translation much easier for unauthorized translation groups. Huzzah!

Someone who isn't me (SWIM) has been providing RealLive hacking assistance to the #fluffy translation group which has necessitated updating rldev to handle opcodes introduced between 2006 and the end of RealLive's lifecycle (the present?). So my updated disassembler and compiler may be able to handle it.

SWIM tried to humanise some of the cryptic and unhelpful error messages in the disassembler and added some fuzziness to the disassembler so that it can attempt to disassemble files containing unknown bytecodes in the future.

SWIM also added an extensible game.nfo file to which custom xor keys can be added for specific games.

Excerpt from game.info:

Quote:
Originally Posted by game.info
Code:
game CLAN "Clannad" by Key
	for RealLive 1.2.3.5
	
game CFV "Clannad Full Voice" 
	by Key inherits CLAN
	for RealLive 1.5.0.4

	using key from 256 for 257:
		0xaf 0x2f 0xfb 0x6b 0xaf 0x30 0x77 0x17
		0x87 0x48 0xfe 0x2c 0x68 0x1a 0xb9 0xf0
Although I suspect that isn't really necessary. It seems very likely even to me that game specific encryption keys are quite possibly embedded in the game archive (SEEN.TXT). Some had previously theorized that the custom key was included in the executable.

But it has been reported that SEEN.TXT files encrypted using different keys were successfully opened and executed by multiple versions of RealLive. When there is time and motivation perhaps that same person who isn't me will dream of reverse engineering that pesky annoyance and and factoring it out.

(A newer annoyance in early instances of SiglusEngine would exit to shell if a simple, and unremarkable Am I Running in Japan? test failed: the test was checking one of the standard system dll files in Windows\system32 to see if the version resource had a Japanese language flag set. Lame and lazy!)

rldev 1.45 hasn't been publickly released anywhere as far as I can tell, but perhaps I can help you out with your disassembly problem if you'd like. ;-)

tl;dr?

R23
Reply With Quote
  #4  
Old 2011-05-04, 08:22
Richard 23's Avatar
Richard 23 Richard 23 is offline
Regular
 
Join Date: Jun 2006
Location: Washington, USA
Posts: 67
Default just say no!

Quote:
Originally Posted by Polarem View Post
Clannad FV (and Clannad ME, and Little Busters!) have an extra level of encryption that older games didn't have. The version of rlDev on Haeleth's website is old and doesn't know about the extra encryption, which is why kprl doesn't have the -y option and complained about the error.
Done your homework, eh?

Quote:
Originally Posted by Polarem View Post
To use that patch, you'll need to re-compile rlDev after applying the patch to its source code... but the better method is to get the pre-patched code from the SVN repository of the previous Clannad translation team, because the SVN version is newer. Compiling rlDev involves OCaml 3.09 (it must be 3.09 -- the newer versions won't work). I'm not sure if you have OCaml already; let me know if you'd like more details.
It's a royal alchemical pain in the ass. Don't install OCaml or GODI or hassle through the twisty turny path to line up all the prerequisites if you don't have to! And if you're used to imperative programming languages and haven't already embraced or heaped scorn upon the functional programming idiom... FOR GOD'S SALE JUST DON"T DO IT!!!

Quote:
Originally Posted by Polarem View Post
The patched rlDev will be able to use the XOR key to decrypt SEEN.TXT.
Assuming Clannad FV doesn't have previously encountered bytecodes like Little Busters et al.

Quote:
Originally Posted by Polarem View Post
BTW, the original Japanese SEEN.TXT is ~9MB. The 7.7MB version is probably an English version with debugging symbols removed -- you don't want to use that one, as kprl can't disassemble a non-Japanese SEEN properly (most of the text will be left in the .ke files, instead of getting pulled into the .utf files)
I think that's a consequence of "kepago" script files that have been compiled for use with rlBabel actually. Normally RealLive script files (.org actually, .ke is a haeleth invention) text is included inline Anything that doesn't look like a function call or built-in command is interpreted as text to be sent to the window, rendered using Japanese lineation (no linebreaks since it DrawString isn't used).

When compiled for rlBabel, resource strings are assigned to a string variable and sent to rlBabel via a dll function call. Since the disassembler assumes that string variable assignments are for dynamic text assembly (building strings programmatically from snippets of substrings), such operations are not exported to the resource file, another Haeleth [tm] invention.

The reason for passing the text to rlBabelized in string variables and going through some rather intensive overhead is probably because the boss didn't find a convenient method of intercepting the standard textout calls, otherwise a game compiled for rlBabel wouldn't die without it, as seems to be the case. I'm hoping there's a non-trivial way to hook the standard textout functions which would reduce the compexity and necessity for rlBabel specific compilation. But time just sails by.

Quote:
Originally Posted by Polarem View Post
Out of curiosity, are you from KeySF?
Writing a book? Lol!

R23
Reply With Quote
  #5  
Old 2011-05-07, 11:03
howwouldIknowthat howwouldIknowthat is offline
Visitor
 
Join Date: Apr 2011
Posts: 2
Default

Guys I really appreciate your help, but as you must have already noticed im not an english speaker and more to it - not a programmer so most of your discussion here is really double dutch for me... I visited SVN repository and rewrote necessary files but still it would not give me anything, so could you please explain me in a simlpler language (step by step if possible) what i should do to disassemble SEEN.txt v1504????????????????????
Its just that the original tl team managed to dis\assemble it somehow, right, so i just want to know which steps i should follow...
BTW if someone has a disassembler (preferably rldev version) that is able to do that please send it here: alexushka2@hotmail.com
many thanx
PS KeySF? never heard of it...

Last edited by howwouldIknowthat; 2011-05-07 at 11:29.
Reply With Quote
  #6  
Old 2011-05-10, 14:47
Richard 23's Avatar
Richard 23 Richard 23 is offline
Regular
 
Join Date: Jun 2006
Location: Washington, USA
Posts: 67
Default

Quote:
Originally Posted by howwouldIknowthat View Post
Guys I really appreciate your help, but as you must have already noticed im not an english speaker and more to it - not a programmer so most of your discussion here is really double dutch for me...
I'm sorry that you're having trouble with a seen file and that the error message is about as helpful as a page torn out of a book in someone else's language. And no, I didn't notice that you are also trying to get help using a frustrating language too. If you feel like you're being picked on or I'm trying not to help you because I'm cruel, it's all in your head. So I apologize. I do know the feeling though as most of the info and hacking on RealLive is done in Japanese and in Japanese forums... I've got a big pile of documentation that's all in Japanese so getting to the good stuff is slow and presents an agony of choice.

So I apologize for making you uncomfortable. I'm not unhelpful on purpose, okay?

Quote:
Originally Posted by howwouldIknowthat View Post
I visited SVN repository and rewrote necessary files but still it would not give me anything, so could you please explain me in a simlpler language (step by step if possible) what i should do to disassemble SEEN.txt v1504????????????????????
That's just it, don't you see? You've already done exactly the steps that you would have suggested for you. You already know what you're doing. You don't need anyone to step by step you, you've shown that you can walk and think and reason out the process but something isn't working. Welcome to the club!!!

I know what it's like to work out a couple of those keys and the technique is far from my specialty and not at all fun when you aren't sure how to do it. The hours spent can become agonizing. Rldev is a great tool for what it can do and, as I've been tinkering with it for longer than I want to admit, it's an admirable and complex package of software. It is very primitive and the error messages are often quite horrible -- in what little they say. One wonders if they should dig into the code to find out what it's trying to say, or just put the 45 into one's mouth and pull the trigger. There is no right answer to that either. D-:

Most people wouldn't have gotten that far, you've done quite well -- that's exactly what I would have done. You maybe think you're doing something wrong that I can see from over here. No. It sounds like you're already a RealLive expert if you know about the extra encryption key and -y, something maybe a dozen people have even attempted. You're already further along than those reading your request for help.

I don't have the ABSOLUTE ANSWER for you, but I can offer some informed guesses based on past experience.

--------------------------------------------

It sounds like you're getting an error very early in the disassembly, and if memory is correct, you're going to see an error due to the xor scrambling (encryption) within about the first 1024 bytes if the data wasn't properly decoded before attempting to disassemble.

SEEN.TXT is obviously not a text file, but it is COMPRESSED. Obviously you wouldn't open a zip or LZ77 compressed file with a hex editor and try to piece together the words in the scrambled data. If you get a binary file as an email attachment -- it's been encoded to survive the trip through servers, but unless you decode the email attachment it's not going to be readable. The same goes for the SEEN.TXT file you're trying to get to unzip and decode -- and it's been encoded at least twice and LZ77 compressed once. Unless all that is reversed without a single mistake it's going to look like pencil shavings to a human as well as to kprl, the decompiler.

I didn't work with the Clannad team, but I have seen their work and a few notes here and there. I don't think they faced new and unknown bytecode like we over at #fluffy did, so you aren't going to be needing anything that isn't already publicly available, you just have to get it working... and sure it's frustrating as hell. We've all been there, I suspect.

I don't recall if the last version at svn.haeleth.net is even the last one he made minor changes to. The Clannad project may have it. That it even worked, assuming it did (I have the original Clannad not FV) would have been a really lucky coincidence. That the second round of encryption was based on a simple 16 byte key that is repeated rather than something a little longer and less trivial to work out.

The -y KEY option broke about as soon as it was hastily thrown in a non-release hack fork of rldev. Little Busters EX uses a longer key which one could not simply paste into a command line. That was a whole lot of no fun to sweat over hoping against hope that the key would reveal itself. I remember that being less than pleasant.

Quote:
Originally Posted by howwouldIknowthat View Post
I visited SVN repository and rewrote necessary files but still it would not give me anything, so could you please explain me in a simlpler language (step by step if possible) what i should do to disassemble SEEN.txt v1504????????????????????
A simpler language wouldn't tell you more than you know. A crayon drawing won't help you. I could act it out with finger puppets, but the finger puppets haven't gone as far as you and without making mistakes that you avoided.

Quote:
Originally Posted by howwouldIknowthat View Post
Its just that the original tl team managed to dis\assemble it somehow, right, so i just want to know which steps i should follow...
You don't actually know that for sure do you? Someone may have resorted to cheating or adapting the original clannad script. I was crazy enough to write a voice patch for Air SE so desperate was I to play a little bit of my game with a few badly translated lines and those great voices. The translation was never going to happen.

A person desperate enough can use a debugger to dump a decoded and decompressed data file from the running application and then separate it from the rest of the dump image. That's also one of the steps one might take in trying to hammer out an unknown key. They may have gotten crass to do it or ExtractData or some other lesser known tool on a Japanese site, or maybe they got a disassembled copy from someone who somehow extracted it who died shortly after handing it over. You really don't know.

Maybe the key you've got is a character short or has a typo. I can come up with more if you want, but since you already know what you're doing, anything further is speculation. Good work so far.

Quote:
Originally Posted by howwouldIknowthat View Post
BTW if someone has a disassembler (preferably rldev version) that is able to do that please send it here: alexushka2@hotmail.com
many thanx
I'll probably email you today, I might be able to kick it in the gut and make it spill for you, but I won't know what steps I will take until I try. Once you've tried all the right things, that's when the less sane and more superstitious arts come into play, and most people don't like to admit what they have to go through sometimes to get past even the slightest amount of obfuscation.

If it's any consolation, I could send you a RealLive hacker badge, because you've rushed in where most know better than to tread.

Quote:
Originally Posted by howwouldIknowthat View Post
PS KeySF? never heard of it...
I think he's referring to KeysFanClub, a Japanese forum / hacks archive / specialized knowledge (in a foreign language for me). I spent some time there in wonderment thinking I'd found a bit of the holy grail, but I didn't know my way around and am not so stupid to think that I'm going to walk around in the dark and stumble upon a free-for-the-taking-home Aya Hirano or KomariMax (MAI WAIFUU OMG KAWAII ^__^) if I bang into walls long enough. I may have to go back there and stagger around again though. Fanatics!

If I can help you with it, I will, but my hands are tired busy and full, from punching around rlBabel and arranging a three version one base translation workflow in place while hacking rldev enough that it can retire with RealLive soon, and with the ability to tear down and rebuild games from an entire decade of that game engine. Heh. I'm going to die.

Say too much, say too little. I never seem to get it right. Phooey.
Reply With Quote
  #7  
Old 2011-05-10, 15:31
Richard 23's Avatar