gemot encubed  

Go Back   gemot encubed > Gemot > Production & Help

Production & Help For discussions regarding production aspects, especially localisation, of visual novels and related games.

Reply
 
Thread Tools Display Modes
  #16  
Old 2011-04-12, 18:11
AngelicDreams's Avatar
AngelicDreams AngelicDreams is offline
Member
 
Join Date: Apr 2011
Posts: 11
Default

Quote:
Originally Posted by Nanashi3 View Post
somewhat unrelated, toradora scripts. (extractor bumped to version 0.2.1)

Would be nice if you managed to have that other guy who (tentatively?) wrote a .dat repacker to release his/her source code.

out,
7-4-3.
I will thank you in the place of the Toradora! P translation project's staff. I have notified them that they can get the scripts that they've had trouble extracting here. This may start their project just as it started ours! :D

I don't think RoByte has finished writing the tool for the .dat file repacker yet. I'll ask him if he's willing to share his source code when he's done though. On another note, somebody from the Toradora! P thread at hongfire seems to have written one but hasn't been on in a while. I attempted to ask him for the repacker before RoByte came back but he didn't reply. If RoByte's willing to share after he's done writing it, I'll post back here as soon as possible. Considering you helped us with the scripts, it's likely he'll be willing to share.

Thanks for all your help!
Reply With Quote
  #17  
Old 2011-04-13, 18:58
RoByte RoByte is offline
Visitor
 
Join Date: Apr 2011
Posts: 4
Default

I'll be glad to release to source once I've finalized the program and cleaned up the code. I'm probably the only one who could use it in the current state :)
Reply With Quote
  #18  
Old 2011-04-16, 17:28
k-wng
Guest
 
Posts: n/a
Default

It seems like I don't need to finish this anymore. It is still usable if you create some batch files.

Code:
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.RandomAccessFile;
import java.util.Arrays;

public class Gpda {
    static final String CHARSET = "SJIS";
    static final int DAT_BLOCK_SIZE = 2048;
    static final byte[] GPDA_SIG = new byte[] {71,80,68,65};
    static final byte[] GZIP_SIG = new byte[] {31,(byte)139,8,0};

    static final int FILE_BUFFER_SIZE = 5120000;
    static boolean gzip = false;

    public Gpda() {
    }

    int findDupName(int cur, String name, String[] names) {
        int count = 0;
        for (int i=0; i<names.length;i++) {
            if (names[i].equals(name)) {
                if (cur < i) {
                    return count;
                }
                count++;
            }
        }
        if (count>1) {
            return count;
        }
        return -1;
    }

    void extractGpda(RandomAccessFile in, long offset, int size, String output, FileOutputStream datlist, String infostring, byte[] indexname) {
        try {
            byte[] buffer = new byte[4];
            in.seek(offset);
            in.read(buffer, 0, 4);

            if (Arrays.equals(GPDA_SIG,buffer)) {
                int filecount,namesize,sameNum;
                int[] filesizes, nameoffsets;
                long filesize;
                long[] fileoffsets;
                String[] filelist;
                byte[][] indexnames;
                String filename;

                filesize = Long.reverseBytes(in.readLong());
                filecount = Integer.reverseBytes(in.readInt());
                fileoffsets = new long[filecount];
                filesizes = new int[filecount];
                nameoffsets = new int[filecount];
                filelist = new String[filecount];
                indexnames = new byte[filecount][];

                for (int i=0; i<filecount; i++) {
                    fileoffsets[i] = Long.reverseBytes(in.readLong());
                    filesizes[i] = Integer.reverseBytes(in.readInt());
                    nameoffsets[i] = Integer.reverseBytes(in.readInt());
                }
                for (int i=0; i<filecount; i++) {
                    in.seek((long)nameoffsets[i]+offset);
                    namesize = Integer.reverseBytes(in.readInt());
                    indexnames[i] = new byte[namesize];
                    in.read(indexnames[i], 0, namesize);
                    filelist[i] = new String(indexnames[i], CHARSET);
                }
                System.out.println("Extracting: "+output);
                new File(output).mkdir();

                datlist.write(("d"+infostring).getBytes());
                datlist.write(indexname);
                datlist.write("\r\n".getBytes());

                for (int i=0; i<filecount; i++) {
                    sameNum = findDupName(i, filelist[i], filelist);
                    filename = output+"/"+filelist[i].trim();
                    if (sameNum > 0) {
                        extractGpda(in, fileoffsets[i]+offset, filesizes[i], filename+"_"+sameNum, datlist, sameNum+","+output+"/", indexnames[i]);
                    } else {
                        extractGpda(in, fileoffsets[i]+offset, filesizes[i], filename, datlist, ","+output+"/", indexnames[i]);
                    }
                }
            } else if (offset==0) {
                System.out.println("not a GPDA file");
            } else if (gzip && Arrays.equals(GZIP_SIG,buffer)) {
                int read;
                String filename = output;
                FileOutputStream out;
                if (output.endsWith(".gz")) {
                    datlist.write(("g"+infostring).getBytes());
                    datlist.write(indexname);
                    datlist.write("\r\n".getBytes());
                } else {
                    filename = output+".gz";
                    datlist.write(("e"+infostring).getBytes());
                    datlist.write(indexname);
                    datlist.write("\r\n".getBytes());
                }
                in.seek(offset);
                buffer = new byte[FILE_BUFFER_SIZE];
                out = new FileOutputStream(filename);
                for (int r=0;r<size;r+=read) {
                    read = in.read(buffer, 0, Math.min(FILE_BUFFER_SIZE,size-r));
                    if (read <= 0) {
                        return;
                    }
                    out.write(buffer, 0, read);
                }
                out.close();
                System.out.println("Extracting: "+filename);
                Runtime.getRuntime().exec(new String[] {"gzip", "-df", filename});
                //does not wait for and check if extracted file is a gpda file
            } else {
                int read;
                FileOutputStream out;
                in.seek(offset);
                buffer = new byte[FILE_BUFFER_SIZE];
                System.out.println("Writing: "+output);

                datlist.write(("f"+infostring).getBytes());
                datlist.write(indexname);
                datlist.write("\r\n".getBytes());
                out = new FileOutputStream(output);
                for (int r=0;r<size;r+=read) {
                    read = in.read(buffer, 0, Math.min(FILE_BUFFER_SIZE,size-r));
                    if (read <= 0) {
                        return;
                    }
                    out.write(buffer, 0, read);
                }
                out.close();
            }
        } catch (Exception e) {
            System.out.println(e.getMessage());
        }
    }

    void writeSig(RandomAccessFile out, long offset, int filecount) throws Exception {
        out.seek(offset);
        out.write(GPDA_SIG, 0, 4);
        if (filecount == 0) {
            out.writeInt(Integer.reverseBytes(DAT_BLOCK_SIZE));
            out.seek(offset+DAT_BLOCK_SIZE-1);
            out.writeByte(0);
            return;
        }
        out.seek(offset+12);
        out.writeInt(Integer.reverseBytes(filecount));
    }

    int writeIndexNames(RandomAccessFile out, long offset, byte[][] list) throws Exception {
        int filecount = list.length;
        int size_index = filecount*16+16;
        int size_names = 0;
        for (int i=0; i<filecount; i++) {
            out.seek(offset+28+i*16);
            out.writeInt(Integer.reverseBytes(size_index+size_names));
            size_names += list[i].length+4;
        }
        out.seek(offset+size_index);
        for (int i=0; i<filecount; i++) {
            out.writeInt(Integer.reverseBytes(list[i].length));
            out.write(list[i], 0, list[i].length);
        }
        return size_index+size_names;
    }

    void writeIndexFiles(RandomAccessFile out, long offset, long fileoffset, int filesize) throws Exception {
        out.seek(offset);
        out.writeLong(Long.reverseBytes(fileoffset));
        out.writeInt(Integer.reverseBytes(filesize));
    }

    void writeDatSize(RandomAccessFile out, long offset, long size) throws Exception {
        out.seek(offset+4);
        out.writeLong(Long.reverseBytes(size));
    }

    int getBlockSize(int size) {
        if (size > 0) {
            return (int)(Math.ceil((double)size/DAT_BLOCK_SIZE)*DAT_BLOCK_SIZE);
        } else {
            return DAT_BLOCK_SIZE;
        }
    }

    long writeFile(File input, RandomAccessFile out, long offset) throws Exception {
        long current_filesize = input.length();
        byte[] buffer = new byte[FILE_BUFFER_SIZE];
        FileInputStream in = new FileInputStream(input);
        int bytes_read;
        out.seek(offset);
        for (int total_bytes_read=0;total_bytes_read<current_filesize;total_bytes_read += bytes_read) {
            bytes_read = in.read(buffer, 0, FILE_BUFFER_SIZE);
            if (bytes_read > 0) {
                out.write(buffer, 0, bytes_read);
            } else {
                return -1;
            }
        }
        in.close();
        return current_filesize;
    }

    private class Node {
        int depth;
        String id,path;
        Node next,parent,child;
        byte[] indexname;

        Node(Node parent, String id, byte[] indexname, String path, int depth) {
            this.parent = parent;
            this.id=id;
            this.indexname = indexname;
            this.path = path;
            this.depth = depth;
            this.child = this.next = null;
        }

    }

    int getDepth(String path) {
        int depth = 0;
        for (int i=0;i<path.length();i++) {
            if (path.charAt(i)=='/') {
                depth++;
            }
        }
        return depth;
    }

    int _fcurrent=0;
    int _fcurrentsize=0;
    byte[] _fbuffer = new byte[4096000];

    byte[] readLine(FileInputStream datlist) throws Exception {
        int start = _fcurrent;
        if (_fcurrentsize==0) {
            _fcurrentsize = datlist.read(_fbuffer, 0, _fbuffer.length);
        }
        int nl = -2;
        for (;_fcurrent<_fcurrentsize;_fcurrent++) {
            if (_fbuffer[_fcurrent] == (byte)13) {
                nl = _fcurrent;
            } else if (nl == _fcurrent-1 && _fbuffer[_fcurrent] == (byte)10) {
                _fcurrent++;
                byte[] ret = new byte[nl-start];
                for (int j=0;j<nl-start;j++) {
                    ret[j] = _fbuffer[j+start];
                }
                return ret;
            }
        }
        if (_fcurrentsize > 0) {
            int copy = _fbuffer.length-start;
            System.arraycopy(_fbuffer, start, _fbuffer, 0, copy);
            _fcurrent = 0;
            _fcurrentsize = datlist.read(_fbuffer, copy, _fbuffer.length-copy);
            if (_fcurrentsize <= 0) {
                return null;
            }
            _fcurrentsize+= copy;
            return readLine(datlist);
        }
        return null;
    }

    void readDatlist(FileInputStream datlist, Node p) throws Exception {
        int depth,cs,ls;
        String path,id;
        Node c;
        byte[] indexname;

        for (byte[] byteline=readLine(datlist);byteline!=null;byteline=readLine(datlist)) {
            String line = new String(byteline, CHARSET);
            cs = line.indexOf(',');
            ls = line.lastIndexOf('/');
            id = line.substring(0, cs);
            path = line.substring(cs+1, ls);
            depth = getDepth(line);
            indexname = new byte[byteline.length-ls-1];
            System.arraycopy(byteline, ls+1, indexname, 0, byteline.length-ls-1);
            if (depth == p.depth+1) {
                p=p.child = new Node(p, id, indexname,path,depth);
            } else if (depth == p.depth) {
                p=p.next = new Node(p, id, indexname,path,depth);
            } else {
                for (c = p; c!=null && depth < c.depth;c=c.parent) {}
                p=c.next=new Node(c, id, indexname,path,depth);
            }
        }
    }

    long writeGpda(RandomAccessFile out, long offset, FileInputStream datlist) {
        try {
            byte[] l = readLine(datlist);
            String line = new String(l);
            if (line.startsWith("d,")) {
                String name = line.substring(2);
                Node start = new Node(null, "d", name.getBytes(),"",0);
                readDatlist(datlist, start);
                return _writeGpdaWithTxt(out, offset, start.child);
            } else {
                return -1;
            }
        } catch (Exception e) {
            System.out.println(e.getMessage());
            return -1;
        }
    }

    long _writeGpdaWithTxt(RandomAccessFile out, long offset, Node filelist) {
        long written_bytes = 0;
        try {
            int filecount = 0;
            for (Node c=filelist;c != null; c=c.next) {
                filecount++;
            }
            byte[][] list = new byte[filecount][];
            int t=0;
            for (Node c=filelist;c != null; c=c.next) {
                list[t] = c.indexname;
                t++;
            }
            writeSig(out, offset, filecount);
            if (filecount==0) {
                return DAT_BLOCK_SIZE;
            }

            written_bytes += getBlockSize(writeIndexNames(out, offset, list));

            String current_file;
            long current_filesize;
            int i=0;
            for (Node c=filelist;c != null; c=c.next) {
                current_file = c.path+"/"+new String(c.indexname, CHARSET).trim();
                System.out.println("Adding: "+current_file);
                if (c.id.charAt(0) == 'd') {
                    current_filesize = _writeGpdaWithTxt(out,offset+written_bytes,c.child);
                    if (current_filesize < 0) {
                        return current_filesize;
                    }
                } else {
                    if (c.id.length()>1) {
                        current_file = current_file+"_"+c.id.substring(1);
                    }
                    if (gzip && c.id.charAt(0)== 'g') {
                        Process p = Runtime.getRuntime().exec(new String[] {"gzip", "-fkn9", current_file.substring(0, current_file.lastIndexOf('.'))});
                        p.waitFor();
                        File tmp = new File(current_file);
                        current_filesize = writeFile(tmp, out, offset+written_bytes);
                        tmp.delete();
                    } else if (gzip && c.id.charAt(0) == 'e') {
                        Process p = Runtime.getRuntime().exec(new String[] {"gzip", "-fkn9", current_file});
                        p.waitFor();
                        File tmp = new File(current_file+".gz");
                        current_filesize = writeFile(tmp, out, offset+written_bytes);
                        tmp.delete();
                    } else {
                        current_filesize = writeFile(new File(current_file), out, offset+written_bytes);
                    }
                }
                if (current_filesize < 0) {
                    return current_filesize;
                }
                writeIndexFiles(out, offset+16+i*16, written_bytes, (int)current_filesize);
                current_filesize = getBlockSize((int)current_filesize);
                written_bytes += current_filesize;
                i++;
            }
            writeDatSize(out, offset, written_bytes);
            return written_bytes;
        } catch (Exception e) {
            System.out.println(e.getMessage());
            return -1;
        }
    }

    void start(String file) {
        try {
            File f = new File(file);
            if (f.isDirectory()) {
            } else if (file.endsWith(".dat.txt")) {
                FileInputStream datlist = new FileInputStream(file);
                String filename = file.substring(0, file.length()-8);
                f = new File(filename+".dat");
                if (f.exists()) {
                    f.delete();
                }
                RandomAccessFile out = new RandomAccessFile(f, "rw");
                long ret = writeGpda(out, 0, datlist);
                if (ret > 0 && out.length() != ret) {
                    out.seek(ret-1);
                    out.writeByte(0);
                }
                out.close();
            } else if (file.endsWith(".dat")) {
                RandomAccessFile in = new RandomAccessFile(f, "r");
                FileOutputStream datlist = new FileOutputStream(file+".txt");
                String name = file.substring(0, file.length()-4);
                extractGpda(in, 0, 0, name, datlist, ",", name.getBytes());
                in.close();
                datlist.close();
            }
            System.out.println("end");
        } catch (Exception e) {
            System.out.println(e.getMessage());
        }
    }

    public static void main(String[] args) {
        if (args.length == 1) {
            new Gpda().start(args[0]);
        } else if (args.length == 2) {
            if (args[1].equals("gzip")) {  //gzip.exe is needed, recompressing can change some files,
                gzip = true;               //without this option and without changing files
            }                              //should produce exactly the same .dat as the original
            new Gpda().start(args[0]);
        } else {
            System.out.println("To unpack: java Gpda <filename>.dat [gzip]");
            System.out.println("To pack: java Gpda <filename>.dat.txt [gzip]");
        }
    }
}
Reply With Quote
  #19  
Old 2011-04-19, 18:11
AngelicDreams's Avatar
AngelicDreams AngelicDreams is offline
Member
 
Join Date: Apr 2011
Posts: 11
Default

EDIT2: Sorry, could you share how you extracted it? I don't think it's a GPDA header .dat so tenoritool can't extract it. I apologize for any inconveniences. :S

EDIT: Nvm, found it. It was something i failed to extract when I extracted from the ISO. Anywho, THANK YOU!

@k-wing
I received your email concerning where I can find the ornament text but I have encountered a problem. I can't find envpsp.dat in the files I've extracted with tenoritool. I'm sure I'm just being stupid overlooking something. Could you please provide me with the exact location of the file? For example USRDIR>FIRST.DAT>and so on.

I apologize for my incompentence in searching for files. xD
I've also emailed back but since you said that you probably wouldn't check that email account, I'm posting here too.

Thank you!

Also, even though you say I don't need to give credits I probably will anyway. It's my opinion that anyone who helps should be mentioned, and you are no exception! Unless you don't want to be mentioned. xD

Last edited by AngelicDreams; 2011-04-19 at 18:19.
Reply With Quote
  #20  
Old 2011-04-20, 09:57
Nanashi3
Guest
 
Posts: n/a
Default

I don't think envpsp.dat is a GPDA archive.

Likewise, there's another localizable item that wasn't handled earlier :
RES.DAT>main_resource.dat>main_resource_data.dat>flowmap.dat>map.dat
Reply With Quote
  #21  
Old 2011-04-20, 12:55
AngelicDreams's Avatar
AngelicDreams AngelicDreams is offline
Member
 
Join Date: Apr 2011
Posts: 11
Default

Quote:
Originally Posted by Nanashi3 View Post
I don't think envpsp.dat is a GPDA archive.

Likewise, there's another localizable item that wasn't handled earlier :
RES.DAT>main_resource.dat>main_resource_data.dat>flowmap.dat>map.dat
Yep, it was also something I couldn't extract. Both of them aren't GPDA. Though, for the flowmap, I think it's exactly what it is: a flowmap of the choices and endings and whatnot. Ornament text are in envpsp.dat according to k-wing, so he/she probably knows how to extract it.
Reply With Quote
  #22  
Old 2011-04-27, 14:41
Nanashi3
Guest
 
Posts: n/a
Lightbulb

To RoByte:
Does this fork for you :
http://forum.xentax.com/viewtopic.php?f=18&t=6313

with or without the index8 palettizing thing.
Reply With Quote
  #23  
Old 2011-04-27, 17:35
RoByte RoByte is offline
Visitor
 
Join Date: Apr 2011
Posts: 4
Default

Quote:
Originally Posted by Nanashi3 View Post
To RoByte:
Does this fork for you :
http://forum.xentax.com/viewtopic.php?f=18&t=6313

with or without the index8 palettizing thing.
gimconv with the flags --format_style psp --format_endian little appears to create a file with the correct header and I can load it with a GIM viewer. I tested converting TKA0020A.gim to PNG and back. The resulting GIM is three times the original file size and didn't work when I inserted it back into image_tukkomi.dat.

I tried combinations of the other flags and never seemed to get anywhere close to the original file.
Reply With Quote
  #24  
Old 2011-04-28, 02:52
Unregistered
Guest
 
Posts: n/a
Default

I tested this, it works, but alfa channel of image is lost.
There are 3 copies of TKA0020A.gim, you should probably change them all.
I used gim2bmp and gimconv from tenoritool archive, and compiled Gpda from k-wng.

java Gpda res.dat
copy res\image_tukkomi.dat\TKA0020A.gim TKA0020A.gim.gz
gzip -d TKA0020A.gim.gz
gim2bmp TKA0020A.gim
<edit bmp>
gimconv TKA0020A.bmp --format_style psp --format_endian little --pixel_order faster --image_format index8
gzip -n9 TKA0020A.gim
move TKA0020A.gim.gz TKA0020A.gim
copy TKA0020A.gim res\script.dat\AKYO_0020T.dat\AKYO_0020T.dat_1\AKYO_0020T.dat_2
java Gpda res.dat.txt
Reply With Quote
  #25  
Old 2011-04-28, 05:43
Nanashi3
Guest
 
Posts: n/a
Default

why not gimconv TKA0020A.gim -o TKA0020A.tga
to preserve alpha channel?
Reply With Quote
  #26  
Old 2011-04-28, 14:56
Unregistered
Guest
 
Posts: n/a
Default

I didn't know about that option,
I tested some more, even with png and a bigger file, it works.

And I forgot to mention, I used UMDGen to reinsert RES.DAT back into the iso.
Reply With Quote
  #27  
Old 2011-05-05, 12:56
Unregistered
Guest
 
Posts: n/a
Default

There's a guy on youtube posting CCed translations of the game in play. Maybe you can talk him into joining your project.

http://www.youtube.com/user/zidanechoo2
Reply With Quote
  #28  
Old 2011-05-06, 20:30
AngelicDreams's Avatar
AngelicDreams AngelicDreams is offline
Member
 
Join Date: Apr 2011
Posts: 11
Default

Edit: Yep, they're translating from the Chinese fan translation with some exceptions while editing.

I will attempt to contact them, though the translation style from what I've seen would be annoying for me to edit to the same style as our current translations. Other than that, there is the problem I am concerned about. If those translations are from the chinese fan translation, then the translation will be a problem. Basically, From Japanese>Chinese>English, meanings or literal translations and other technical issues tend to happen, and would not be accurate enough as it has gone through different languages. I feel that this would affect our translation quality. We'll see how it goes.

Last edited by AngelicDreams; 2011-05-07 at 14:01.
Reply With Quote
  #29  
Old 2011-05-07, 12:40
RoByte RoByte is offline
Visitor
 
Join Date: Apr 2011
Posts: 4
Default Image insertion working

I finally got the image insertion working with a combination of the above methods.

gimconv TKA0020A.gim -o TKA0020A.tga
(Edit the TGA with GIMP)
gimconv TKA0020A.tga --format_style psp --format_endian little

I had been trying to put the modified file into image_tukkomi with no success. Now I'm only inserting it into the appropriate T script and everything appears to be working so far.
Reply With Quote
  #30  
Old 2011-05-16, 18:25
RoByte RoByte is offline
Visitor
 
Join Date: Apr 2011
Posts: 4
Default Source code posted

I've uploaded the code to a few of my tools to SourceForge. https://sourceforge.net/projects/oreimo/
I'll upload more once they're a bit more mature. I've re-written my inserter at least a dozen times so far.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Oreimo/Haganai PSP - Rinjinbu Looking for JP-EN Translators/TLCs *8/3/13* Ziddy Production & Help 10 2013-03-07 08:21
Hacker Wanted Minimoto Production & Help 12 2009-10-08 22:01
[Kono sure wo] Key Staff Diary 2 [gou ni ninmei suru yo!] AstCd2 General Discussion 39 2005-09-25 10:18
Key staff diary AstCd2 General Discussion 113 2005-02-01 09:25


All times are GMT -8. The time now is 07:16.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2021, vBulletin Solutions, Inc.