PDA

View Full Version : Will onscripter support unicode in the near future?


KnightBaron
2007-08-19, 02:03
I'm translating narcissu into local language but the problem is onscripter can't read the script in my language. (ANSI and UNICODE) I have to remake the font that replace english character with my local character (Thai) and that is a load of work and the output isn't good enough. I'm also trying to port everything into Ren'Py (which support my language) but that is too much work too.
So, the question is will onscripter support unicode in the near future?
Well, sorry for my bad english though.

Asceai
2007-09-09, 05:46
I wonder how much trouble it would be to hack the ` command thingy to support UTF-8.. I haven't looked at onscripter's source, but I wouldn't think it would take too much work.

Agilis
2007-09-09, 07:46
I'm not much for reading source, but a long time ago, talking to someone that worked with onscripter a great deal, it was described as "the worst mass of spaghetti code he's ever seen".

If I remember, making it so that the number-pad enter key would advance the text became a non-trivial task...

Asceai
2007-09-10, 01:54
Well, I went and had a peek at onscripter. Hacking this in may be fairly trivial afterall--what I'm going to try is throwing out shift_jis support and replacing that stuff with utf-8.

Asceai
2007-09-10, 02:01
Well, it was pretty easy to get it to display 2-byte UTF-8 characters. Thai is probably in the 3-byte range though, and this would take some more work.

pbsaffran
2007-09-10, 12:37
Woot?
You know, a build with that would be of high interest to most EU fans.
Good job!

Unregistered
2007-09-10, 17:54
Hm... if you can display 2-byte characters, why not just use UCS-16.

Roto
2007-09-10, 19:12
The problem with UCS-16 is that everything (even the basic ASCII set) is a two-byte character. NScripter uses single-byte characters for its commands. You'd have to modify ONscripter to take its commands as UCS-16 strings (and it's all implemented with C strings, not some fancy string library, so you'd probably be looking at a significant rework of the script parser.)

The variable character length of UTF-8 works fine with the existing script parser in terms of commands. Unfortunately, the variability means that text display routines must be altered to handle this variability. As Agilis mentioned earlier, the ONScripter code is *not* a prime example of well-organized, modular code, so there are several different places you have to modify the script parser and text display routines. Tracking down and dealing with edge cases is a bit of a pain; getting basic UTF-8 support working is fairly straightforward but you'll find it breaks on menu choices, fast forwarding, log mode, and other similar cases.

Another factor that must be considered is that ONScripter uses fixed-width fonts. Roman characters are rendered exactly half as wide as Japanese characters, on a fixed grid. It determines whether to use full-width or half-width based on whether a character is one or two bytes. That approach does not work for UTF-8; many multi-byte characters, even in a fixed-width font, are half-width; and some (accents) don't advance the cursor at all. To really provide full Unicode support, it also becomes rather necessary to implement variable-width (proportional) font spacing, especially for languages that don't work with fixed-width characters as well as English or Japanese.

I've done a little bit of experimental work on both areas (see here (http://forums.visualnews.net/showthread.php?t=7452) and here (http://roto1.net/misc/dev/ons_prop/unicode.png)) but it's based on an old version of ONScripter, highly experimental, and I haven't been able to work on it in quite some time.

-
2007-09-11, 08:58
If ONscripter doesn't come around, I guess the second best thing would be to port the whole thing over to KiriKiri (which supports UTF-16 just fine), especially if you don't have branching anyway like in Narcissu. KiriKiri's syntax is close enough to ONscripter's to make it possible with just a few regular expressions. Well, you'd probably have to redo the menus, but that's not THAT much work.

KnightBaron
2007-10-03, 07:52
The problem is how could I learn how to use Kirikiri? I don't know japanese language at all.

PyTom
2007-10-03, 08:20
I would be willing to help people port games over to Ren'Py, if they think that's the way to go.

Asceai
2007-10-03, 11:49
An automated nscripter->renpy tool might be a fun idea. How feasible do you think it would be?

Then we just need to have similar tools for blade and kirikiri, just in case anyone forgets what the best open visual novel engine is! =p

PyTom
2007-10-03, 20:37
I don't think it would be terribly hard to do, at least for a basic translation of the dozen or so commands that are most used in a game. I'd consider doing this if someone was quite serious about a translation.

Asceai too lazy to log in
2007-10-03, 20:51
I think the easiest thing would be for assisting conversion to ren'py.. so, while the actual game script might be pretty easy to convert over, stuff like the menus and such would be more work.

KnightBaron
2007-10-04, 21:39
An automated nscripter->renpy tool would be a great help. There're many good novels on nscripter engine. I think many games could be translated into other languages easily if such a tool is available. I'm also considering to code it myself but please don't expect anything of me since I'm not a professional coder.
Again, sorry about my bad english. Hope you understand what I'm trying to say. :P

Turnsniper
2007-10-07, 10:08
Wow Asceai, did you say that you managed to make ONScripter display 2-byte characters!? Did you hack into ONScripter-EN or the original?

I'm asking because I've been trying to do that for ages, the problem is I'm a translator not a programmer so I had no idea of what I was doing.

I need a version of ONScripter-EN that supports unicode characters from my language, like accented vowels, tilded n and such. If it's not a problem can you send me that modified version you made or at least guide me on what I have to do, I would really appreciate it since this problem has been frustrating me for such a long time.

Unregistered
2007-10-08, 09:21
ONScripter can always display 2-byte characters. It can even do so in single-byte (backtick) mode, given that the offset of the character is even. So you could possibly make it display the right glyphs if you supply the right font.

Turnsniper
2007-10-09, 16:42
Well I have tried a lot of fonts before and I couldn't make it display those 2-byte characters I need.

I dont understand what you mean by "single-byte (backtick) mode", is there a way to switch modes in ONScripter?

zalas
2007-10-10, 06:57
Lines beginning with ` are marked as single byte script lines in the ONScripter builds insani uses.

dsp2003
2007-11-04, 21:33
All problems with the worldwide languages support for OnScripter is on the way of being solved. ^_____^

It will be a surprise for you all, but Haeleth is working on the Unicode-oriented fork of the OnScripter since the summer of 2007th. The first ready-to-be-released title powered by "Proportional-OnScripter" is "Red Twilight", a Russian translation of the "常世の星空" ("Red Shift" in the Insani's English translation).

Please, don't bug Haeleth about the POnScripter right now, because we (me & zagevalo) haven't released our piece of work yet (there's a few bugs in the engine and they're needed to be fixed).

More info coming soon here:

http://www.dsp2003.narod.ru/red_twilight.htm

dsp2003
2007-11-15, 03:27
Red Twilight has been released. Now you can bug Haeleth about POnScripter... :D

Turnsniper
2007-11-15, 10:53
Well I tried to mess around a little with this POnScripter but couldnt do much at all. I investigated about this "Red Shift" and I guess you guys modified that experimental Onscripter branch insani used for the game.

I was hoping POnScripter could help with my problem with the unicode characters but couldnt make it to work with my script, does POnScripter have all the functions ONScripter-En has? if so, which are the changes in the command references?

dsp2003
2007-11-15, 11:51
There's a few changes:

1. The script MUST be encoded into UTF-8 instead of Shift-JIS and called 0.utf, not 0.txt (otherwise the backwards-compatibility mode will be activated).

2. The "`" character is not used for 1-byte characters anymore - you must replace it with "^".

Note: unlike "`", the "^" is situable for Unicode characters as well.

3. Ruby & vertical text is not supported anymore.

4. Window coordinates & proportions is now differs from the original (see our script for reference).

5. ;mode800 directive isn't working (due to a bug, you'll have to wait for the Haeleth's release of POnScripter).

6. This is not the modification -- it's a separate dev fork.

Yup, the 2007-07-21 version is not 100% OnScripter replacement, it's an unstable experimental build, unfortunately.

Haeleth
2007-11-15, 15:15
Hee, looks like the cat's out of the bag. :)

Please note that I will not be supporting Ponscripter until I've released it myself. The version used in Red Twilight is already somewhat behind the development version in many respects, and the whole program is still a bit buggy and underdocumented. Use at your own risk.

(In the specific case where Unicode is necessary to display text properly in a given language, I'm willing to offer extremely limited support of the sort I offered dsp2003's team. Drop me an email.)

dsp2003
2007-11-16, 01:12
We're using the outdated revision, yes. And that's not our fault. >_>

A little bug report\request:

1. caption directive doesn't supports unicode characters (shows "?" instead).
2. Where's the MOD\S3M\XM\IT music support (the weight of those is a lot less than MP3\OGG - saves a lot of space and improves the quality of synthesized music)?
3. Do we have to rebuild src every time to translate the built-in menu strings ("Yes"\"No", "Return to Title?" etc.)? It will be a logical to give them a few of script alias. For example:
gui_yesno "^Yes","^No"
gui_returntotitle "^Return to Title?"
gui_quit "^Quit?"
gui_months "^Jan", ... ,"^Dec"
...or even allow to substitute a bitmapped buttons and captions! ^_^
gui_yesno ":c/2;gui/yes.png",":c/2;gui/no.png"
It's not a useless stuff, it's a must-have simple things (even outdated closed IKURA CDBros engine have those).

No offence, but in the current state POnScripter doesn't looks like an progressive Open Source project at all. :(

pbsaffran
2007-11-16, 15:12
Well I still am only 6% into the Higurashi code so I guess I can wait...

dsp2003
2007-11-18, 22:44
Whoops! :-D

Directive ;mode800 is working. The problem is in the Windows-generated UTF-8 files (you'll have to cut out the first 3 bytes from them in an HEX Editor -- BOM header - 0xEF 0xBB 0xBF).

pbsaffran
2008-04-01, 07:59
Let's give this a mighty bump.
I have gotten free time for a few weeks and in that time, I want to finish the first Higurashi game. I am now about 80% done with it.
I have looked a bit around for PONScripter and various versions allowing special characters, but I couldn't find any in a compiled state.
Does anyone know if a release is planned or do I have to learn C++?

dsp2003
2008-04-03, 07:19
Please ask Haeleth... i doubt he'll release the full package in the nearest future (the English version of Narcissu Side 2nd is still w.i.p., doh...) >_>

dsp2003
2009-02-02, 09:08
Will PONScripter be released at least in the current year? T_T Haeleth.... pleeеееееase.... soooo sloooow....

Agilis
2009-02-02, 11:44
The answer to your question is deeper than you think.
Narci2 runs on PONscripter, and has been designed to work with it from the start.
Moving to anything else would be a logistical disaster at this point...
When you see one, the other can't be too far behind.

dsp2003
2009-02-02, 23:11
The thing is... I'm being got at. Since 2007th, Russian community grew up a bit (the hell, it's not a "bit" but "kilobyte") and every noobish creature is seeking for PONScripter, and of course they've been directed to me, even at the anonymous imageboards like iichan.ru. Of course, I'm doing my best to redirect them to Haeleth, but it starts to annoy me too much.

I understand completely that this is my problem, I should've keep my mouth shut but I wasn't. Stupid, so stupid... T_T

pbsaffran
2009-02-03, 07:44
Can't they use the build in your russian project, at least for the time being? It's not like your build didn't work at all, is it?

dsp2003
2009-02-03, 08:45
pbsaffran, yep, it works, but do I have a right and\or permission to put it at my site (including source code) for the public downloading (since my internet connection is not cheap)? I don't think so.

jyuichi
2009-02-03, 18:31
pbsaffran, yep, it works, but do I have a right and\or permission to put it at my site (including source code) for the public downloading (since my internet connection is not cheap)? I don't think so.

Its GPL/open source isn't it? If so,you have the legal right to redistribute it in any form (with the license).

Hosting costs are different problem...

Sonozaki Futago-tachi
2009-02-03, 20:43
pbsaffran, yep, it works, but do I have a right and\or permission to put it at my site (including source code) for the public downloading (since my internet connection is not cheap)? I don't think so.

You could just use a file-sharing service like mediafire. We do that - haven't spent any money on blog or hosting (just on games & lots of translation time...)

So long as you include the GPL and give contact info so someone could ask for the source code, I don't see any problem.

Mion (Sonozaki Futago-tachi)

dsp2003
2009-04-04, 09:26
Well, I finally did...

http://sl.criminalrussia.ru/dsp2003/onscripter.htm

Here you'll find 2 binaries and 1 source code 7-zip packages: modified 20070721 build with "localestring" feature and original 20080326. The source is only available for 20070721 mod.

Sorry, folks. That's all I have.

P.S. Please note: most of POnScripter builds represented there are unstable and it's highly NOT recommended to use them.

pbsaffran
2009-04-05, 03:02
The link to the modified pons doesn't work. Was that intended?

dsp2003
2009-04-06, 00:28
The link to the modified pons doesn't work. Was that intended?
Whoops! ^^' Fixed.

pbsaffran
2009-04-06, 02:32
Thanks.
Now I just have to find out why the games work with one version then bug with the other version.
Also, your 20080326 is like 2.4mb and mine was only 870kb. I am jealous. (not really)
I'll try things around and see what is recognized and what not. (at a end-user level of course)

dsp2003
2009-04-06, 03:17
A~ha~ha. Actually, the EXE of yours is compressed with UPX, but I've decompressed it for better re-compression with 7-Zip\LZMA.

pbsaffran
2009-04-14, 04:05
Ah well, a guy can try.
No really new development trying out your versions - I did find it funny though that switching from your build to mine (which are essentially the exact same, if I understand correctly) lead to reading errors. This is really way beyond me.

dsp2003
2009-05-09, 03:00
Well, now I'm a bit annoyed. While OnScripter-En is truly functional-rich (now supported bunch of new commands, including recreated mp3fadeout, mp3fadein and others, lots of built-in effect.dll's emulation), the PONScripter is still out of business... how cruel, it makes impossible to translate lots of nice VNs. :(

pbsaffran
2009-05-09, 07:44
I guess it IS a problem with all the new script functions designed in the newer games, but I for instance have almost no trouble adapting Higurashi. The last few bits needed are localestring messages.

Sonozaki Futago-tachi
2009-05-09, 10:35
Well, now I'm a bit annoyed. While OnScripter-En is truly functional-rich (now supported bunch of new commands, including recreated mp3fadeout, mp3fadein and others, lots of built-in effect.dll's emulation), the PONScripter is still out of business... how cruel, it makes impossible to translate lots of nice VNs. :(

I'll get PONScripter updated when I can... ^^;

Just more familiar with the ONScripter code when it comes to adding features.

Mion (Sonozaki Futago-tachi)

dsp2003
2009-05-09, 22:03
I'll get PONScripter updated when I can... ^^;

Mion, that's not the point. ^^||| I know you're making a great efforts for this engine (a lot more than Haeleth himself, I think). The thing is --- there's still no "official" distribution, the SVN is login+password-protected.

I know, I'm a bit impatient (but waited for 2 years though), so... there will be no harm (I think) if something like up-to-date pre-release will be putted outside of <s>beats's lair</s> Haeleth's hard drive. ^3^

Skullchukka
2009-05-29, 06:27
Hello everyone. I've got a question.
I have the intention to translate Umineko into my local language. (Turkish)
I've got the required tools, and ponscripter.
When I decrypt the nscript.dat into a .txt file and open it; after editing any line with Turkish, I realised not even ponscripter can read it.

(For example; I'm typing "düzgün" and since "ü" is a Turkish letter, it only appears "dn" when I run it with ponscrypter.)
I've renamed nscript.dat into pscript.dat, it still didn't work. Am I missing out something?

Asceai
2009-05-29, 06:47
Hello everyone. I've got a question.
I have the intention to translate Umineko into my local language. (Turkish)
I've got the required tools, and ponscripter.
When I decrypt the nscript.dat into a .txt file and open it; after editing any line with Turkish, I realised not even ponscripter can read it.

(For example; I'm typing "düzgün" and since "ü" is a Turkish letter, it only appears "dn" when I run it with ponscrypter.)
I've renamed nscript.dat into pscript.dat, it still didn't work. Am I missing out something?
I just tried it myself, and ponscripter supports ü just fine. Are you sure you aren't screwing up the file encoding or something like that?

The only copy of ponscripter I seem to have on this machine is the one that came with the narci2 microtest, but it has no issues with umlauts. (http://img8.imageshack.us/img8/8263/narcissu.jpg)

dsp2003
2009-05-29, 07:40
Skullchukka, it was mentioned before in the same thread, but I'll post anyway. You'll need 2 things:

* UTF-8 encoded script file
* Font file with required character set

phlk
2009-07-07, 16:58
Hello there.

I'm using the PONScripter in an attempt to produce an original Visual Novel with a group of friends around here. Our native language need some characters that the english ONScripter doesn't support, so, this is the only option we have right now (besides producing a whole new engine -- which I'm not willing to, haha, or using another tool -- which I'm not willing also.)

It was all good until I got this bug: When I use the systemcall windowerase and then move the mouse cursor during the time the game freezes out or crash. Note that this also happens with the Haeleth's ONScripter windows binary that's available for download on his website.

Is this a known issue? I just downloaded Sonozaki Futago-tachi's version and it seems to be working properly without any code changes (except for the ^'s obviously). If there's any known solution, I'd like to hear.

Thanks in advance.

Edit:Ah yeah, thank you very mucy dsp2003, for making PONScripter available. :)

Sonozaki Futago-tachi
2009-07-07, 19:52
Hi phlk,

Funny you should ask :) I was just fixing some ponscripter bugs today, that one among them. Here's a new build: http://www.sendspace.com/file/bayg3t

Don't hesitate to mention any other problems.

Mion (Sonozaki Futago-tachi)

phlk
2009-07-07, 19:55
Whoa!

That's what I call effective support! Hahaha

Thank you very much!

dsp2003
2009-07-08, 05:31
Another "bug" is the volume settings doesn't work for OGG Vorbis played at BGM channel.

pbsaffran
2009-07-08, 05:47
YOu seem to have fixed the h_fontstyle command in that build, as well. Impressive.

We're almost there, aren't we?

Sonozaki Futago-tachi
2009-07-08, 06:20
No, still a ways to go... I'll need to incorporate a bunch of what I changed in onscripter-en, which includes bugfixes...

I should get something together to handle bug reports from you guys ^^;

But anyway, hope this helped with some nasty showstoppers. Definitely let me know about stuff that crashes the program.

Mion (Sonozaki Futago-tachi)

dsp2003
2009-07-27, 08:52
How about Linux build and\or source code? ^^

The thing is... Vendor of Honyaku-Subs started to port ONScripter VNs to Ren'Py, since there's no Linux build available. T_T'

Unregistered
2009-10-21, 04:07
Anyways, which application should I use to convert my script file into the .dat file (for POnscripter)? The one I've got (from Insani) doesn't seem to be able to convert it properly. Download link would be very appreciated. Thanks.

dsp2003
2009-10-21, 05:47
Unregistered, rename your 0.utf to 0.txt and do the same thing as you did before, goddamn it. Or take a goddamn hex editor and xor the whole file with 0x84, then rename xored one into pscript.dat.

Note: this will not help you to hide your script source from advanced users.

Unregistered
2009-10-21, 07:04
Since the file has been a .txt file from the start (when I’ve ripped it from the source), I’m assuming that .utf is just you referring to a UTF-8 encoded .txt file. In that case, yes, I’ve saved it on that format, ran the application, but the result was unsuccessful (not able to convert properly <> not able to convert at all). If .utf refers to a completely different thing, then I’m in a total loss.

Bear in mind that I’m not a lifelong user of this engine and I’m trying to use POnscripter to translate an Onscripter VN into my native language. In other words, I didn’t get much of what you’ve said and I’m trying to make myself clearer. Being scolded as a noob and still get a technical answer, as if I could understand it, was pretty funny, though.

Note: I really don't care about hiding my script from others; I just want to make it work.

dsp2003
2009-10-21, 08:06
Way of conversion from ONS to PONS is like that:

1. Convert 0.txt from Shift-JIS to UTF-8.
2. Important: replace ` for non-japanese text with ^, otherwise your text will be considered as japanese and any spaces will be ignored in text window output.
3. Rename converted 0.txt to 0.utf
4. ??????
5. PROFIT!

Unregistered
2009-10-21, 08:37
HAHAHAHA

So thaat's what you meant. Now I just feel dumb. XD While I must say I'm pleased that I can just type in the script and run it directly, isn't there any kind of packaging at all? I mean, usually the .dat file is compressed in comparison. If it's more trouble than it's worth, I might as well just drop it. Also, it might be convenient adding that note somewhere.

Anyways, thanks a lot for the help and sorry for the trouble. o/

Roto
2009-10-21, 08:41
No, compression isn't supported for script files. The .dat file is literally just the text file with every byte XOR'd with 0x84. The only thing it really does is prevent people from seeing a text file and just double-clicking it and trying to read it.

On Mac OS X, you can put game resources inside of the application bundle, which makes things a lot cleaner, but there is no bundle support on Windows. In PONScripter, there's also preliminary support for packaging certain game resources inside the executable itself, but I'm not certain if that works for the script file (it's used for the game icon in the distributed PONScripter builds, but I haven't really tried it for anything else.)

And yes, PONScripter documentation is a little sparse right now. Don't hesitate to ask about stuff if you get stuck again.

Unregistered
2009-10-21, 10:05
Hum. So I've probably misunderstood it and misinterpreted it. Although, while there isn't script compression, it's possible to use a .nsa file to pack all images, etc. neatly and save some HDD space.

And thanks again for your words of encouragement.

dsp2003
2009-10-21, 10:26
Just a little hint (and my personal preference): never compress your distribution internally except the streamed music and sounds. With current file compressions like LZMA and even zlib it's a LOT more effective to compress the whole data at once than each file\datablock individualy. This strategy will cause your distribution to take more space at the user's side, but will allow you to make your distribution package smaller.

Good example of this is Aoi Shiro - it takes about 7 Gb installed, but perfectly fits on single layer DVD as installer. That's because mostly all of it's CG data isn't compressed.

helios
2009-10-30, 19:33
That only applies if none of the files are compressed, which is actually pretty stupid. Even if LZMA is able to find that several BMPs are remarkably similar, it'll still do a worse job than lossy algorithms like JPEG or OGG.
For example, here I have a resource archive. As JPEG, it uses 83.8 MiB. As BMP, it uses 708.8 MiB (JPEG ratio: 11.8%).
Compressed with 7-Zip with all compression parameters maxed out, the BMPs go down to 124.8 MiB (LZMA ratio: 17.6%). The JPEGs go down to 79.1 MiB (94.4%).
If I extrapolate this to your example, that DVD could have been reduced to (assuming the DVD is completely full [4480 MiB]) 3000 MiB installed (of course, it depends on what percentage of that were images).

dsp2003
2009-10-30, 23:44
helios, I was talking about LOSSLESS encoding, not about crap called JPEG.

helios
2009-11-04, 18:37
Even PNG performs better than LZMA. You can't compare special purpose compression to general purpose compression.