PDA

View Full Version : xclannad on Mac OS X


Roto
2005-10-23, 14:54
Between wanting to play things like the Tomoyo demo and the less expensive Standard Editions of Air and Kanon without having to start up Virtual PC and the fact that there's no RealLive interpreter for the Mac, I started messing around with trying to build xclannad on the Mac.

So far, I've got everything ported over into an XCode project, and after a lot of tweaking, it actually compiles.

Screenshot (Main menu) (http://www.roto1.net/misc/dev/xclannad.png)
Screenshot (Config) (http://www.roto1.net/misc/dev/xclannad-config.png)

It brings up the menus in CLANNAD and one can navigate through most of the menus without crashing. Color/transparency is obviously an issue (the Key logo is barely visible, and everything that's supposed to be transparent is appearing as yellow.) As soon as I click "New Game" it crashes. I've tried the Tomoyo demo as well, but it crashes on launch (not all that surprising.)

All of the sound stuff is disabled (Mac OS X handles audio significantly differently than Linux) and I've had to disable a few things like setting the window title and movie playback that were using features of SDL that are specific to X11.

My C++ is shaky, I've never programmed for X11 or SDL before, and I really don't know all that much about RealLive, so I'm kind of using the "tweak it until it works" approach here. It's more of an experiment than anything. Obviously, it's not usable at all, but it's a step in the right direction. I will post my XCode project if anyone is interested in working on it or playing around with it.

I was just curious whether anyone else has tried to get xclannad running under Mac OS X.

mr.aufziehvogel
2005-10-23, 15:30
i can't tell for sure, but maybe these color errors are occuring because the powerpc processor of the mac's only work in big endian mode (while the x86 cpu's are working in little endian mode)
what kind of cpu do you have? g4 cpu's are supporting little and big endian but the G5 cpu's only work big endian.

Roto
2005-10-23, 15:43
My processor is a G4.

So far from what I've seen, the code is fairly endian-independent. The code seems to suggest that it will run on sparc, and sparc is most definitely big endian.

I'll check out the graphics functions though and see if there might be somewhere that byte order is hard-coded.

Haeleth
2005-10-23, 16:09
You probably want to apply some of my patches (http://dev.haeleth.net/xclannad.html) if you haven't done so already. In particular, "Read flags" is needed to run just about any game more recent than Clannad.

You will probably also find RLdev (http://dev.haeleth.net) generally useful if you're planning to try to work on xclannad. Even if you daren't attempt to build it, it does include the only publicly available RealLive documentation I'm aware of.

Finally, bear in mind that xclannad is very incomplete, and in some places plain wrong, so don't necessarily expect its code to make much sense. (The implementation of the sprite/object system, for example, is clearly based on attempting to reproduce observations, rather than analysing the underlying system. I've tried once or twice to fix it, and every time backed down after realising that it would probably take a complete rewrite of the subsystem...) Good luck if you do decide to try to fix stuff, though. :/

Roto
2005-10-23, 16:57
Yeah, I'm aware of the patches. I've tried them briefly, but thought it might be best to at least get it semi-working with Clannad first, at least until I understand RealLive a bit better to figure out what's going on. The patch I thought least likely to break Clannad was the operator precedence patch, but it breaks Clannad (it launches, but I get a black screen with the cursor and nothing else.)

I've been trying to build RLdev. I have all of the prerequisites listed in the INSTALL file, but keep hitting this error when I try to run omake:

Cannot find file /usr/local/lib/ocaml/site-lib/ulex/ulexing.cmxa

The contents of /usr/local/lib/ocaml/site-lib/ulex are:
META pa_ulex.cma ulex.cmi ulexing.cmi
cset.cmi pa_ulex.cmi ulexing.cma utf8.cmi

Any idea what the problem is/what I need to do differently when installing ulex?

Finally, bear in mind that xclannad is very incomplete, and in some places plain wrong, so don't necessarily expect its code to make much sense.

I kind of gathered that from the xclannad page, your comments, and the source code... I suppose if I get really ambitious I'll get out my disassembler and take a close look at RealLive, but that's not likely to happen ;)

Haeleth
2005-10-24, 04:17
The patch I thought least likely to break Clannad was the operator precedence patch, but it breaks Clannad (it launches, but I get a black screen with the cursor and nothing else.)
Oops. Yeah, I just tried that and got the same result. Er, I, uh, must have forgotten to test it again after Jagarl released 0.06... ehehe, I'll try and fix that... ;_;

Erm, also the online documentation is broken. The HTML manual in all the downloads has incorrect formatting that makes all the examples illegible. Use the version on the website instead. ;;_;;

I've been trying to build RLdev.
Brave. Allow me to apologise for its rather esoteric composition.

Any idea what the problem is/what I need to do differently when installing ulex?
ulex has a silly makefile where "make all" doesn't make all. You need to do "make all.opt" as well. That should sort it. (I'll add a note to INSTALL.)

Haeleth
2005-10-24, 07:10
Update: I've fixed the precedence patch, and tested the lot; I can confirm that Clannad now works just as well with all my patches as without (proof (http://haeleth.net/temp/lookitworks.jpg)).

As for your crashing upon starting the game... where are stdout and stderr going? xclannad prints a lot of diagnostics, which might give some hint as to exactly what's not working for you that works for me.

Roto
2005-10-24, 09:52
Cool. Clannad's now getting to the menu with all patches installed.

On stdout, I'm not getting anything besides this:


Settings:
Locale : C
Root Path : /Volumes/CLANNAD
Font : /cc.ttf
CD-ROM : /dev/cdrom


On stderr, I get a bunch of the command unsupported 0x23 errors that seem to correlate with fading something on the screen. At the time of the crash, SDL segfaults and xclannad exits with status 245.

stderr.txt (http://www.roto1.net/misc/dev/stderr.txt)

Attempting to launch the Tomoyo demo results in a segfault in SDL and an exit status of 246.

I got rldev to build (well, kprl and rlc anyway.) I've been having some trouble with xml-light and getting it installed in a manner that ocamlfind will actually find it, but I'm working on that...

Edit: I got xml-light installed with ocamlfind, but now I'm getting this error when I try to build vaconv:

Files image.cmx and /usr/local/lib/ocaml/site-lib/xml-light/xml-light.cmxa
make inconsistent assumptions over implementation Xml

Haeleth
2005-10-24, 11:35
Re crashing: hmm, no very obvious cause.

The 0x23 errors are expected - you'll get one of those for every opcode xclannad fails to recognise, and there are thousands it doesn't know about. (0x23 is the byte that begins a command opcode in RealLive bytecode.)

Re xml-light: the "inconsistent assumptions over interface" thing normally happens when you have two files that were built with different versions of a library... did you manage to partially build vaconv before getting xml-light installed properly? If so, an "omake clean" in the vaconv directory might solve the problem.

It's probably not a big problem if you can't build vaconv, though; there are alternative bitmap convertors, like the one insani used on planetarian, which is written in C with no dependencies. Just reconfigure with "--no-vaconv --no-rlxml" if you find the hassle getting too much. ^_^

Roto
2005-10-24, 12:08
The crash seems to be happening on line 1057 of scn2k_cmd.cc:

if (*d == 0x23) {

I gather that d is supposed to be pointing at the current location in the seen file in memory. (If I set a breakpoint there and launch the program, it's pointing at a location in memory that contains SEEN9030, as expected.) At some point it's getting pointed at a bad location and the segfault happens. I'm in the process of trying to figure out what causes that to happen.

I tried an omake clean, but still the same error building vaconv. I'll look into insani's converter. I just want to see what some of these graphics are to better understand what's going on and maybe provide some insight into the transparency problems.

zalas
2005-10-24, 13:12
Perhaps it is jumping to the wrong address? Is there some sort of indirect jump in RealLive, like for instance jumping to an address that is stored in a variable? I suspect perhaps that the variable was uninitialized from some previous code.

<offtopic>
I read omake as おまけ -.-;
</offtopic>

Roto
2005-10-24, 14:09
Whatever it is, it's inconsistent. Sometimes I can get further into the game than others. Sometimes it crashes right after clicking New Game, sometimes it lets me get through the stuff with a white background, and I've gotten as far as partway through the sequence with the credits.

Is reallive.kfn the best reference for the bytecode? Or is there a better document somewhere to which I should be referring?

Edit: Every time before it crashes, the last command it executes is

0x23 - cmd 00-01&#58;000d&#58;00&#91; 0&#93;

Haeleth
2005-10-24, 14:40
Is reallive.kfn the best reference for the bytecode? Or is there a better document somewhere to which I should be referring?
I'm afraid it's the best there is. There isn't a single document: the correspondence between bytecode and function names is defined in reallive.kfn, and their functionality is explained in the reference section of the manual. I never got round to combining the two. (But I will be quite happy to field any questions you can't dig out the answers to.)

Is there some sort of indirect jump in RealLive, like for instance jumping to an address that is stored in a variable?
Not in the bytecode itself. But the implementation probably uses all sorts of variables to store various addresses in, that could be getting clobbered or left uninitialised.

In this case, I have to say that, totally on a hunch, I suspect what's getting clobbered is the call stack; Clannad uses an awful lot of inter-SEEN jumps to display graphics and the like, so if there's a problem in the graphics code (as the bright yellow bits suggest), it seems quite plausible that that's what it's affecting.

One thing you could try would be applying this (http://haeleth.net/temp/xc-trace.patch), which enables the debug printfs to log all SEEN changes - then check the stderr output to see if the segfaults tend to correlate closely with them.

(The patch also adjusts the opcode logging to display stuff in the same format as reallive.kfn and the actual internal RealLive debugger, which may be useful.)

Edit: didn't see your edit. That pretty much confirms my suspicion: the command you cite is <tt>rtl</tt>, the "return from global call" opcode.

Roto
2005-10-24, 16:36
I appear to be looking for a heisenbug (http://www.catb.org/~esr/jargon/html/H/heisenbug.html). It seems like the more carefully I try to step through the code in GDB, the less likely it is to crash. I'm guessing, then, that something is not getting initialized properly or something's writing to memory it's not supposed to. I guess it's time for me to learn the finer details of memory management in C++ and step through the relevant parts of the program in my head.

Stepping the program in the debugger, I managed to get quite a ways into the game (partway through the second day.) I only got the game to crash after I cleared my breakpoints and just held down the shift key to fly through it all.

This allowed me to see a bit more of the graphics problems. Some of the graphics formats used by RealLive seem to do fine... the PDT cursors and whatever is used by the CGs and backgrounds, for example. One of the formats seems to be resulting in the yellow pixels, and yet another seems to have the channels swapped (without carefully analyzing it, it looks like it's using red for alpha, and alpha, green, and blue for RGB (though not necessarily in that order.) I suspect that for that case at least, mr.aufziehvogel's assessment is spot on and it's got the byte order messed up.

This, of course, yields some results that are rather interesting (http://www.roto1.net/misc/dev/xc-interesting.png) as well as some that are downright scary (http://www.roto1.net/misc/dev/xc-scary.png)!

Haeleth
2005-10-25, 05:13
What happens if you modify G00CONV::Copy_32bpp in system/file.cc to reverse each 4 bytes as it copies them, instead of just using memcpy? SDL isn't something I know much about, but maybe it's dependent on programmers sorting out endianness for it.

Incidentally, the PDTs used for the cursor aren't working fine: they're coming out with their RGB channels reversed, judging by your screenshots. (The alpha channels seem to be fine for some reason, perhaps because they're stored separately.)

The CGs and backgrounds, on the other hand, do indeed seem to be working. Those are G00 format 0, which also happens to be the only purely 24-bit format used by RealLive, which may be significant (it forces xclannad to load it pixel by pixel instead of copying whole blocks).

Roto
2005-10-25, 10:42
That fixed the colors on the type 2 G00s. Screenshot (http://www.roto1.net/misc/dev/xc-color.png)

Incidentally, the PDTs used for the cursor aren't working fine: they're coming out with their RGB channels reversed, judging by your screenshots.

Ahh, okay. The replacement colors didn't seem that off to me, but looking at your screenshot and after starting up the real RealLive interpreter in Virtual PC, I can see that they are. I'll work on that.

Aside from the cursor colors being off, most of the graphics look good now. The yellow pixels are gone (I'm guessing that they were a result of one of the color channels being misinterpreted as alpha.) I did notice, however, that the lens flare on the title screen (the one with the blue sky that pans up) has a translucent black background on it, so something's not quite right there. Perhaps it's a 16 bit graphic? (I haven't modified the 16 bit copy yet)

I've gotten closer to tracking down the reason for the segfaults. It's something having to do with mmap. I'm guessing that OS X's mmap doesn't function quite like xclannad's expecting it to under certain cases. I've temporarily disabled mmap and the segfaults on scene changes have gone away.

STaNLI
2005-11-01, 02:21
shmget No space left on device
I'd think this important in the crashing; On linux at least, I found that when xclannad crashes for some reason or another, it leaves some SHM/IPC/something memory allocated, and linux can only allocate 16 units of it (I can reboot, run xclannad 16 times fine, then on the 17th it always crashes with "No space left on device").

IIRC after segfault there were some xclannad processes still running with 0% cpu use, so they were off the botom of "top"'s output, and I didn't see them for ages :( Perhaps killing them would've worked, but for various reasons I never seem to be in a position to check and remember to at the same time (I'm at school now, and will have forgotten by the time I get home <_<)


> other crashes

Does OSX have an equivalent to valgrind? It adds bounds and other checking to C programs at runtime, so it can catch memory write errors as they happen rather than when something tries to use the messed up memory. It seems the original valgrind is x86 only though :(

Roto
2005-11-01, 12:16
Yeah, I noticed the shmget errors too. They're definitely a concern but didn't seem to be the cause for the crashes I was experiencing (It was still crashing even right after a reboot when it wasn't getting that error.)

Mac OS X has "Guard Malloc" that carefully watches how an application uses memory and stops it as soon as it tries to write somewhere that it's not supposed to. However, it's pretty resource-intensive (it uses virtual memory for everything, rather than real RAM, and of course it has to be constantly checking to make sure the program doesn't mess up.) My PowerBook has been out of commission with a dead logic board while I've been working on this, so I'm working on an iBook with limited resources... so finding and fixing memory problems with the debugger is a little bit tedious. Once I get my PowerBook going again, it will be easier to look at some of these things...

The crashing is definitely related to mmap. Since I've disabled mmap, I've yet to see it crash again while running Clannad. I haven't tried to go very far into it, but it hasn't been crashing in the spots it was before. For now, it's a temporary fix; I'll look into what's wrong with its use of mmap later.