Re: OHRRPGCE Data debugging, rc1

2008/12/13 David Gowers <00ai99 <at> gmail.com>:
> Hello,
>
>>
>> What a coincidence! Just two days ago, I was looking at how to load
>> OHR tilemaps and other lumps with Python. I saw that you used a little
>> bit of NumPy for this in Jormungand, but I used array.
> Array is really simple, it works fine for things like GEN which are
> basically a huge lump of different datas of questionable parentage.
>
> NumPy provides complex and nestable datatypes though, which are
> essential for clear data structuring.
> For example I have a convention where I number lumps of 'unused' data
> and treat them as one opaque
> lump; and in my GEN dtype, old scattertable is in it's own
> 'oldsctable' field (of 160 16bit ints).
> I also found this support for nested dtypes essential in handling the
> different types of string -- by
> separating the 'length' and 'value' fields I made strings mostly
> represented in a way that makes sense to the viewer.
>
>>
>> This is really giving me certain ideas :) Do you intend any other
>> purpose for this?
>
> Well, it's a doctest, so the real purpose of it is to illustrate how
> to do things.
> I hope you can read it and see easily what is happening, and feel free
> to insert lines of your own if you
> want to investigate some particular thing more thoroughly.
>
> However,
> What I plan to do with this is finish it up into a module with YAML
> serialization and some other features I mention in the doc, and upload
> it on PyPI, maybe with some simple PyGTK widget to display OHR-type
> images included.
>
> I would then hope that people would use it to make useful things like
> inter-RPG data transfer tools, literal 'graphic debuggers',
> during-game enemy/attack editors, and other appropriately kickass
> things :)

I look forward to it, and might be making such things

>>
>> How long has the crash on sound initialisation been happening? A few months?
>
> Dunno. My sound setup is kind of FUBAR admittedly (my speakers and
> speaker cables are proven working, but I only get half the stereo
> image, the other speaker is silent. In Windows, I don't get any sound
> at all).
>
> If I open up VIKINGS in custom and go to the Music menu,  it freezes
> as soon as I select song #2
> (presumably a ogg/mp3 one.). When I hit Ctrl+C, I am given this message :
>
> "Aborting due to runtime error 9 ("interrupted" signal) in
> music_sdl.bas::MUSIC_STOP()"

That sounds very much like this bug:
http://rpg.hamsterrepublic.com/bugzilla/show_bug.cgi?id=659

>
>>
>>>  * 'nose --with-doctests --doctest-extension rst memmapped-fileformats.rst'
>> Should be --with-doctest. Under windows (I should really boot into a
> Damn, I made that mistake all the time until I wrote a script for it.
>> hassle-free OS) I had to run
>> python C:/python25/Lib/site-packages/nose/core.py --with-doctest
>> --doctest-extension rst memmapped-fileformats.rst
>> You also forgot to mention yaml as a dependency. I think?
>
> I rather thought you could run it without yaml. see lines 134..145 --
> it explicitly checks for yaml installation and pretends that the yaml
> is OK if it can't actually use pyYAML to
> validate it.

Right, my mistake
>
> yamlarray.py rather ironically doesn't even try to use the yaml
> module, it's really just a hackish text automangler.
>
> After yamlization actually works consistently, I might consider a
> dependency on pyYAML.
> (I don't need to use pyYAML to write YAML, just to read it.)
>
>>
>> I don't really have time to have a good look at it, but I got a long
>> list of exceptions when trying to run it, I'm not sure if those were
>> expected or not. The first "Failed example" is:
>
> Hmm, I reduced them as much as possible.
>
> I get 5 exceptions, regarding:
> 1. loaded_ui_yaml  (no worky 'cause yamlization is currently broken)
> 2,4. 'No closing quotation' -- this means weird embedded string
> characters caused my yaml automangler to
> become confused
> 3. The autopadding code doesn't like finding that it needs to do no
> padding. harmless error.

You mean this?
Failed example:
   f.seek(-1)
Exception raised:
   Traceback (most recent call last):
     File "C:\Python25\lib\doctest.py", line 1212, in __run
       compileflags, 1) in test.globs
     File "<doctest memmapped-fileformats.rst[226]>", line 1, in <module>
   IOError: [Errno 22] Invalid argument

> 5. STT needs autopadding but doesn't have it yet, so chokes.
>

I also get a DeprecationWarning (I'm using numpy 1.2.1):
C:\Python25\lib\site-packages\numpy\core\memmap.py:261:
DeprecationWarning: ``close`` is deprecated on memmap arrays.  Use del
 DeprecationWarning)

followed by some exceptions in memmap.__del__

Besides that, all my other errors are to do with trying to access
/tmp, and the hdf5 stuff.

>>
>> line 373, in memmapped-fileformats.rst
>> Failed example:
>>    assign_recarray (uicolors, loaded_ui_yaml)
>> Exception raised:
>>    Traceback (most recent call last):
>>      File "C:\Python25\lib\doctest.py", line 1212, in __run
>>        compileflags, 1) in test.globs
>>      File "<doctest memmapped-fileformats.rst[89]>", line 1, in <module>
>>    NameError: name 'loaded_ui_yaml' is not defined
>>
>> I also tried running it with wander instead of vikings, and python crashed.
>
> Yes, all the input filenames are currently hardcoded 'viking.foo'
> except for the un-prefixed ones.
> I figured this didn't matter because, as I mentioned in the doc, I
> plan to add a nicer way of accessing things
> that should hide those details.
>
> David

RE this comment:
 Skip the BLOAD header (I later found out that memmap has an 'offset'
parameter, but
 it's granularity is limited to the structure size)

I had a look at numpy/core/memmap.py, and it seems that the
explanation of the offset parameter given (" 'offset` should be a
multiple of the byte-size of `dtype` ") is wrong. It's not the offset
amount that needs to be a multiple of the size of the data type, it's
the length of the remainder of the file. I removed the BLOAD header
from gen_dtype and specified a 7 byte offset, and sure enough it
worked.

I'm still reading through the rest of this...

TMC