Well, the easiest way is just to include the directory in the file header for a given file. For all the files in the sound directory, their file name stored in the package would read "sounds/????????.???"

Now, if you want to get tricky, you use recursive package files. As in, the sounds directory is really a package stored in the master package file. This speeds stuff up tremendously when accessing files, period.

My package format supports/uses both.

- Splat

Neat idea, Splat. So you header conatins an entry, say, sounds that actually points to a nested header with info about all of the sounds info?

I never thought of that. hmmm.... Gonna give it a try. Thanks.

By the way, anybody here used zLib? Anygood? I'm not sure if I should use it, as it seems to be a little overkill.

Yeah, pretty much. The "sounds" entry could actually be extracted from the main package and would yield a completely valid archive on its own. Quite cool. The packing utility took a little tinkering before this feature worked as intended, but its cool now. The packer is almost ALWAYS more complex than the depacker (in-game).

- Splat

[This message has been edited by Splat (edited November 17, 1999).]

I havn't had problems makeing packers,my problem is reading the data back in then making good use of it

I agree with Zenroth. I've found it relatively easy to pack the data. It's extracting it properly that seems to be more complicated.

Are you using recursive packages? With me, the loading process is VERY straightforward. First, the directory of the main package is loaded, which contains file headers containing the filename, size, offset into the current package, and compression format.

Then, for each subpackage, the process is repeated, creating a tree of file headers.

Whenever my game needs a file, it passes the complete filename it wants, ie "sounds/death.wav". First, it checks to see if a file header with that exact name is present. If not, it "peels" off the directory and looks for a file header of a subpackage with that name. Recurse the same function into the subpackage. Repeat until file is found or conclusively not there.

Once the file is found, we have (automagically using the loader function) an absolute offset from the beginning of the WHOLE package. Its just a matter of reading it in.

The packer, however, is a real recursive nightmare. Reason being, each file header is NOT the same size. The file name has a variable size and is stored by writing a byte for how long it is then the actual contents. So to pack the archive, you need to reserve the file header space at the beginning, then compress the files, then come back and record the file header's with offset, size, and compression type.

Its much more important to have a simple loader than a simple packer. You have unlimited time to pack the data, but a game user does NOT want to wait a minute while your game loads its data. Redesigning a file format for easy loading is critical.

- Splat

[This message has been edited by Splat (edited November 17, 1999).]

That is very interesting. Thanks for the clarification. The only question I have left is, why do you use a variable size header, especially if you're concerned about speed? Wouldn't it be faster to use a static size?

Well, here's the thing: I am concerned about speed, but only on the loading end. While it takes a little longer to pack, because you have to traverse the file list twice in effect. (note: I may be fixing this by placing the file header at the end of a package rather than the beginning: this also speeds up additions and removals to the package without a need to completely repack it)

However, reading a byte then a variable number has neglible effect on performance, and that is all the loader does. The trick is, loading of the package directory is done at registration (in my game engine you register a package file to get an integer package id). So that is also just a one shot thing. Plus, it keeps support for long filenames while not having to reserve 255 bytes per entry. You got several thousand files, that 255 bytes per will start to add up and make the file header so very large.

I guess in the end its just a personal choice. 400k here or there isn't that big a deal.

- Splat

Ok. SO one last question.

How do you deal with compression? DO you compress the packed file, or each file before packing it? I see advantages both ways:

one big archive - Less load time, decompress at startup. After that, just extract data. More memory is required to hold the decompressed file.

bunch of archives - less memory. Only unpacked as they're needed. Slow because they're only unpacked as they're needed.

So, what do you think? Compress before or after packing, or just skip packing altogether. Although, if I'm gonna skip it, I might as well just compress it before and after startup.

Did any of you guys consider not re-inventing the wheel and use ZIP? Just wondering if there's something important I missed

/Niels