Niels: its all about the learning experience. i know when i write a program, i rewrite all the code, even if i know where i can get what i'm looking for. Isn't that why we're (us pre-professionals) here, making our own little games, so we can learn as much as we can before we go off to college or work? I think its good to reinvent the wheel. it expands the mind and really gives you a true sense of how and why a program works, rather than just knowing it does and trusting that.

---Gecko

Ok, say I don't want to reinvent the wheel. Which libraries do I use? Can someone please point me to the resources? Thanx.

If you want compression, you should definately compress each file on an individual basis, or else you'll have to de-compress the whole archive just to get at one de-compressed file, and that would suck wouldn't it?

Splat, I'm not familiar with the details of your system, but I would say that fixed size file headers (or sub-archive headers, whichever you prefer) would definitely NOT add significant size to the file on your hard drive. For reference, I use a fixed 128 byte header for each directory/file, so if I had 2000 files in the archive, i would need 256K for the headers. Ok, this might seem like a lot, but how big would 2000 files be? Easily half a gigabyte, so 256000 / 500000000 = 0.000512 = .0512% or one twentieth of a percent of the total archive size. Don't worry about fixed size headers...

Also, you usually don't need to modify an archive at run-time (during a game) so I wouldn't bother using an on-the-fly system where you tack on file/headers at the end of the archive for "easy modifiability". To me it isn't worth it. I put all directory/file info at the start of the archive, and the file data after. Works better for me.

Out of curiosity, what kind of speed advantage can be gotten by using a packed file format? Currently I am storing several thousand files as bitmaps each in their own separate file. Totals up to about 150 megs. What kind of speed increases can one get by using a packed file system over just relying on the user's filesystem?

Gees, a lot of messages!

Zer: first off, that 150 will drop a lot because of the excess size due to cluster size. A hard drive is formatted with a cluster size that defines the increment for files. So if you are using a 2k cluster size, a file will be rounded up to a 2K multiple. Packing reduces that a lot.

Niels: Reasons for writing my own:
1) Learning, as was said above.
2) Portability, althought I do concede that libraries like zLib are much more portable than mine and support compression.
3) Test bed for my own compression algorythms, which may or may not prove quite efficient.
4) Perfectionism - I like it my way! My package format is simple, straight-forward, small, fast, and expandable. I'm not really tied into ANYTHING with this.

foofightr: Your files are much bigger than mine. And you have less. I like the idea of variable size names. I have support for unlimited size names, unlike your *hard coded* (hate those words) 128 character limit. I'm trying to cut what I see as the fat on typical package formats. Also, if you really want to get technical, variable sized strings are faster because a) you load less data from disk, b) you already know the length, which saves you time from searching for the first null char.

As for the moving the file headers to the end, no, it's not for on-the-fly updating in game. Its for dynamically recompressing a 150MB directory when only 200k changed, including added files. The file header at the end means you can move the file header down and add the new files, and add their entries to the file header, and recursively update the affected offsets and sizes in the rest of the file. Saves a tremendous amount of time.

And lastly, it's my game (er, game engine) damnit! hehe

_kill_: http://www.wotsit.org has lots of stuff on ALL file formats, including most common compression types.

*gasps for breath*

- Splat

[This message has been edited by Splat (edited November 18, 1999).]

Oh shoot, forgot Stark:

You definately compress individual files in the package for several reasons. Mainly speed, memory restrictions, and the fact that different types of data are more easily compressed by different types of compression. Libraries like zLib even split files into "blocks" that can each be compressed differently.

- Splat

the datafile format that i use happens to use zlib for compression, and i must say it's very easy. i compress the files individually, and stick them into the datafile. zlib address: http://www.cdrom.com/pub/infozip/zlib. the zlib tool article is also very useful: http://www.dogma.net/markn/articles/zlibtool/zlibtool.htm

------------------
http://members.xoom.com/mutex0

Splat, I'm looking at it from a practical standpoint. I believe my approach is more practical, it suits my needs just fine, end of story.

I agree to disagree.

Put a fork in it, this thread is done!

- Splat

I'm trying to implement a packed file format for my game engine, and have been looking for resources on how to go about it. So far, I'm able to pack multiple files into one, but I can't quite figure out how to use subdirectories within the files to make a virtual file system. I think this could be useful for organising sounds, graphics, etc...

That way the engine would make a request for a sound, and the file manager could look it up in the packed file's sound dir.

Good idea? Hmmmm.... Anyways, I'd appreciate any resources that deal with this kind of thing. Thanks.