[size="5"]How This Document Is Organized

1. Introduction
2. Resource Files In Depth
3. Design and Theory
4. Implementation
5. For the Future

[size="5"]Introduction

The time has come for you to venture into a new world of the game development universe - the elegant world of resource files. Perhaps you may be wondering what a resource file is, or you may have a vague idea of what they are and you're interested in using them. Perhaps you may even be fluent in the use of resource files. Regardless of which position you're in, we'll now go over what resource files are, and what they aren't. Let's begin by taking a non-technical approach to visualizing what they are; this way, we can have a much better overall understanding of the concept behind them. Suppose you have a large leather magic bag, which has the property of being essentially bottomless. You can take various items and put them in the bag, and you can these same items out. Although you don't necessarily have the option of taking a good gander at the contents of the bag, there is no problem in finding a particular item in order to take it out. This bag is also virtually weightless - you can easily hold it, carry it, and manage it. Now lets take this strange model of thought and apply it to the software-programming field, in our case, video game development. The magical leather bag becomes the resource file, and the items that we could put into the bag become the various pieces of data of which are used by the game - bitmaps, sounds, game-logic scripts, and so on. Additionally, the idea of being able to easily manage the bag still holds true, only in this case, we are managing the data in the resource file.

So basically, a resource file acts as an abstraction layer for managing multiple items of data; however, this is only one of many advantages that surface once you partake in the use of such files. Before we discuss the other advantages to using them, as well as going over a couple of the disadvantages, we will have a look at where resource files are already in use today.


[size="5"]Resource Files In Depth

Almost everywhere you look you will see some form of a resource file in use - perhaps even unknowingly. You may be wondering if resource files are in fact database files, and indeed, they are. Nevertheless, we still use the term "resource file" to discern between the two: a database usually keeps track of similar items of data, whereas a resource file has the ability of storing dissimilar items of data. Keeping this concept in mind, we can see that resource files are used by a great number of computer games as well as applications. Let's look at a couple specific examples of resource files in use. The "Zip File" archiving file format, in wide use today, is a great illustration of what a resource file is supposed to be. It can take multiple individual files and store them internally in a single file. Additionally, you have the option of compressing the files you add to a Zip file for optimum storage and transference. This is a good example because it shows you that you can take the data you want to add to a resource file and mutate it for the better without destroying it. Another example we'll look at is the WAD file format used by id Software in the Doom engine. The file format allowed the various items of game data to be stored in a single file - this included images, sprites, sounds, music, map data, and the likes. This will be the type of resource file format that we'll be most interested in, although we will implement our own original file format. Nonetheless, this will be that path that we take.

So, why do we even want to go through all the trouble of implementing resource files in our games or game engines? Why not just have all of the individual files located in a number of sub-directories within the directory of our game? Here, I will explain the reasons behind wanting to use resource files to store data for games. Obviously, being able to easily manage multiple items of data was an advantage we already discussed. Without resource files, and in a game where there is a lot of external data (images, sounds, etc.), you would have to store the data as individual files in a complex directory structure. This can amount to a huge mess that is almost impossible to manage and to keep up to date. With resource files, all of your data is located in one or a couple of large resource files - all of the data is there, elegantly encapsulated within a single construct. Additionally, one cannot go on without mentioning the professional "look" that resource files provide - this is a big plus. But, is there another advantage to having all of the files located within a resource file? Indeed, there is and it happens to be something very important - security. Without using resource files, most of the game data would be located out in the open, and most likely in it's native format. This would allow any average Joe, who happened to have your product installed, to modify, replace, or pirate the game data. In many cases, this is rather undesirable. When the data is in a resource file, the end user of your software has greater difficulty getting access to the individual items of data; of course, anyone can get access to data in a resource file with a fair amount of hackin' effort. However, having your data located within a resource file acts as a slight deterrent against this sort of activity. Also, when using resource files you can add additional security support. For instance, you may want to add a form of checksum error checking where you would make sure the size of each lump or the data itself hasn't changed; this would be protection against illegal modification of the data by adding, removing, and modifying bytes. Even though you may decide to restrict the access of the data in your resource files, many people ship game editors with their products anyway. Regardless, you can still restrict unofficial modification of the file.

The question we must ask our selves now is "should we go through all of the trouble of implementing a resource file format?" Sure, you can also tackle security issues by creating your own file formats for the individual types of data like images, sounds, and on and on - plus, you can get around the complex directory structure problem. So, is there another reason for using resource files? The answer is undeniably yes. Due to the inefficiency of the file systems used by Windows and DOS (FAT16 and FAT32), a lot of disk space is lost. In the FAT16 file system, each file block takes up a minimum of 32KB; therefore, if you had a file that was 2KB in size, it in reality would take up 32KB. Of course, the FAT32 file system is much better, but it still isn't perfect. Anyway, when using resource files, all of the data is packed together, allowing for more efficient disk space usage.

Now, if those aren't good enough reasons for using resource files for storing data for your games, I don't know what is! Of course, there are a couple of disadvantages to using resource files. The first such disadvantage is that of requiring more time and effort to implement a good resource file manager. But once you have your implementation up and running, it's smooth riding from there on. The second disadvantage is that of requiring more memory to operate - you must use up a fair amount of RAM when working with resource files. Nonetheless, this is a trivial problem, especially considering that most PC's now ship with at least 64 megabytes of RAM. It is quite lucidly apparent that the advantages most certainly out-weigh the disadvantages - so what are we waiting for! Let's begin our adventure into the technical side of resource files.


[size="5"]Design and Theory

I'm sure you're quite worked up about jumping in and implementing a resource file management library, and heck, so am I! But first, we must go over some design issues. I will use the term "lump" to refer to a data item within a resource file; thus, a lump can be an image, or a game object, or any piece of data we may stick in a resource file. Also, when I talk of an interfacing API, I mean a front-end API that can be used to manage resource files. Now, in general, the structure of a resource file can be broken up into three sections: the header, the data lump information table, and the actual data lump section. The header section of a resource file contains general information about the file; for instance, the number of data lumps within the resource file, the location of the Lump Information Table within the file, and a means for identifying resource files from other file types. The Header section is critical because it provides the interfacing API with the information required in order loading the file in from disk. The Lump Information Table section of the resource file contains information about each data lump within the file. The information stored about each lump might entail it's position and size within the file, as well as it's name or identification medium. For each Data Lump in the resource file, there is one corresponding entry or node in the Lump Info Table; thus, the Info Table is a list of nodes where each node corresponds to and contains information about a particular Data Lump. The Data Lump Section is the section in the resource file where the data for each Data Lump is actually stored. The lumps are generally stored in a linear fashion such that the data for "Lump One" is to be located first, the data for "Lump Two" is to be found next, and so on. These data lumps may be compressed or encrypted depending upon the file format implementation desired by the programmer. However, I should mention the position of each of these sections within the file. For a few reasons that will later become apparent, we will position the Data Lump Info Table at the end of the file, having the actual data lump section in the middle. We do this because it is much more efficient to do so, if we didn't do this, we'd end up doing multiple passes during the save process. Anyway, as we now have the general structure of how we're going to set up our files, we'll discuss our interfacing API that we must design.

So, what do we need in a resource file management library? Well, the interface must be elegant and intuitive, whilst at the same time providing enough flexibility and scalability to deem it as reusable. Well, let's first discuss the user-friendly interface, then we'll talk about scalability. So, what sort of functionality are we going to need? We should provide routines for quick creation, modification, and clean up; hence, we'll need three routines for opening, saving, and closing resource files. We must also provide some routines for managing lumps; this will consist of routines for creating, deleting, modifying, loading, and unloading data lumps. Those are the basic routines we'll need; although, some extra routines for error checking and for displaying information about a resource file wouldn't hurt - but what about flexibility and scalability?

Indeed, this is something that we must take into consideration if we're going to use our resource file management library more than once. Basically, we should provide some means of being able to handle different data lump types within our manager. But how are we going to accomplish this? It's all quite simple really - we'll create a lump handler system. A lump handler will consist of three low-level routines, all for managing a particular data lump type. These routines would allow the interfacing API to load, to save, and to unload a particular data type. So, what we would do is create a lump handler for managing a bitmap lump type, a second handler for managing sound lumps, and so on. The interfacing API would then look at a lump, figure out which lump handler to use, and call the correct routines. In order to provide scalability, we should allow a programmer to create his or her very own custom lump handlers and allow the lump handler to be added to the lump-handling list on the fly. So, we'll need two more routines within our interfacing API: one to add a lump handler, another to remove a handler. Because we'll using OOP and C++ to implement our resource file management library, a lump handler will be an object that will be used by the resource file management class. Anyway, now that we have an outline of what we're going to need to do in order to get a good implementation happening, let's actually implement this baby!


[size="5"]Implementation

Let us now begin our design of the structures that will make up our resource file format. We'll start with the structure of the resource file's header.

As you can see, the header is indeed a critical part of a resource file - let's go over the various fields in this structure. The first field, "cSignature", is merely a file identification medium of which we use to identify resource files from other file types: it contains the non null-terminated string "RESF". The next two fields are the resource file's version numbers; we can use these values for keeping track of future revisions of the file format. The "wFlags" field is used to hold any special settings the resource file may have. The "dwNumLumps" holds the value indicating the number of data lumps that are currently within the file. Next, "dwChecksum" works as a simple and quick error checking method, which is used to make sure the size of the resource file was not illegally modified. Finally, "dwLumpListOffset" is the offset into the resource file where the data lump information table is stored.

Although the Lump Info Table is stored at the end of the file, we'll go over the structure that is used to create the table.

Basically, the Lump Info Table is an array of RESFILELUMPINFO structures. The "dwLumpListOffset" field of the RESFILEHEADER structure simply points to the first entry in this list. When a resource file is opened and loaded in, this entire table is loaded into memory as a linked list. This allows us to easily add new lumps to the resource file. However, when we load in resource file, we load the list into memory using another structure...

Basically, this is the same as RESFILELUMPINFO, although this structure has a few more fields for managing lumps in memory. The "lpName" is the equivalent of a file name, only in this case, it happens to be a lump name. I should make a point of how the "lpData" field can be used. It doesn't necessarily have to point to a raw data buffer, it can also point to a structure. Then, it's merely a process of casting to the structure pointer type. This is useful when writing lump handlers that directly load lumps into structures; for example, loading a bitmap lump into a DirectDraw surface. Notice that the "dwType" field is evident in both of the lump structures. This is used as for identification when matching a particular lump with the corresponding lump handler. The management class just searches through the lump handler list, until it finds a match. Let's have a look at the structure used for keeping track of a lump handler.

This structure also keeps a "dwType" field handy so that we can match it to a lump. Note the function pointers - they point to the functions that load, save, and unload a lump of that type. The loading and saving functions take a file stream pointer, along with a pointer to the lump's info. Because the management class takes care of positioning the file pointer to the beginning of the lump, writing these routines is rather trivial. In my implementation, I defined a "Raw Data" lump type that becomes the default lump type when one can't be determined. Here are the lump handling functions for the "Raw Data" lump type.

It's now time to take all of the structures and put them to good use; it's time to have a look at the resource file management class which happens to be our interfacing API.

To open and load in a resource file's Lump Info Table, we first read in the header, seek to the position in the file where the Info Table begins. At this point, we load each entry in the table into a linked list. This is all done by Open() method of the CResFile class; additionally, this method sets up the header and the Lump Info Table if a new resource file is opened. Now, when someone wants to load in a data lump, the name of the lump would be passed to the LoadLump() method; this method searches through the Lump Info list until it finds a matching node. Once a match is found, it seeks to the position in the resource file where that particular data lump begins and gets ready to load it in. As was mentioned earlier, the lump handler list is searched at this point for a lump handler that matches the lump type. Once the lump has been loaded in, the LoadLump() method returns a pointer to the data or the lump's structure, depending upon the type of lump. Note that a lump is only loaded into memory once; moreover, if LoadLump() is messaged more than once using the same lump name, it will return the same pointer. If you want individual copies of a lump in memory, you must perform the "cloning" process on your own. Of course, you could always add functionality to the resource file manager to do this.

The process of saving a resource file is basically the same - we write out the header, then the data lumps, and finally the lump info table. When we first write out the header, we don't know of the position that the lump info table within the file; thus, we will have to seek to the beginning of the file and update the header section once all of the data has been saved. As the actual data lumps are being saved, we update the internal lump info linked list with the positions of the data lumps. Finally, we write out the Lump Info Table and return to the beginning of the file to update the header. This job is done by the Save() method of the CResFile class; of course, the file had to be opened using a write or modification access mode. Again, the lump handler list is queried for the correct lump handler when saving the lumps to the file.

Creating a new lump is simple; all we have to do is add a new node to the Lump Info Linked List and set a couple of flags in the node's structure indicating that we just created it. Deleting a lump is merely a process of removing it from the list. These functions are preformed by the CreateLump() and DeleteLump() methods of the CResFile class. If the lump type is not recognized, then the lump handler chosen will be the raw data lump handler.

Adding a custom lump handler is simpler than ever. Simply pass pointers of your lump handling functions to the RegisterLumpHandler() method along with the lump type for which the lump handler will process. You can also remove a lump handler using the RemoveLumpHandler() method.

I recommend that you have a good look at the source code at this point as it is well commented and it should give a good understanding of logic behind a resource file management library.


[size="5"]For the Future

Once you have your resource file management library up and running, you'll most likely want to make an editor so that you can easily create and manage various resource files. There are a couple of approaches you can for doing this. Probably the easiest way of doing this is creating a console application that accepts parameters on the command-line. You may also want to make an editor with a GUI that is similar to Windows Explorer. You could also incorporate a resource file editor into your game editor so that you have one application that acts as your tool set when creating the various resources. You will also probably want to create your own custom lump handling routines.

Still, is there anything else we can do to improve our resource file implementation? Of course there is, as the possibilities are limitless; however, I'll go over a few of them here. One thing you can do to organize the lumps in your resource files is by building an internal directory structure. This directory structure would emulate the one found in DOS, for instance. You could create numerous directories within the resource file and organize the data lumps into these directories. This leads to more efficient maintenance of resource files.

Another thing that would be interesting to do would be to create a data & file I/O streaming system that would be used by your resource file management library. You would create a base streaming class, and derive other streaming classes off of that. That way, you could have a compression streaming class, an encryption streaming class, an imbedded data streaming class, and so on. Then, you could add some settings to your resource file format that would allow you to compress and encrypt data as it's output to the file.

If you want to allow your resource file management library to be used to across a wide range of platforms you will have some extra work cut out for you. Of most concern would be the endian byte order used by the host platforms. You will have to take this into account by shifting the bytes around so that they will be compatible with the machine; this process would take place during the loading and saving of a resource file.

Another concept, which is interesting, is that of data caching. Since accessing a hard drive is much faster than reading off of a CD or DVD drive, a temporary cache file could be created on the hard drive. Then, the most frequently accessed data would be placed in the cache file for quick access. This would greatly speed up the loading sequences in your game once the cache file is created and operational.

Of course, there are many other things you could do with a resource file management library - it all depends on what you want to use the management library for. Anyway, I hope you've enjoyed this article, good luck with your coding endeavors, and code-on!