File Management Using Resource Files

File Management Using
Resource Files
Bruno Sousa, Fireworks Interactive
gems2@fireworks-interactive.com


As games increase in size (I think the grand prize goes to Phantasmagoria with
seven CDs), there is a need for organization of the game data. Having 10 files in
the same directory as the executable is acceptable, but having 10,000 is not. Moreover,
there is the directory structure, sometimes going five or more levels deep, which
is a pain to work with. Because our games will hardly resemble Windows Explorer, we
need to find a clean, fast way to store and organize our data. This is where resource
files come into play. Resource files give us the power to encapsulate files and directories into a single file, with a useful organization. They can also take advantage of compression, encryption, and any other features we might need.

What Is a Resource File?
We already use resource files all the time in our daily work—examples of these are
WinZip, the Windows installer, and backup programs. A resource file is nothing
more than a representation of data, usually from multiple files, but stored in just one
file (see Listing 1.15.1). Using directories, we can make a resource file work just like a
hard drive's file system does.

Listing 1.15.1 Resource file structure.

Signature = "SEALRFGNU" + '\0'

Version = 1.0

Number of Files = 58

Offset of First File = 19

[File 1]

[File 2]

[File 3]

[File .]

[File .]

[File .]

[File Number Of Files - 1]

[File Number Of Files]

Each lump (we will start calling files "lumps" from now on) in the resource file
has its own structure, followed by all of the data (see Listing 1.15.2).


Listing 1.15.2 File lump structure.

File Size = 14,340

Filename = "/bmp/Bob.bmp" + '\0'

Flags = COMPRESSED

Flagslnfo = OxF34A400B

[Byte 1]

[Byte 2]

[Byte 3]

[Byte .]

[Byte .]

[Byte .]

[Byte File Size - 1]

[Byte File Size]

Design

Before we do anything else, we'll need to name our resource system. We can then use
the name to give each component a special naming scheme, one that will differentiate it from the other parts of the game. Let's call this system the "Seal Resource File System," abbreviated to SRFS, and use "si" for class prefixes.

First, we need a resource file header. By looking at Listing 1.15.1, it's easy to see
that we are keeping our system simple. However, that doesn't mean it isn't powerful, it means that it was designed to accommodate the most-needed features and still retain a fairly understandable syntax and structure.
Our resource file header gives us all the relevant information about the system.
Multiple file types are used in games, and for each type, there is usually a file header
that contains something unique to differentiate it from other file types. SRFS is no
different, so the first data in its header is the file signature. This is usually a 5- to 10-
character string, and is required so that we can identify the file as a valid Seal resource file. The version information is pretty straightforward—it is used to keep track of the file's version, which is required for a very simple reason: if we decide to upgrade our system by adding new features or sorting the lumps differently, we need a way to verify if the file being used supports these new features, and if so, use the latest code. Otherwise, we should go back to the older code—backward compatibility across versions is an important design issue and should not be forgotten. The next field in the header is for special flags. For our first version of the file system, we won't use this, so it must always be NULL (0). Possible uses for this flag are described in the For the Future section. Following this is the number of lumps contained in the resource file, and the offset to the first lump. This offset is required to get back to the beginning of the resource file if we happen to get lost, and can also be used to support future versions of this system. Extra information could be added after this header for later versions, and the offset will point to the first lump.

We now move to our lump header, which holds the information we need to start
retrieving our data. We start with the lump size in bytes, followed by name and directory, stored as a fixed-length, NULL-terminated string. Following this is the flags
member, which specifies the type of algorithm(s) used on the lump, such as encryption or compression. After that is information about the algorithm, which can contain a checksum for encryption or dictionary information for compression (the exact details depend on the algorithms). Finally, after all of this comes the lump information stored in a binary form.
Our system has only two modules: a resource file module and a lump module. To
be able to use a lump, we need to load it from the resource file and possibly decrypt or decompress it into a block of memory, which can be accessed normally. Some systems prefer to encapsulate all functionality into the resource file module, and even allow direct access to lump data from within this module. This approach certainly has advantages, but the biggest disadvantage is probably that we need to have the wholeresource in memory at once, unless we use only raw data or complicated algorithms to dynamically uncompress or decrypt our lump data to memory. This is a difficultprocess and beyond the scope of this gem.We need functions to open the resource file, read the header, open individual
lumps, read information from lumps, and get data from lumps. These are covered in
the Implementation section.

Implementation

The sample code included in the CD is written in C++, but for the text, we will use
pseudocode so it will be easy to implement in any language.
The sICLump Module
Our lump module is similar to file streams in C++ or other language implementations
of files in that we can write to it. Unfortunately, updating the resource file with a
lump is very troublesome due to the nature of C++ streams. We can't add data to the
middle of the stream—we can only replace it—and we can't modify the parent
resource file.

DWORD dwLumpSize;

STRING szLumpName;

DWORD dwLumpPosition;

BYTE [dwLumpSize] abData;

The variable dwLumpSize is a double word (32 bits) that specifies the size of the
lump, szLumpName is a string describing die lump's name, dwLumpPosition keeps the
lump's pointer position, and abData is an array of bytes with the lump information.
Here are the sICLump module functions:

DWORD GetLumpSize (void);

STRING GetLumpName (void);

DWORD Read (BYTE [dwReadSize] abBuffer, DWORD dwReadSize);

DWORD Write (BYTE [dwReadSize] abBuffer, DWORD dwWriteSize);

DWORD Seek (DWORD dwSeekPosition, DWORD dwSeekType);

BOOLEAN IsValid (void);

GetLumpSizeO retrieves the lump's size, and GetLumpName() retrieves the lump's
name. Read() reads dwReadSize bytes into sbBuffer, and Write () does the exact
opposite, writing dwWriteSize bytes to sbBuffer. Seek() moves the lump's pointer by
a given number from a seek position, and I sValid () verifies if the lump is valid.

The sICResourceFile Module

This module has all the functionality needed to load any lump inside the resource.
The module members are nearly the same as those in the resource file header.

DWORD dwVersion;

DWORD dwFlags;

DWORD dwNumberOfLumps;

DWORD dwOffset;

STRING szCurrentDirectory;

FILE fFile;

The use of these members has already been described, so here is a brief definition
of each. dwVersion is a double word that specifies the file version, dwFlags is a double
word containing any special flags for the lump, dwNumberOfLumps is the number of
lumps in the resource, dwOffiet gives us the position in bytes where the first lump is
located, szCurrentDirectory is the directory we are in, and fFile is the actual C++
stream.
Now for the real meat of our system, the sICResourceFile functions—those that
we use to access each lump individually.
void OpenLump (STRING szLumpName, slCLump inOutLump);
void IsLumpValid (STRING szLumpName);
void SetCurrentDirectory (STRING szDirectory);
STRING GetCurrentDirectory (void);
Each of these functions is very simple. IsLumpValid () checks to see if a file with a
given szLumpName exists in the resource. SetCurrentDirectory () sets the resource file
directory to szDirectory. This directory name is prepended to each lump's name
when accessing individual lumps within the resource file. GetCurrentDirectory()
returns the current directory.
Now for our Open function. This function opens a lump within the resource file,
and the logic behind the algorithm is described in pseudocode.
Check flags of Lump
if Compressed
OpenLumpCompressed (szLumpName, inOutLump)
if Encrypted

OpenLumpEncrypted (szLumpName, inOutLump)
if Compressed and Encrypted
OpenLumpCompressedEncrypted (szLumpName, inOutLump)
else
OpenLumpRaw (szLumpName, inOutLump)
end if
Depending on the lump type, the appropriate function to open the lump is
called, thus maintaining a nice design and simple code. The source of each function is
included in the CD.

Last Words about the Implementation
Some support functions that are used to open the file or to keep track of information
that can't be called directly are not represented in the preceding text. It is advisable to
check the source code on the CD, which is well commented and easy to follow The
ON me ca algorithms for compression and encryption are simple RLE compression and bit-wise
encryption, the actual implementations of which are beyond the scope of this gem
and must be researched separately. Information about useful public domain algorithms
is at [WotsitOO], [WheelerOO], and [Gillies98].

Conclusion

This system can be easily upgraded or adapted to any project. Some possibilities
include supporting date and time validation, copy protection algorithms, checksums,
a data pool, and better compression and encryption algorithms. There is no limit.
References
[Hargrove98] Hargrove, Chris, "Code on the Cob 6," available online at
www.loonygames.com/content/1.11/cote/, November 2-6, 1998.
[TownerOO] Towner, Jesse, "Resource Files Explained," available online at
www.gamedev.net/reference/programming/features/resfiles/, January 11, 2000.
[WheelerOO] Wheeler, David J, et al, "TEA, The Tiny Encryption Algorithm," available
online at www.cl.cam.ac.uk/ftp/users/djw3/tea.ps.
[WotsitOO] Wotsit.org, "The Programmer's File Format Collection: Archive Files,"
available online atwww.wotsit.org, 1996—2000.
[Gillies98] Gillies, David A. G., "The Tiny Encryption Algorithm," available online
at http://vader.brad.ac.uk/tea/tea.shtml, 1995-1998.