Preamble

This article has been written based on my personal experience, if you think you are offended in some ways because of my personal opinions about programming and / or my coding style, please stop reading now. I am not a guru programmer and I don't want to impose anything on anyone.

Introduction

During the development of my engine(s), I have always had the need for better handling of game assets, even if in the past it was sufficient to hard code the needed resources at startup. Now with the added speed of modern computers it is possible to create something more advanced, dynamic and general-purpose. The most important points I needed for a resource manager were:

• reference counted resources.
• If a resource is already present in the database, expose it as raw pointer, else load it.
• know automatically when to free the resource.
• fast access using string to retrieve resources
• clean everything on exit without manually deleting the resources.
• mapping resource using a referenced structure.

At this point it looks like I am going to reinvent the wheel, given the assumption that smart pointers are now included with the new C++ standard. My personal problem with the smart pointers is that it was very difficult to create the kind of data structure I wanted. My idea was to create an unordered map of shared pointers and then hand out a weak pointer when the resource was asked for. The problem with this data organization is that the weak pointer basically 'observes' the resource but doesn't hold it. On the contrary using an unordered map of weak pointers and handing out a shared pointer created different problems, which could be resolved using a custom deleter and other indirect strategies. One of this was that if I loaded the resource for the first time, its count was set to 1 and when the same resource was asked for its count was increased to 2. So I still had the problem to ignore the first reference, which was again solvable using a custom deleter and tracking how many resources where effectively still left. Further more, shared pointers are thread safe and according to the standard they are synced even if there is no strict necessity. I am not saying that smart pointer are useless, they are used quite extensively, but my approach needed something different; I needed to have total control of reference counting. Basically what I needed was a wrapper class to store the pointer to the loaded resource, and its reference counting. Note that the code I copied from my engine uses some other utility functions - they are not necessary, because they handle error messaging and string 'normalisation' (eliminating white spacing and lowering the string down), so you can easily ignore those functions and substitute them with yours. I have also removed all debugging printing during the execution, to keep things clearer. Let's have a look at the base class for resources.

class CResourceBase 
{ 
    template < class T > friend class CResourceManager; 

private: 
    int References; 
    void IncReferences() { References++; } 
    void DecReferences() { References--; } 
    
protected: 
    // copy constructor and = operator are kept private 
    CResourceBase(const CResourceBase& object) { } 
    CResourceBase& operator=(const CResourceBase& object) { return *this; } 
    
    // resource filename 
    std::string ResourceFileName; 
    
public: 
    const std::string &GetResourceFileName() const { return ResourceFileName; } 
    const int GetReferencesCount() const { return References; } 
    
    //////////////////////////////////////////////// 
    // ctor / dtor 
    CResourceBase( const std::string& resourcefilename ,void *args ) 
    { 
        // exit with an error if filename is empty 
        if ( resourcefilename.empty() ) CMessage::Error("Empty filename not allowed"); 
        
        // init data members 
        References = 0; 
        ResourceFileName=CStringFormatter::TrimAndLower( resourcefilename ); 
    } 
    
    virtual ~CResourceBase() { } 
}; 

The class is self-explanatory - the interesting point here is the constructor, which needs a resource filename, including the full path of your resource and a void pointer to an argument class in case you wanted to load the resource with some initial default parameters. The args pointer comes in handy when you want to instantiate assets during runtime and don't want to load them. There are some cases where this is useful, for every other case the constructor will serve our purposes well. All of our assets will inherit from this class. There is, obviously, the reference counter and some functions for accessing it.

The Resource Manager

The resource manager basically is a wrapper for an unordered map. It uses the string as a key and maps it to a raw pointer. I have decided to use an unordered map because I don't need a sorted arrangement of assets, I really do care at retrieving them in the fastest possible way, in fact accessing a map is O(log N), while for an unordered map is O(N). In addition just because the unordered map is constant speed (O(1)) doesn't mean that it's faster than a map (of order log(N)). Anyway, in my test cases the N value wasn't so huge, so the unordered map has always been faster than map, thus I decided to use this particular data structure as the base of my resource manager. The most important functions in the resource map are Load and Unload. The Load function tries to retrieve the assets from the unordered map. If the asset is present, its reference count is increased and the wrapped pointer is returned. If it's not found in the database, the function creates a new asset class, increases its reference count, stores it in the map and returns its wrapped pointer. Note that its the programmer's responsibility to create an asset class with a proper constructor. The base class, inherited from CResourceBase class, must provide a string containing the full path from where the asset needs to be loaded and an argument class if any - this will be clearer when the example is provided. The Unload function does exactly the opposite: looks for the requested asset given its file name, if the resource is found, its reference counter is decreased, and if it reaches zero the associated memory is released. Since I think that a good programmer understands better 1000 lines of code rather than 1000 lines of words, here you have the entire resource manager:

template < class T > class CResourceManager 
{ 
private: 
    // data members 
    std::unordered_map< std::string, T* > Map; 
    std::string Name; 
    
    // copy constructor and = operator are kept private 
    CResourceManager(const CResourceManager&) { }; 
    CResourceManager &operator = (const CResourceManager& ) { return *this; } 
    
    // force removal for each node 
    void ReleaseAll() 
    { 
        std::unordered_map< std::string, T* >::iterator it=Map.begin(); 
        
        while ( it!=Map.end() ) { delete (*it).second; it=Map.erase( it ); } 
    } 
    
public: 
    /////////////////////////////////////////////////// 
    // add an asset to the database 
    T *Load( const std::string &filename, void *args ) 
    { 
        // check if filename is not empty 
        if ( filename.empty() ) 
            CMessage::Error("filename cannot be null"); 
            
        // normalize it 
        std::string FileName=CStringFormatter::TrimAndLower( filename ); 
        
        // looks in the map to see if the 
        // resource is already loaded 
        std::unordered_map< std::string, T* >::iterator it = Map.find( FileName ); 
        
        if (it != Map.end()) 
        { 
            (*it).second->IncReferences(); 
            return (*it).second; 
        } 
        
        // if we get here the resource must be loaded 
        // allocate new resource using the raii paradigm 
        // you must supply the class with a proper constructor 
        // see header for details 
        T *resource= new T( FileName, args ); 
        
        // increase references , this sets the references count to 1 
        resource->IncReferences(); 
        
        // insert into the map 
        Map.insert( std::pair< std::string, T* > ( FileName, resource ) );
        
        return resource; 
    } 
    
    /////////////////////////////////////////////////////////// 
    // deleting an item 
    bool Unload ( const std::string &filename ) 
    { 
        // check if filename is not empty 
        if ( filename.empty() ) 
            CMessage::Error("filename cannot be null"); 
            
            // normalize it 
            std::string FileName=CStringFormatter::TrimAndLower( filename ); 
            
            // find the item to delete 
            std::unordered_map< std::string, T* >::iterator it = Map.find( FileName ); 
            
            if (it != Map.end()) 
            { 
                // decrease references 
                (*it).second->DecReferences(); 
                
                // if item has 0 references, means 
                // the item isn't more used so , 
                // delete from main database 
                if ( (*it).second->GetReferencesCount()==0 ) 
                { 
                    // call the destructor 
                    delete( (*it).second ); 
                    Map.erase( it ); 
                } 
                
                return true; 
            } 
            
            CMessage::Error("cannot find %s\n",FileName.c_str()); 
            
            return false; 
        } 
        
        ////////////////////////////////////////////////////////////////////// 
        // initialise 
        void Initialise( const std::string &name ) 
        { 
            // check if name is not empty 
            if ( name.empty() ) 
                CMessage::Error("Null name is not allowed"); 
                
            // normalize it 
            Name=CStringFormatter::TrimAndLower( name ); 
        } 
        
        //////////////////////////////////////////////// 
        // get name for database 
        const std::string &GetName() const { return Name; } 
        const int Size() const { return Map.size(); } 
        
        /////////////////////////////////////////////// 
        // ctor / dtor 
        CResourceManager() { } 
        ~CResourceManager() { ReleaseAll(); 
    } 
};

Mapping Resources

The resource manager presented here is fully functional of its own, but we want to be able to use assets inside a game object represented by a class. Think about a 3D object, which is made of different 3D meshes, combined together in a sort of hierarchial structure, like a simple robot arm, makes the idea clearer. The object is composed of simple building blocks, like cubes and cylinders. we want to reuse every object as much as possible and also we want to access them quickly, in case we want to rotate a single joint. The engine must fetch the object quickly, without any brute force approach, also we want a name for the asset so we can address it using human readable names, which are easier to remember and to organize. The idea is to write a resource mapper which uses another unordered map using strings as keys and addresses from the resource database as the mapped value. We need also to specify if we want to allow the asset to be present multiple times or not. The reason behind this is simple - think again at the 3D robot arm. We need to use multiple times a cube for example, but if we use the same resource mapper for a shader, we need to keep each of the shaders only once. Everything will become clearer as the code for the mapper unfolds further ahead.

template < class T > class CResourceMap 
{ 
private: 
    ///////////////////////////////////////////////////////// 
    // find in all the map the value requested 
    bool IsValNonUnique( const std::string &filename ) 
    { 
        // if duplicates are allowed , then return alwasy true 
        if ( Duplicates ) return true; 

        // else , check if element by value is already present 
        // if it is found, then rturn treu, else exit with false 
        std::unordered_map< std::string, T* >::iterator it= Map.begin(); 
        while( it != Map.end() ) 
        { 
            if ( ( it->second->GetResourceFileName() == filename ) ) 
                return false; 
                
            ++it; 
        } 
        
        return true; 
    } 
    
    ////////////////////////////////////////////////////////////////////////////// 
    // private data 
    std::string Name; 
    
    // name for this resource mapper 
    int Verbose; 
    
    // flag for debugging messages 
    int Duplicates; 
    
    // allows or disallwos duplicated filenames for resources 
    CResourceManager *ResourceManager; 
    
    // attached resource manager 
    std::unordered_map< std::string, T* > Map; 
    
    // resource mapper 
    // copy constructor and = operator are kept 
    private CResourceMap(const CResourceMap&) { }; 

    CResourceMap &operator = (const CResourceMap& ) { return *this; } 
    
public: 
    ////////////////////////////////////////////////////////////////////////////////////// 
    // adds a new element 
    T *Add( const std::string &resourcename,const std::string &filename,void *args=0 ) 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (5)" ); 
            
        if ( filename.empty() ) 
            CMessage::Error("%s : filename cannot be null",Name.c_str()); 
        
        if ( resourcename.empty() ) 
            CMessage::Error("%s : resourcename cannot be null",Name.c_str()); 
            
        std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); 
        
        // looks in the hashmap to see if the 
        // resource is already loaded 
        std::unordered_map< std::string, T* >::iterator it = Map.find( ResourceName ); 
        
        if ( it==Map.end() ) 
        { 
            std::string FileName=CStringFormatter::TrimAndLower( filename ); 
            
            // if duplicates flag is set to true , duplicated mapped values 
            // are allowed, if duplicates flas is set to false, duplicates won't be allowed 
            if ( IsValNonUnique( FileName ) ) 
            { 
                T *resource=ResourceManager->Load( FileName,args ); 
                
                // allocate new resource using the raii paradigm 
                Map.insert( std::pair< std::string, T* > ( ResourceName, resource ) ); 
                return resource; 
            } else { 
                // if we get here and duplicates flag is set to false 
                // the filename id duplicated 
                CMessage::Error("Filename name %s must be unique\n",FileName.c_str() ); 
            } 
        } 
        
        // if we get here means that resource name is duplicated 
        CMessage::Error("Resource name %s must be unique\n",ResourceName.c_str() ); 
        return nullptr; 
    } 
    
    ///////////////////////////////////////////////////////// 
    // delete element using resourcename 
    bool Remove( const std::string &resourcename ) 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (4)"); 
            
        if ( resourcename.empty() ) 
            CMessage::Error("%s : resourcename cannot be null",Name.c_str()); 
            
        std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); 
        
        if ( Verbose ) 
            CMessage::Trace("%-64s: Removal proposal for : %s\n",Name.c_str(),ResourceName.c_str() ); 
            
        // do we have this item ? 
        std::unordered_map< std::string, T* >::iterator it = Map.find( ResourceName ); 
        
        // yes, delete element, since it is a reference counted pointer, 
        // the reference count will be decreased 
        if ( it != Map.end() ) 
        { 
            // save resource name 
            std::string filename=(*it).second->GetResourceFileName(); 
            
            // erase from this map 
            Map.erase ( it ); 
            
            // check if it is unique and erase it eventually 
            ResourceManager->Unload( filename ); 
            return true; 
        } 
        
        // if we get here , node couldn't be found 
        // so , exit with an error 
        CMessage::Error("%s : couldn't delete %s\n",Name.c_str(), ResourceName.c_str() ); 
        return false; 
    } 
    
    ////////////////////////////////////////////////////////// 
    // clear all elements from map 
    void Clear() 
    { 
        std::unordered_map< std::string, T* >::iterator it=Map.begin(); 
    
        // walk trhough all the map 
        while ( it!=Map.end() ) 
        { 
            // save resource name 
            std::string filename=(*it).second->GetResourceFileName(); 
            
            // clear from this map 
            it=Map.erase ( it ); 
            
            // check if it is unique and erase it eventually 
            ResourceManager->Unload( filename ); 
        } 
    } 

    ////////////////////////////////////////////////////////// 
    // dummps database content to a 
    string std::string Dump() 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (3)");

        std::string str=CStringFormatter::Format("\nDumping database %s\n\n",Name.c_str() ); 
        
        for ( std::unordered_map< std::string, T* >::iterator it = Map.begin(); it != Map.end(); ++it ) 
        { 
            str+=CStringFormatter::Format("resourcename : %s , %s\n", (*it).first.c_str(), (*it).second->GetResourceFileName().c_str() ); 
        } 
        
        return str; 
    } 

    ///////////////////////////////////////////////////////// 
    // getters 
    ///////////////////////////////////////////////////////// 
    // gets arrays name 
    const std::string &GetName() const { return Name; } 
    const int Size() const { return Map.size(); } 

    ////////////////////////////////////////////////////////// 
    // gets const reference to resource manager const 
    CResourceManager *GetResourceManager() { return ResourceManager; } 

    ///////////////////////////////////////////////////////// 
    // gets element using resourcename, you should use this 
    // as a debug feature or to get shared pointer and later 
    // use it , using it in a section where performance is 
    // needed might slow down things a bit 
    T *Get( const std::string &resourcename ) 
    { 
        if ( ResourceManager==NULL ) 
            CMessage::Error("DataBase cannot be NULL (2)"); 
            
        if ( resourcename.empty() ) 
            CMessage::Error("%s : resourcename cannot be null",Name.c_str()); 
            
        std::string ResourceName=CStringFormatter::TrimAndLower( resourcename ); 
        std::unordered_map< std::string, T* >::iterator it; 
        
        if ( Verbose ) 
        { 
            CMessage::Trace("%-64s: %s\n",Name.c_str(),CStringFormatter::Format("Looking for %s",ResourceName.c_str() ).c_str()); 
        } 
        
        // do we have this item ? 
        it = Map.find( ResourceName ); 
        
        // yes, return pointer to element 
        if ( it != Map.end() ) 
            return it->second; 
        
        // if we get here , node couldn't be found thus , exit with a throw 
        CMessage::Error("%s : couldn't find %s",Name.c_str(), ResourceName.c_str() ); 
        
        // this point is never reached in case of failure 
        return nullptr; 
    } 

    ///////////////////////////////////////////////////////// 
    // setters 
    void AllowDuplicates() { Duplicates=true; } 
    void DisallowDuplicates() { Duplicates=false; } 
    void SetVerbose() { Verbose=true; } 
    void SetQuiet() { Verbose=false; } 

    //////////////////////////////////////////////////////////// 
    // initialise resource mapper 
    void Initialise( const std::string &name, CResourceManager *resourcemanager, bool verbose,bool duplicates ) 
    { 
        if ( resourcemanager==NULL ) 
            CMessage::Error("DataBase cannot be NULL 1"); 
            
        if ( name.empty() ) 
            CMessage::Error("Array name cannot be null"); 
            
        Name=CStringFormatter::TrimAndLower( name ); 
        
        // normalized name 
        string ResourceManager=resourcemanager; 
        
        // copy manager pointer 
        // setting up verbose or quiet mode 
        Verbose=verbose; 
        
        // setting up allowing or disallowing duplicates 
        Duplicates=duplicates; 
        
        // emit debug info 
        if ( Verbose ) 
        { 
            if ( Duplicates ) 
                CMessage::Trace("%-64s: Allows duplicates\n",Name.c_str() ); 
            else if ( !Duplicates ) 
                CMessage::Trace("%-64s: Disallows duplicates\n",Name.c_str() ); 
        } 
    } 

    ///////////////////////////////////////////////////////// 
    // ctor / dtor 
    CResourceMap() 
    { 
        Verbose=-1; // undetermined state 
        Duplicates=-1; // undetermined state 
        
        ResourceManager=NULL; 
        // no resource manager assigned 
    } 

    ~CResourceMap() 
    { 
        if ( Verbose ) 
            CMessage::Trace("%-64s: Releasing\n",Name.c_str() ); 
        
        Clear(); 
            
            // remove elements if unique 
    } 
}; 

Basically, the class is a wrapper for the resource database operations. Among the private data, as you can see, its present an unordered map where the first key is a string and the mapped value is the pointer directly mapped from the resource database. Let's have a look at the function members now. The Add function performs many tasks. First, checks if the name for the asset is already present, since duplicated names for the assets are not allowed. If name is not present, it performs the attempt to upload the assets from the resource database, then it checks if the filename is unique and if the duplicates flag is not set to true. Here I have used a brute force approach, the reason behind it is that if I wanted to have a sort of bidirectional mapping, I should have used a more complex data structure, but I wanted to keep things simple and stupid. At this point the resource database uploads the asset, and if it's present, it hands back immediately the address for the required resource. If not it loads it, making the process completely transparent for the resource mapper and it gets stored in the unsorted map data structure. Note again, that all the error checking are just wrappers for a throw, you may want to replace with your error checking code, without compromising the Add function itself. The Remove function is a little bit more interesting, basically the safety checks are the same used in Add, the resource is erased from the map, the resource database removal function is invoked, but the resource database doesn't destroy it if it is still shared in some other places. By 'some other places' I mean that the asset may be still be present in the same resource mapper or in another resource mapper instantiated somewhere else in your game. This will be clearer with a practical example further ahead. The Clear function basically performs the erasure of the entire resource map, using the same counted reference mechanism from the resource database. The Get function retrieves the named resource by specifing its resource name and gives back the resource pointer. The Initialise function attaches the resource mapper to the resource database.

Example of Usage

First of all, we need a base class, which could be a game object. Let's call it foo just for example

class CFoo : public vml::CResourceBase 
{ 
    //////////////////////////////////////////////////// 
    // copy constructor is private 
    // no copies allowed since classes 
    // are referenced 
    
    CFoo( const CFoo &foo ) : CResourceBase ( foo ) { } 
    
    //////////////////////////////////////////////////// 
    // overload operator is private, 
    // no copies allowed since classes 
    // are referenced 
    CFoo &operator =( CFoo &foo ) 
    { 
        if ( this==&foo ) 
            return *this; 

        return *this; 
    } 
    
public: 
    //////////////////////////////////////////////// 
    // ctor / dtor 
    // this constructor must be present 
    CFoo(const std::string &resourcefilename, void *args ) : CResourceBase( resourcefilename,args ) { } 
    
    // regular base constructor and destructor 
    Cfoo() {} ~CFoo() { } 
}; 

Now we can instantiate our resource database and resource mappers.

CResourceManager rm; 
CResourceMap mymap1; 
CResourceMap mymap2;

I have createed a resource manager and two resource mappers here.

// create a resource database 
rm.Initialise("FooDatabase", vml::CResourceManager::VERBOSE ); 

// attach this database to the resource mappers, bot of them allowd duplicates 
mymap1.Initialise( "foolist1",&rm, true,true ); 
mymap2.Initialise( "foolist2",&rm, true,true ); 

// populate first resoruce mapper 
// the '0' argument means that the resource 'a' , whose filename is foo1.txt 
// doesn't take any additional values at construction time 
mymap1.Add( "a","foo1.txt",0 ); 
mymap1.Add( "b","foo1.txt",0 ); 
mymap1.Add( "c","foo2.txt",0 ); 
mymap1.Add( "d","foo2.txt",0 ); 
mymap1.Add( "e","foo1.txt",0 ); 
mymap1.Add( "f","foo1.txt",0 ); 
mymap1.Add( "g","foo3.txt",0 ); 

// populate second resource mapper 
mymap2.Add( "a","foo3.txt",0 ); 
mymap2.Add( "b","foo1.txt",0 ); 
mymap2.Add( "c","foo3.txt",0 ); 
mymap2.Add( "d","foo1.txt",0 ); 
mymap2.Add( "e","foo2.txt",0 ); 
mymap2.Add( "f","foo1.txt",0 ); 
mymap2.Add( "g","foo2.txt",0 ); 

// dump content into a stl string which can be printed as you like 
std::string text=rm.Dump();

Running this example and printing the text content gives:

Dumping database foodatabase 
Filename : foo1.txt , references : 7 
Filename : foo2.txt , references : 4 
Filename : foo3.txt , references : 3

This concludes the article. I hope it will be useful for you, thanks for reading.