I am currently attempting to make an asset loading system for my game engine (written in C++) and it's pretty basic. I'm still a newbie, so nothing really sophisticated, basically, though, for each asset, there is an asset loader. For example, to load a mesh, I would do MeshLoader::load(path_to_mesh) this design was working fine until I got to shaders, and the problem I'm having with shaders is that since they require multiple files, or at least for OpenGL, it's not as easy as just doing ShaderLoader::load(path_to_vertex, path_to_fragment) technically it would work however I am working on an editor and I'm experimenting with loading assets by drag and drop and that is mainly what screws everything up because I don't think I can load 1 at a time otherwise the shader would be incomplete so it almost feels like I might need to make a whole system just for shader assets?

steamdog said:
For example, to load a mesh, I would do MeshLoader::load(path_to_mesh) this design was working fine until I got to shaders, and the problem I'm having with shaders is that since they require multiple files, or at least for OpenGL

It doesn't. Technically, yes, you need to pass two separate strings for vertex/pixel. But you can still use one file, as long as you provide a way to separate vertex/pixel.

// vertex shader goes here:
...

#fragment // manual token; parse after loading to split into vertex/fragment

// fragment shader goes here:
...

You define a custom token where vertex goes above, and fragment goes below. You then do a simple string-split and create two separate strings which you can pass to two calls of glShaderSource, allowing you to use one file for the shader.

Also, in general you don't have such a close binding between assets in the simulation and assets in the hardware.

Generally systems will build an abstraction. Shaders are typically abstracted by a material. The material is used in the simulation's realm. The material may or may not have the shaders actually loaded from disk, and the shaders may or may not be loaded in the card, but the simulation can use the material immediately.

It is similar for all other assets. A mesh is useful on the simulation side, but the actual mesh model may or may not be loaded from the disk, and the mesh model may or may not be loaded in any particular LOD into rendering hardware. A texture on the simulation may have a variety of graphics data loaded, the files may or may not be loaded, the files may or may not be on the graphics card. An audio clip similarly, the actual data may or may not be loaded, and may or may not be streaming.

One conceptual pattern is a store, proxy, cache, and loader, although engines call them by various names. The store, the world, the simulation, basically it's the collection of all the things that exist. When you tell the store to create a new resource object it can immediately return a proxy object for the resource. Telling it to create a material can instantly return an object that serves as a proxy to the underlying shaders even though they haven't been loaded from disk yet. Telling the store to create a mesh can instantly return a mesh proxy before the data file is loaded. Repeat for any other resource, you don't need to actually load the final resource, just have enough metadata to describe what that data file would have when it eventually loads. When the game engine decides you're close enough to actually need the resources, near enough to a graphical thing, near enough to an audio source, or if they're otherwise flagged as important or about to be used, the store uses the cache. First the store can check to see if an instance of the resource is already loaded into the cache, and if not, the resource can be loaded using the loader to open of the data files (locally, across the Internet, from a shared network data store, whatever) and pull them into the cache of loaded resources. It will also likely have different caches for different resources as they'll probably be treated differently, audio is likely to be different from physics data, which is likely different from graphics data. At some point later when you're far enough away, or when all the things are gone from memory, the cache can unload the actual resources from the hardware even though the store still says the proxy objects exist.

@Juliean I have not considered that it would be possible to do something like this and it might be what I end up doing, however, I did also realize that it would probably be better to separate the creation of the shader from the loading process so something like this:

ShaderProgram myProgram = createShaderProgram();
ShaderStage vertexStage = loadShaderStage(path_to_vertex);
ShaderStage fragmentStage = loadShaderStage(path_to_fragment);
myProgram.setVertexSource(vertexStage);
myProgram.setFragmentSource(fragmentStage);

This way you can still drag and drop individual files to load them into the editor, but you would just have to create an empty shader and then drag and drop the loaded files into it to set the source which doesn't sound like it would be a terrible idea.

steamdog said:
This way you can still drag and drop individual files to load them into the editor, but you would just have to create an empty shader and then drag and drop the loaded files into it to set the source which doesn't sound like it would be a terrible idea.

Could work, but it's definately not common from all I've seen to have separate files for shader-stages. All common game engines have 1 file per shader. If you want to share a vertex-shader for multiple shader files, you can use an #include. Otherwise it becomes very tiring to have to create and manage 2+ files per shader (which could be more, if you have geometry and tessellation too). I think I even started with separate files in my engine, but merged them later because it was annoying. You can still do it, of course, but I would not recommend it for usability.