Hello! I need some advice on my new library, I'm not really confident about it... First a little of introduction:

One of my biggest problems when creating a game or a game engine was how to perform the resource management (loading files, editing them, etc). I could just use a default "Data" folder, put all my files here and load them using the standard ways but we know that this is something problematic when the project starts to grow. I also don't think (or at least I don't know) there is a good library that could help with this task.

Yeah I could use zlib or something like that but I decided to create my own, I did some improvements here and there, created a decent (at least I think) introduction/tutorial on its git page but I really don't know if any developer would use it on its currently state.

Here is the link https://github.com/RodrigoHolztrattner/Packet (written in c++). It uses hashed paths to perform fast-mapping searches, it has 2 operation modes (one that allows editing and the other that is focused on performance), all resources uses reference counting and it has some other things that are listed on its github page (like the hot-reload capability for example). Unfortunatelly there are no working examples for now but I plan to add them in the future.

Now I ask you, what do you think about using a library like this? Is it something that you would use for your game? Have I done some kind of overthinking when building it? Like I said I'm not really confident about it (although I'm using it on my projects and I feel it suit them really nice).

Thank you for your time!

I only read your ReadMe, not the code, but it looks pretty good

Support for threading, resource dependencies and different modes for shipping vs development are all key features that would be easy to neglect. 

In my engine, instead of a read/write mode and a read-only mode, our resource management system has:

* read only from loose files on local disk (game in dev mode) 

* read only from packed files (game in shipping mode) 

* write/create only to packed files (building shipping data from toolchain). 

* write only to loose files on local disk (building dev data from toolchain). 

As well as the resource loading file system, games also need a completely different read/write file system for saving user data (user profiles, configuration, save games, etc). Is it possible to instantiate two copies of your system - one in compressed read-only mode for resources, and another in edit mode for save games and configs? 

23 hours ago, Hodgman said:

In my engine, instead of a read/write mode and a read-only mode, our resource management system has:

* read only from loose files on local disk (game in dev mode) 

* read only from packed files (game in shipping mode) 

* write/create only to packed files (building shipping data from toolchain). 

* write only to loose files on local disk (building dev data from toolchain). 

This is interesting. Do your system handle packed data updates? Like comparing a manifesto containing info about each file and then adding each one that is outdated/new into the pack itself?
I'm planning to add this feature soon because right now I'm adding network code to my (learning) game engine and I'm planning on building an auto updater for it.

23 hours ago, Hodgman said:

As well as the resource loading file system, games also need a completely different read/write file system for saving user data (user profiles, configuration, save games, etc). Is it possible to instantiate two copies of your system - one in compressed read-only mode for resources, and another in edit mode for save games and configs? 

Yes this is something that can be done in this way, but I'm thinking about including an option just to support thinks like this, but for now two (or more) systems can be instantiated (the mode don't matter) without problems.

2 hours ago, Flicklizic said:

This is interesting. Do your system handle packed data updates? Like comparing a manifesto containing info about each file and then adding each one that is outdated/new into the pack itself?
I'm planning to add this feature soon because right now I'm adding network code to my (learning) game engine and I'm planning on building an auto updater for it.

Nope. If the package is out of date, I just rebuild the entire thing from scratch instead of trying to incrementally update it.

We use steam to auto-update our game, and it works by splitting all large files into 1MB chunks, hashing them, then works out which chunks the user already has / needs to download, then assembles files by appending the appropriate chunks in the right order. 
So, to work well with Steam, when we build a package, I align large files to 1MB boundaries within the package file.
e.g. if the package is currently 456.25MB and I'm about to add a 5MB file, then first I write 0.75MB worth of "zero" bytes first so that my new file will start at a 1MB offset. This way, even though my package builder is completely dumb and rebuilds the entire package from scratch every time, our incremental updates over steam tend to be extremely small.

On that note, all of our "loose" files that are loaded by the development version of the game are also write/create-only, never edited. A build tool compiles "intermediate" asset files created by artists/designers/etc (e.g. a ".PNG" file) into game-ready resource files (e.g a ".texture" file) and writes them into a data directory. The ".texture" is never modified -- instead the PNG is modified and the tool will re-compile it on demand to create the ".texture" again from scratch.
The package generator then takes the manifest of all the loose files that were created by the last build and appends them together into an archive file.

I handle our packages the same way but in 64k chunks. My tool tries to puzzle our asset files into those spaces with as low as possible fragmentation and fills anything else with random bytes because of package protection. The tool has the capability to encrypt the package and/or sign it so we could make modding the game way more complicated. I do this because the engine is performing loads using async Memory Mapped File reads in multiple threads so a chunk is loaded into several system pages and disk cache optimized

15 hours ago, Hodgman said:

We use steam to auto-update our game, and it works by splitting all large files into 1MB chunks, hashing them, then works out which chunks the user already has / needs to download, then assembles files by appending the appropriate chunks in the right order.

So, to work well with Steam,...

10 hours ago, Shaarigan said:

I handle our packages the same way but in 64k chunks. My tool tries to puzzle our asset files into those spaces with as low as possible fragmentation and fills anything else with random bytes because of package protection. The tool has the capability to encrypt the package and/or sign it so we could make modding the game way more complicated. I do this because the engine is performing loads using async Memory Mapped File reads in multiple threads so a chunk is loaded into several system pages and disk cache optimized

This was really helpful. Thanks for your insight guys.

I totally forgot about the steam auto-update system, Its something that I plan to support on my engine.

I'm a little more confident with this project and I'm planning to continue improving it. ?