Awtsmoos said:
Even so it would be ideal to just load the 2mb blender file
Oh, if the geometry data is that small, i do not see a need for instancing at all. To make some guess, i'd say the octree should be something like 10 or 20mb at most. Depends ofc, but 230mb seems just way too much!
Using binary files would be the first thing to try. Assuming the worst case, JSON text file uses two bytes per character to support some unicode. A floating point number may need 6 digits, which is 12 bytes vs 4 bytes with a binary file.
But even if we can reduce the size to 230/3, it still feels very large to me.
The next step would be to look at the octree data structure itself.
Awtsmoos said:
here's the code I'm going off of https://github.com/yaakovyitzchak/reebooyaw/blob/main/Octree.js
I'm not good with JS, but i make some assumptions:
The octree adds a triangle to any node it intersects. So we can assume each triangle is maybe in 4 leaf nodes on average. So we duplicate the data 4 times.
I'm not sure if the nodes just store an index to a triangle array (would be good), or if they store actual duplicates of the whole triangle data (e.g. 9 floating point numbers for the vertex points).
If you can answer it's the latter, we have the reason of the big memory requirements. And you should change this to use the former method instead.
The octree stores bounding box per node. But because this is not a loose octree (where each triangle goes into only one node, requiring overlapping bounds), this is not needed.
Instead, you can calculate the bounding box during traversal from the grid coords and tree level.
This adds some complexity to the traversal code and also makes things like raytracing less efficient, but if memory usage is your problem, i'd certainly do this. Because it reduces memory bandwidth, it could be even faster.
It's not clear to me what's the stopping criteria of the octree build process.
There are some options:
Keep subdividing the tree until no node has more than X triangles. (Which might go forever if the geometry has many triangles at a single spot.)
Or, set a maximum of N tree levels in advance, and stop subdivision after that, no matter how many triangles may end up in a single node. (Because your geometry is static, you can tehn find a good memory / performance compromise for N with trial and error.)
It's not clear to me if the octree stores triangles in internal nodes or just in leaf nodes. Explaining those options:
Storing all triangles in leaf nodes is simpler, but it only works well if all triangles are of similar size.
If you have some very large triangles in the scene, a single large triangle would go into a whole lot of leaf nodes, which costs memory and becomes inefficient. Then it's better to put it into internal nodes, because those nodes have much larger cell bounds to solve this problem.
Actually i guess you may end up implementing your own octree data structures, build process and range query / raytracing traversal logic. That's some work, so the above points are probably the main arguments to keep in mind.
It's not easy to find good tutorials on the subject, but very easy to find bad ones.
I just remember this one, which surely is good because the guy is expert and pioneer of realtime raytracing: https://jacco.ompf2.com/2022/04/13/how-to-build-a-bvh-part-1-basics/
It's about BVH, dynamic senes, and surely overkill for your needs, but may be worth reading to grab some general concepts.