Advertisement

Pathfinding grid and memory consumption.

Started by May 21, 2009 08:51 AM
17 comments, last by SymLinked 15 years, 6 months ago
Quote: Original post by Emergent
Quote: Original post by SymLinked
It's the graph, a bunch of nodes each with its own list of neighbours.


Then just let the graph be implied by an occupancy grid. Don't explicitly store neighbors anywhere. The neighbors are just whatever cells in the occupancy grid that you can get to in one move that are not occupied. You could even wrap this stuff in accessor functions so that it "looks" like a graph; e.g., consider a "getNeighbors" function that, given an index into the occupancy grid, returns the indexes of all adjacent cells that are not occupied. You can wrap this up in whatever OO-prettiness you want in the language of your choice; e.g., you could have an abstract "GraphNode" class of which "OccupancyGridGraphNode" is an implementation. Under the hood, "OccupancyGridGraphNode" could be implemented as indexes into an occupancy grid, but A* doesn't have to know.

Quote: but it isn't said anywhere how their allocation works in regards to "removed" nodes.


Wait... instead of maintaining a CLOSED list, you want to somehow remove those nodes from memory? I get it. But you still need some representation of your map somewhere in memory at all times, so rather than copying that and then deleting that copy as you go, why not just work with the original representation?


I'm thinking we're on a different technical level here or we're simply talking about different stuff. :) I'm talking about the grid which occupies the whole area meant to be pathfinded trough. Each node has a neighbour list and a integer which determines the type (walkable, wall - etc). That's the list that's too large.

I think I've found that I'm going to settle on a hash_map that I can access via the coordinates of the grid. That's going to cut down memory consumption a ton, I'm just not sure how well it will perform just yet. Hence my question about Boost :)
Quote: Original post by SymLinked
I'm thinking we're on a different technical level here or we're simply talking about different stuff. :) I'm talking about the grid which occupies the whole area meant to be pathfinded trough. Each node has a neighbour list and a integer which determines the type (walkable, wall - etc). That's the list that's too large.


Yeah, basically what they're suggesting is have a data-compilation stage in your pipeline. Walk your map and create a node graph:

struct Node{    Vector3 position;    std::vector<Node*> neighbors;};


So you programatically explore your maps and essentially throw away all the nodes that are not walkable. Then at runtime you pathfind through this pruned node list rather than your grid.

What's the size of your grid? Novice mistake but are you possibly creating the node grid on the stack instead of the heap ( Node myNodes[width][height] instead of Node *myNodes = new Node[width * height]; )

-me
Advertisement
Quote: Original post by Palidine
Yeah, basically what they're suggesting is have a data-compilation stage in your pipeline. Walk your map and create a node graph


Actually, I'm saying exactly the opposite! Don't build a whole big redundant data structure so you can have a "graph." You have a map somewhere in memory already (you must! You need to draw it, among other things). That's all you need. The "graph" that A* runs on exists in your head and maybe in your function names, not as some data structure built out of structs and pointers which sits in your computer's memory and which contains no information beyond what's in your map!

Quote: Original post by SymLinked
Each node has a neighbour list


Why?! If your map is a grid, you know exactly what your neighbors are implicitly.

Here's what I mean:

Do you have a 4-connected or an 8-connected grid? Let's say 4-connected for simplicity. And say I'm at (10,12). What are my neighbors? They are whichever of (11,12),(9,12),(10,13),(10,11) are walkable. You know that just by looking at your map. You don't need to build some list ahead of time with this information!!

Rather than (pseudocode),
For each element in Node's neighbor list   do stuffend

skip having a neighbor list altogether and do
For each cell C of the four cells in my grid adjacent to Node // You do this by adding to your (x,y) coordinates.   If C is walkable // Check your occupancy grid for this       do stuff   endend


Do you get what I mean?
I think the OP is saying that linking to neighbors implicitly won't work because he doesn't want to store the grid in its entirety, but rather only the nodes of interest (i.e. those that are reachable). Also, it doesn't sound like a bitfield will work, since the cells store more information than simply 'blocked or unblocked'.

The links could still be made implicit with the hash table method, but IMO it would probably be easier to make the links explicit in this case (at least that's what I'd do).
Quote: Original post by jyk
I think the OP is saying that linking to neighbors implicitly won't work because he doesn't want to store the grid in its entirety, but rather only the nodes of interest (i.e. those that are reachable).


I think I've started to "get" that this is what he's thinking. But,
1 - "Storing only the nodes of interest" isn't necessarily smaller, because it necessitates a more complicated data structure. There's also a lot of redundancy when you have explicit pointers to neighbors: If your graph is undirected (meaning: If I can walk from a to b, then I can always walk from b to a), then why do we need to store a pointer in each direction? That's 64 or 128 bits right there for something we knew a priori anyway.
2 - If I were a betting man, I'd put money on OP having the entire map in memory anyway. How else can he draw the level, for instance? If he has this, then also having another data structure is just redundant and inefficient.

Quote: Also, it doesn't sound like a bitfield will work, since the cells store more information than simply 'blocked or unblocked'.


Yeah, I kind of noticed that too... This still doesn't really change my argument. Say you can have any of 232 kinds of terrain (I sincerely doubt he has more than 256, in which case he can use bytes, but my point is that this even holds when the number of kinds of terrain is relatively large). With the 1-1 hash method (i.e., just a 2d array) you only need to store one 32-bit integer per node. Whereas with the explicit links method, you need to store both this integer and four 32-bit pointers (assuming a 4-connected grid). That's 5x the memory for no purpose.

Anyway, I think I've probably made my point by now, so I'll stop banging on about this...
Quote: I think I've started to "get" that this is what he's thinking. But,
1 - "Storing only the nodes of interest" isn't necessarily smaller, because it necessitates a more complicated data structure. There's also a lot of redundancy when you have explicit pointers to neighbors: If your graph is undirected (meaning: If I can walk from a to b, then I can always walk from b to a), then why do we need to store a pointer in each direction? That's 64 or 128 bits right there for something we knew a priori anyway.
2 - If I were a betting man, I'd put money on OP having the entire map in memory anyway. How else can he draw the level, for instance? If he has this, then also having another data structure is just redundant and inefficient.
Well, he could (presumably) draw the level by iterating over the nodes in the reduced list and drawing each in turn. But you're right - since we don't really know much about the application, these are all just guesses, more or less :)
Advertisement
Quote: Original post by Palidine
So you programatically explore your maps and essentially throw away all the nodes that are not walkable. Then at runtime you pathfind through this pruned node list rather than your grid.

What's the size of your grid? Novice mistake but are you possibly creating the node grid on the stack instead of the heap ( Node myNodes[width][height] instead of Node *myNodes = new Node[width * height]; )


It's allocated on the heap. I used a build I found on the web and they store something like ~48 bytes per node. That's a lot of memory. All three examples I've looked at store their neighbours in a Vector per node, etc.

Quote: Original post by jyk
Quote: 2 - If I were a betting man, I'd put money on OP having the entire map in memory anyway. How else can he draw the level, for instance? If he has this, then also having another data structure is just redundant and inefficient.

Well, he could (presumably) draw the level by iterating over the nodes in the reduced list and drawing each in turn. But you're right - since we don't really know much about the application, these are all just guesses, more or less :)


I have the entire map in memory, yeah. But it's nothing you can use to pathfind, or else I would have used it. It's a seperate data structure of AABB's and its much less dense.

Anyhow, I'll get to work. If I do try the Boost libs I'll post back the results here. Thanks alot for the discussion guys!
Quote: Original post by SymLinked
Quote: Original post by Palidine
So you programatically explore your maps and essentially throw away all the nodes that are not walkable. Then at runtime you pathfind through this pruned node list rather than your grid.

What's the size of your grid? Novice mistake but are you possibly creating the node grid on the stack instead of the heap ( Node myNodes[width][height] instead of Node *myNodes = new Node[width * height]; )


It's allocated on the heap. I used a build I found on the web and they store something like ~48 bytes per node. That's a lot of memory. All three examples I've looked at store their neighbours in a Vector per node, etc.

Quote: Original post by jyk
Quote: 2 - If I were a betting man, I'd put money on OP having the entire map in memory anyway. How else can he draw the level, for instance? If he has this, then also having another data structure is just redundant and inefficient.

Well, he could (presumably) draw the level by iterating over the nodes in the reduced list and drawing each in turn. But you're right - since we don't really know much about the application, these are all just guesses, more or less :)


I have the entire map in memory, yeah. But it's nothing you can use to pathfind, or else I would have used it. It's a seperate data structure of AABB's and its much less dense.

Anyhow, I'll get to work. If I do try the Boost libs I'll post back the results here. Thanks alot for the discussion guys!
The Boost Graph library *might* be overkill for what you're doing, IMO. I'm just speculating though - I haven't actually used it myself (I'm just going by what I see in the documentation).

In any case, it seems to me that there are some pieces missing from the puzzle here (either that, or we're misunderstanding the problem). You say that the entire map grid is stored in memory for pathfinding purposes, but also that each node stores an array (std::vector?) of links to neghboring nodes. Something doesn't add up here :) If the map is a regular grid and you're storing the entire thing in memory (including unreachable cells), then you *certainly* don't need to store the links explicitly, much less in a dynamically resizable container (which is what I'm assuming you mean by 'vector').

I don't know where the 48 bytes per node is going exactly; I also don't know how many bytes you're using to store each link. However, it seems likely that dropping the explicit links would reduce the memory footprint considerably.

What are the dimensions of your map, by the way? Also, is the 'node' struct or class something you could post the declaration for?

Again, I would hold off on Boost Graph for the moment, and instead focus on ensuring that you're not storing more information per node than is necessary.
Thanks for being so persistant in answering my questions. :)
I've gone trough 6 different pathfinders now, 4 of them done by individuals from the game industry and they claim their implementations are efficient. With that said, all 4 of these implementations use the 8 adjacent node neighbours list idea, for each node. - That's why I used it. No point in going against the flow, I thought.

Each node has:
Quote:
4 bytes for index.
8 bytes for position.
4 bytes for "extra information".
A std::vector for neighbours.
4 bytes for state.


I would remove all of these except "state" if I understood how the sparse graph is used. But I don't. The grid is currently 1024x1024, but that's the maximum I'd expect to use. Still, I'm bumping into 160 MB used and that's just way too much.

I'm going to throw all this onto another "lib" now (STL A* Search by Justin Heyes-Jones) since it doesn't seem to care how the node structure looks like which gives me a chance to use a hash table just made up of states, I believe. Hopefully that'll work out.

At least I can jump between different implementations as long as their license is compatible with our use.

----------------------------------------

Edit: The library I plugged in now works just like I wanted. (STL A* Search by Justin Heyes-Jones). I started off with a simple array data structure, made up of integers, and replaced it with a hash_map sorted by indexial values. Profiling will show later if it ends up taking too much time but from what I've heard it shouldn't be the bottleneck.

[Edited by - SymLinked on May 22, 2009 4:48:06 PM]

This topic is closed to new replies.

Advertisement