Hi

Im trying to implement ECS pattern in my game but I have a few questions

We have some entitites, all have position component (x and y) and sprite component, someones have a pickable component (boolean)

In the draw o rendering system, we iterate over 2 arrays, position array and sprite array, no problems

But, for instance, if I need to create a pick system (checks if a user clicks in a entity), the system needs to iterate over the position array (ok, all entities have) and need the pickable component, that no all entities uses.

I dont know what is the best aproach

• Create arrays for each system : echa system uses the data they need, (too much arrays??) , manager adds components in each array
• Iterate over entities : but this is not ecs pattern
• Create arrays of ids of entities for each system: each system has an array with the ids of the entities , and all the systems looks the same array of entities with the id of the entitite (the index) and check the components
• iterate over array of positions, and when we get the correct position, check if the entity of that component has a pickable component, I dont know if this is slow

What do you think, what is the way of doing this?

Thanks

testing6 said:
Iterate over entities : but this is not ecs pattern

First of, you should try to get that out of your head. You don't need to be afraid that the ECS-police is going to arrest you if you stray from what the core-pattern is about ?

I've personally always had this pattern for systems from the get-go:

void System::Update(void)
{
	for (const auto& [position, sprite, collision] : GetComponentPacks<Position, Sprite, Collision>())
	{
	}
}

The first implementation did actually just iterate over all entities and build this list each frame. This is of course pretty slow, but even that was fast enough for most use-cases.

My current, optimized approach builds a cache of those components, interally using some template-magic, but in essense, I create an array of the requested components which I update when entities are created/updated/removed. Essentially, its interally like this:

std::vector<std::tuple<std::reference_wrapper<Position>, std::reference_wrapper<Sprite>, std::reference_wrapper<Collision>>>;

And “GetComponentPacks” returns an interator into this structure.
The first iteration btw simply cleared the entire cache whenever anything mandated its layout to chance, the current applies updates after all systems are done iterating. You could make the process of updating a bit faster by not storing the components itself but indices, but this will also need updates (when components are removed in between) and I have found that the speedup from not having to do lookups in the different component-stores far outweight the cost of updating the cache (which can be mitigated alltogether by using standard-practices like pooling).

And thats why I originally said - don't worry too much about if this or that implementation is strictly ECS or not. If you have a well-enough interface you can change implememtations to meet your requirements, and you can just start with something that works instead of having to come up with the perfect solution (which rarely exists anyways).

Juliean said:
I create an array of the requested components which I update when entities are created/updated/removed.

That sounds basically like what I do too. I have a set of interest_lists, which contain the entityid's of those entities that have an interest in being handled because of their component x. Those are in a global scope, accessible from every system. They are managed by a system which adds, removes entity id's as needed. Only that one system is allowed to change the content of these interest_lists.

Essentially this is a question about how you want to do the bookkeeping.

testing6 said:
the system needs to iterate over the position array

Why?

It's not uncommon for each system to have its own array of components. The “pickable” system will have components only for objects that are pickable. Thus, you iterate over the pickable components. The pickable component then needs to expose some interface to know its position. There are many ways to do that:

1. Whenever the object moves, copy the position into a field in the pickable component.
2. The pickable component contains a pointer to the position component of its representative entity.
3. The pickable component contains a pointer to the entity, and you can then get the position from there.

Also, when I say “pointer” I could just mean “index into array” if you don't want to use direct pointers for whatever reason (savegames are easier without pointers, for example, as are Rust games.)

Also, it's totally OK to make the law that “every entity has a position, so you there's no separate position component – position is just a field in the entity record itself.”

Unity as one of the most prominent users of ECS(-ish systems) is doing exactly that when they introduced BURST. An example was a system which first created an array of entities and filled that array by iterating over all entities present in the game by certain criteria. Then that entities who for example match a set of components were put into that array and the array was passed every time BURST called for an update. So it is absolutely fine to iterate entities in order to find those who match certain needs of a system. You should take care to do that only once something changes or you initialize a new system to be added to the update loop, don't do that on every frame. Or you register some kind of update pipeline whenever a component is attached to an entitty or removed from it, different systems listen to that pipeline and either add or remove that entity to/from their internal array.

For those pipelines I started to like the reactive programming pattern https://gist.github.com/staltz/868e7e9bc2a7b8c1f754​​​ . You however need to implement that in a way that is good for games performance and memory wise

hplus0603 said:
Also, it's totally OK to make the law that “every entity has a position, so you there's no separate position component – position is just a field in the entity record itself.”

The one downside of this is that it couples the entity with the position. In my engine, 2D and 3D are entirely separate implementations with separate transforms (Transform2D/Transform3D). And while its true that every entity always has a Transform, it is not given which, at least on the entity-framework level.

I personally did simply make a separate type of component, which is specialized for always existing on every entity. This uses a separate store, and a O(1) array-lookup for an entity, but it still allows me swap those transforms, alongside some other properties.

Now you can decide whether or not this decoupling makes sense to you (for my design it does), but if you don't think you need it, then sure, it can also be a field as you described.

The problem I have with ECS is that people seem under the assumption that you always iterate all entities with a specific combination of components. From my experience this is very far from what game engines do and you only want to iterate a very small subset of components in practice . Namely those which actually changed or those which are actually updating. E.g. In a physics system you don't iterate the rigid body component after you updated the physics world (think Havok or PhysX), but you only update those entities where the rigid body in your physics systems moved. And this is very often none.

Here is an example:

// Instead of this...
void PhysicsSystem::Update( float dt )
{
	mWorld->Step( dt );
	
	// This can be a couple of 10.000 components...
	for (const auto& [RnRigidBody] Body: GetComponentPacks<RnRigidBody>())
	{
		RnTransform Transform = Body->GetTransform();
		Body->GetEntity()->SetTransform( Transform );
	}
}

// ... you actually do this
void PhysicsSystem::Update( float dt )
{
	mWorld->Step( dt );

	// This is most often in the 10s or maybe 100s
	for ( RnBody* Body : mWorld->GetActiveBodies() )
	{
		RnTransform Transform = Body->GetTransform();
		Body->GetEntity()->SetTransform( Transform );
	}
}

Another example would be animation. You might want to update the animation graph and move the character, but you only create a pose if the character is actually going to be rendered on the screen. Also that assumes that animation is updated every frame, but the NPC graphs might be on much slower clocks in practice. In my opinion this is all an over simplification → namely everything is an array.

Dirk Gregorius said:
The problem I have with ECS is that people seem under the assumption that you always iterate all entities with a specific combination of components. From my experience this is very far from what game engines do and you only want to iterate a very small subset of components in practice .

This might be true to an extend. The main counter-argument to that is that the way ECS is setup, having to iterate over a bunch of entities that are not really truly active has a tiny overhead, if you don't do any work. So if the code did this:

void PhysicsSystem::Update( float dt )
{
	mWorld->Step( dt );
	
	// This can be a couple of 10.000 components...
	for (const auto& [RnRigidBody] Body: GetComponentPacks<RnRigidBody>())
	{
		if (Body.isActive)
		{
			RnTransform Transform = Body->GetTransform();
			Body->GetEntity()->SetTransform( Transform );
		}
	}
}

In a well-optimized ECS, then having to iterate over 10000 enities when only 10 are active has less of an overhead then you might expect. Sure, it is still O(N) for 10000 instead of 10/100 whatever, but its usually tolerable.

Furthermore, an ECS doesn't prevent you from creating more optimized data-structures for more specific processing-requirements. There is nothing keeping you from creating a link between the active bodies and the components, and using the active-list from your example #2 instead of iterating over all entities. The much more common example in my own codebase is actually scripting. I have a top-down rpg with complex, custom interactions on pretty much any map. Of course I don't write my scripts with a GetComponentPacks-loop when I need to change the texture of a chest that has been opened. This is also happening on an individual entity basis.

Does that overall mean that ECS doesn't fit all the requirements of a game-engine? Yeah, sure. But it doesn't have to. A game-engine is a complex enough beast that it requires lots of different patterns and structures to work, ECS in my opinion can fit quite well as a ground-level foundation/glue between other systems. You still need enough flexibility to stray from the pattern where its required, so as long as you don't treat ECS absolutely literal like in its definition, and don't try to apply it literally to everything, it works quite well (at least in my experience).

@Juliean: The main counter-argument to that is that the way ECS is setup, having to iterate over a bunch of entities that are not really truly active has a tiny overhead, if you don't do any work

This is unfortunately not true. I understand that there is a common understanding that if just everything is in a array iteration is free. Say your overhead is 500us (micro seconds) which is not unlikely in the above example since your innocent looking Body.isActive is not a simple boolean in the cache but will query the state of the PhsicsSDK body through a pointer. This seems not much, but it is not tolerable and will add up. Assume you have 10 systems like this then a third of your frame time is wasted for nothing.

When we shipped Half-Alyx the major performance work in the end was to get rid of these tolerable overheads.

Actually in one place I worked the animation programmer was of the opinion that if he just has all bones linearly in memory (which you absolutely should!) he could just iterate over all skeletons and naively build poses. Well, guess what, it didn't work and the first thing we added was a callback from the rendersystem to skip this step if the character is not rendered. So I would argue a thing like ‘tolerable overhead’ does not exist.

Dirk Gregorius said:
This is unfortunately not true. I understand that there is a common understanding that if just everything is in a array iteration is free. Say your overhead is 500us (micro seconds) which is not unlikely in the above example since your innocent looking Body.isActive is not a simple boolean in the cache but will query the state of the PhsicsSDK body through a pointer. This seems not much, but it is not tolerable and will add up. Assume you have 10 systems like this then a third of your frame time is wasted for nothing.

Well if its not a boolean but a query then sure, the overhead is going to be much larger. I was assuming that you are able to mark the components as active/inactive when this state actually changes. If thats not then case then sure this adds up, but than thats the overhead of querying the physics-system and not having to iterate over inactive components. If it can work like in my above example with a bool, I'm pretty confident the overhead should be next to nothing, and never add up close to a third of frame-time (based on my own extensive testing of my own system).

Dirk Gregorius said:
So I would argue a thing like ‘tolerable overhead’ does not exist.

Now we are getting into a larger debate; but if overhead is never tolerable than everybody would always write perfectly hand-crafted assembly since obviously you cannot waste the 0.1us from some instruction that the compiler generated that you wouldn't have? Of course the first priority should be having code/tools that are easy to work with, and as long as you can hit your target-framerate with some a lot of leeway, then everything else is IMHO absolutely tolerable.

As an example, my (ECS-based) 2D collision-detection currently uses O(N^2) collision-checking, since I have at max 50-100 collidable entities on screen. But my game-build still runs at up to 3000 FPS if left uncapped. So yes, I'd again say that this is absolutely overhead that is acceptable. Until I need to have more collible units, then I'd need to write a more sophisticated algorithm.