I'm not sure if this is the right forum, but I think that my AI is the core of the issue.
Ok, the issue is to make a AI written in lua multithreaded. I've several issues with this.

1. (my) AI controls the lifecycle(create->update->destroy) of many entities and manipulates its internal states(position,physics properties, visual properties etc.) which are needed by other parts of the engine (reader-writer problem)

2. I'm not sure if you can really efficiently share a single lua state between several threads and access it concurrently.

3. Avoid frequent locking via (OS) mutex to prevent stalling.

This is my current approach of multithreading in my engine:
To avoid the reader-writer problem, a writing thread use either a copy of the original data (double buffering) or has exclusive access. In case of my AI, the whole AI is executed exclusively in a single thread.

My idea would be to execute parts of the AI (entity behavior trees) with a lockstep approach where each part can only manipulate either a copy of the data or enqueues a post-processing action which will be executed after the parallel lockstep in a single thread. But to do this I need to access a single lua state concurrently by multiple threads.

I hope this is enough information to understand my problem. Maybe lockstep mechanism used in multiplayer RTS games is the solution (I'm not familiar with it), or there's is no solution to this problem at all.

I would appreciate any help

The manual used to state - I haven't read it lately, but I doubt it has changed much - that it can't be done without writing your own wrapper that manages locking of Lua state. Can't you have multiple Lua states instead, if you have different parts of AI executing? My own engine uses a separate Lua state for each entity in the gameworld, some of which control others through message passing constituting AI so that they can be running in separate threads. If you already have AI written with single Lua state in mind then not much else to do but to guard all Lua entry points with your own locks. Having multiple Lua states is much easier in general though, if it works for you of course.

The approaches I've seen to concurrent Lua all rely on multiple states, generally at a ratio of at least one state per thread.

Hmm... looks like I need multiple lua states. My first idea is to use one "world" state like the one I currently got and use several "worker" states to do some of the easier, but still time consuming tasks(need to profile this) .

To be more concrete, I would copy a small part of AI data(some properties, stimuli) for each active entity to the worker and let my behavior tree work on this data as long as it reached a node which needs to be executed in the "world" context. In my behavior tree implementation lot of work is done in checking changing conditions, which could be done here.

To copy the AI data between world and worker state I would use lua lanes (based on linda objects).

Still, I'm somewhat suspecious if the performance boost will win over the management overhead (synchronisation, data copy etc.), I need to do some testing here.

Is anybody familiar with lua lanes or got some experiences with a similiar approach ?

The reader-writer problem is common and is probably best avoided by not having it at all

Double buffer your state public state (position etc) and have all AI tasks reach some form of 'completion' before moving on to other processing. I don't mean complete the AI completely just do a certain segment of the AI work, save state and return (Lua co-routines would probably help you here).

You still need multiple Lua states but your game logic should become easier to manage.