A lot of major games on the market have a custom scripting language for their engine. I was wandering how does the binding between the engine and the custom scripting language works? Like for example if we take a script function, when that function is reached by the execution agent does the function get translated to native c++ engine functions being called (depending on the script function content) which results in the creation of game objects?

I've written a few scripting languages. Unless you are doing something fancy like JIT (Just In Time) compilation, you will probably end up implementing an interpreter. Quite often that means you have something like byte codes in memory, that your source gets translated to on the fly, and then you have some code that interprets those byte codes, does your basic program logic, but also calls down to a native language API. Translation to C++ means you have to have a C++ compiler handy or basically write your own which is of course a huge task and not at all worth it. Also there is no real need to write your own scripting language unless you are pretty picky about your syntax and semantics. There are plenty of freely available languages that have C or C++ APIs such as Lua, AngelScript etc…

If you really want to write your own, you might want to get a good complier design book before starting. At least that's what I did, but I haven't done this stuff in a while now.

TBH, most engines use some form of existing language as their “script” layer. Rpg-Maker had Ruby/RGSS, Unity has C#. Unreal had Kismet, which is custom, but thats rather out of the norm. Other large languages are LUA, AngelScript, Python, …

But from a technical standpoint, a lot of the scripting-languages are based on some form of interpreter. That is kind of a virtual machine, processing a stream of instructions, called Bytecode (in a string/blob) is execute on the abstract representation of what a CPU does internally. For example, in my own backend for my visual scripting, there is a list of OpCodes, one of which is a call to a c++-function:

using CallFunctionWrapper = void(*)(const ExecutionState&);

case OpCodes::CallNative:
{
	const auto pFunction = stream.ReadData<CallFunctionWrapper>();
	pFunction(state);

	break;
}

You'd see something similar in LUA where a LUA_State has to be passed to functions to be able to interface with the interpreter/VMs-state.

This has the downside of being slow, so (almost all) languages that are build this way eigther offer some form of JIT-compiler (translating the bytecode int ASM), or nativization (parsing the instructions into, for example, c++ to be then compiled).

That is of course, only true for a subset of languages. C# doesn't have any VM at all and mandates everything to be compiled via JIT before execution. And since pretty much every language could be used for scripting (including C or C++), there is not one right answer. But I think this goes over a few common cases. Ultimately, what each engines choses is ultimately a matter of pro/cons, personal perference, among other factors.
For example, my visual-scripting offers a few features not really exposed via common text-based languages, so using a custom backend was the best option. Unreal on the other hand used Kismet in U3 before Blueprints where invented, and decided to simply use Kismet as their backend for their visual-scripting; though they now heavily incentive using nativization to C++ for performance-reasons. Unity als made the IL2CPP the de-facto standard, and only uses the actual C#/Mono-JIT backend for editor, or if the user chooses to (on platforms the even support it). So

Gnollrunner said:
I've written a few scripting languages…

There`s a lot to learn about the topic. I have to admit the details you`re giving are a bit overwhelming, there`s plenty of study to be done on my behalf.

Juliean said:
TBH, most engines…

thanks Juliean, I have found some things to keep...

From what I learned over the past years, scripting languages may be both, easy to implement and simple but also hard to master if done right. But I alsolearned that it helps if you know something about how .NET works in the background, especially writing code in IL.

This is however what our META language is based on. We're using it as a starting point for code generation https://gamedev.net/forums/topic/711334-any-good-code-generation-library/5445152/​​ which is technically what was mentioned here already. Our solution is targeting visual scripting, that could barely belong into “scripting language” but has a different approach seriously.

The questions is what is the benefit of a scripting language vs. compiled code and this should be base for thinking about implementing scripts into a game engine. In the end, it doesn't make production easier, it makes it even harder as you have to give support for two or even more coding languages in your project and you'll have a compiler installed in your production environment anyways. From my experience, game designers even don't want to code most of the time so a visual scripting API is enough for them and this could anyways be translated into real source code when building the game. The benefits of a scripting language in my opinion is that you could have a far more rapid iteration because not everything needs to be recompiled but the script code and loaded dynamically, scripts could also be created dynamically during runtime; something like inline assembly for your engine.

So taking these two benefits of scripts into acount, the logical way to implement scipting languages in your engine is to have either some kind of JIT or a runtime that is able to work along a stream of compiled op-codes (like .NET IL for example). Performance might be quiet good, not as good as compiled C++ assembly, but good enough for a game. Op-codes are just flags which control the runtime flow, similar to assembly instructions on a CPU, with the difference that you have to code all the logic on your own, so popping values from the stack and pushing results for example. But you are also free to have special op-codes for some complex commands. An example for such a command is the .NET WriteLine op-code which performs a System.Console.WriteLine call providing a string.

Now to answer your question, I've already answered a similar question asked by @juliean some time ago in this thread https://www.gamedev.net/forums/topic/709730-parameter-pack-expansion-order-for-automatic-script-function-wrapper/5438008/​.​ A good way to have transition from scipt to C++ is to use a stack approach and pushing function arguments as you would usually do when calling any other function in code. You could simply create/register some delegate like struct to the C++ function, which already pops the desired arguments in the right order as I showed in my answer. Then your script just needs to pass the arguments on the stack properly and calls the op-code pointing to that delegate to have it perform the call for you. It should then push the result back to the stack and the call is done. There might however be a more efficient way of doing this but I can't imagine of one yet ?

Shaarigan said:
From what I learned over the past years, scripting languages may be both, easy to implement and simple

I have a somewhat shallow experience with Lua from a past project I had, this time I`m trying to get a better picture of how it all works behind the scenes. Thanks for taking the time to share from what you know about the topic.

why not just write your own a Config File Parser and have config file for various modules of your game …

btw i do wrote a config file parser and doing something with it for load params ... etc…

Because its not what OP asked for !?

Enzio599 said:
why not just write your own a Config File Parser and have config file for various modules of your game … btw i do wrote a config file parser and doing something with it for load params ... etc…

thanks for posting Enzio, though that`s not exactly what I was asking for, it is something that I could try in the future for sure.