Hello community,

I want to raise our data driven build tooling pipelines to the next level and therefore, thought about something which is already pretty standard in gameplay programming and shader design; Visual Scripting. I have seen a lot of graph based visual solutions for almost everything but I haven't seen any visual scriptable build/render or whatever pipeline so far in the major game engines. So I thought about creating a visual scripting package for our SDK, in order to allow our build tool to have it's data driven pipeline visually scripted by the user as well. And maybe also everything else in our game engine but let's start at the bottom first. So I'm on the road of writing a solution which allows to define nodes in built assemblies but also code and translate this into a compact but still human readable text format to have a “compiler” generate language specific source code in the end. For this I'm looking for any libraries and/or projects which make code generation easier.

Does anyone know any good (open source) C# libraries which offer code generation for C# and/or C++ or a generic solution without a specific output language?

For those interested in this topic as well, we haven't found anything satisfying and so decided to go for a custom solution. For this we created a code meta language, a language used to describe a program flow in some kind of pseudo code like expressions. A parser is creating a code graph from those expressions and can be passed to code generation backends in order to turn the code graph into language dependent source code.

A first release will be made on our GitHub soon

All of them I've used were custom tools. Generally they saved the data to XML, and either exported to a custom language or had the main system import the XML and parse it directly.

What we're using it for primary is to offer scriptable e.g. visually scriptable pipelines for everything from building the project up to the render system. I like the idea of going forward not just for design tools but also some convenience for programmers as well.

My baddest experience was the C#/NodeJS “build pipeline” in my former job, that had some components written in batch script. There was an error one day in the script, a typical weekend commit, which then broke everything and debugging a 500 line batch script isn't that fun anyways. I know there are other system, which we also used partially, but all of them requiere maintaining plain scripts. That's why I really quickly decided for our hobby project to go for a solution that has the really minimum of configuration in a JSON file and everything else is implicit logic.

However, this is why I wrote our Hecate tool which right now is a really good solution if you follow the rules. The next step is to offer visually scriptable build pipeline with a bunch of predefined nodes and an extensible node library. And we decided to go for our Meta Language, which is plain simplified way of writing code. Something like

 if   eql x 1
.call "System.Console.WriteLine" "Hello World"

is parsed into a node code graph of different nodes which offer some error checking. Except for the function identifier in this example, everything is plain generic and could be transposed into C# to be added to our C# based tools and/or C++ to be integrated into the engine code base or whatever language you want it for. Additional error checking needs to be made from the specific code generator when it reads the graph.

We will then have a bunch of JSON files which build the extensible visual node library. Those files define how many and which input connectors a node has, the output connectors and they also allow for defining Meta. Those visual node graphs are then read and their Meta code is collected and chained into the input stream for the Meta Language parser to form the final node graph which is then passed to the code generator.

Our Alchemy text processor already allows to preprocess text streams and offers not only macros but also import derectives that work somehow similar to C++ includes so we don't have to add this to the language. (We make use of it for JSON files as well, as Alchemy also has support for comments)

As Hecate makes use of NPM and theirfore allows for package.json files being present in a code module, I seized the peerDependencies field to have Hecate look for plugin modules when building. So one can specify a code module on a known path so Hecate is able to lookup the package and then it automatically adds it to the build even if it isn't specified somewhere but the package specifies one of the modules currently contained in the build. I wanted to use this for having our visual editing tool spawn a custom Hecate instance to hook into the build pipeline but it also works for code generated custom build steps.

Long story short ?

There are many, many, options in this area – the question is what you really need.

Most “scripting” works fine by just interpreting an abstract syntax tree graph through recursive walk. That necessarily has to be specific to the particular implementation, and thus “custom," so there's not a whole lot of libraries to do it (other than parser generators for the AST.) To save and load these “scripts,” use standard graph serialization methods (serializing to JSON, XML, whatever.)

If you want high performance (e g, for texture processing algorithms, geometry processing, and so on,) then you'll want a code generation platform. You could use Graal, or LLVM, or WasmEdge, or one of tons of other execution runtimes that come with JIT compilers.

hplus0603 said:
the question is what you really need

I wrote in my previous post what we use it for and how we decided our solution should work for us