Do any existing contributors have any tips (perhaps there's something existing online or in the documentation) for getting started with understanding how AS works under the hood? Does it go something like this (for the most basic case)?

1. tokenize script source
2. create AST
3. compile into byte code
4. (byte code optimization step)
5. Execute each byte-code instruction [ asCContext::ExecuteNext() ]

Thank you very much

On a very high level the compilation and execution of scripts works as you listed.

There is no documentation for the code. I do try to keep the code comments as clear as possible though, and the code is broken down by the logical modules. Each file normally only holds one class, with a similar name as the file itself, so it should be easy to localize in which file to look for something.

Here's a brief explanation of what the principal modules are:

as_tokenizer.cpp has the logic for identifying individual tokens in the source code. It uses the definitions in as_tokendef.h for that.

as_parser.cpp has the logic for interpreting the sequence of tokens into declarations, statements, etc (i.e. building the AST)

as_builder.cpp orchestrates the script compilation. it is responsible for passing the source code to the parser, identify variables, types, and functions that has been declared, and then invoke the compiler to compile the bytecode for the functions

as_compiler.cpp has the logic for compiling the bytecode based on the AST (this is by far the most complex piece of the library)

as_bytecode.cpp holds the intermediate structure that the compiler creates and also the logic for doing post-compilation bytecode optimizations

as_restore.cpp has the logic for saving and loading already compiled bytecode (this is the second most complex piece of the library)

as_module.cpp is the structure where the final compiled script is stored

as_context.cpp is the virtual machine that executes the bytecode

as_engine.cpp is the central piece that holds everything together, and where the application registered interface is stored :)

The SVN has a test_features project that I use for testing. Every time anything is changed I compile and execute this to verify the result. This is to avoid unexpected side-effects (of course, sometimes they slip through anyway ). With every bug I fix I add new test cases to this project so the same bugs shouldn't reoccur later on.

Every time I implement something new, I usually start by thinking about what needs to be changed in the code (usually by marking the impacts with // TODO: in the code itself). Once I have a fair idea of the impacts I write the test cases with the expected results, and only after this do I start the actual coding of the new feature. This way I can make incremental changes without breaking everything else.

Thank you for the explanation.

In as_compiler.cpp, it looks like there are some debug-only byteCode.DebugOutput() calls which output info about individual script-functions, etc. into individual files (?).

If we temporarily ignore byte code optimization.. is there any way of outputting a 'disassembly' per module? Basically I compile foo.as and, for debug purposes, I want a single file output with my script-lines as comments and the generated byte code below them.

Example input:

void main()
{
    int variable = 2 + 2;

    print("variable = " + variable);
}

Outputs:

; void main()
; {
;    int variable = 2 + 2;
... bytecode here that the above line produced
;    print("variable = " + variable);
... bytecode here that the above line produced
}

If this is not currently available, how would you rate the difficulty?

Thank you.

You can turn off bytecode optimizations with SetEngineProperty(asEP_OPTIMIZE_BYTECODE, 0); Doing this is useful for verifying that the compiler is producing the proper bytecode sequences.

The debug output of the bytecode already is a 'disassembly' of the functions, and it contains the line numbers to the original source code. You'll find the files in the AS_DEBUG directory which is created in the current working path where the application is executed. If you clean all the files in this directory before compiling the module you will then see exactly what is produced for that module.

To turn on the debug output compile the library with AS_DEBUG defined.

I don't see a need to change the way the debug output is currently generated. The change would complicate the code more than the benefits it would bring.

I got something close to what I had in mind working locally (just to help me learn).

This script

float calc(float a, float b)
{
    // Print the value that we received
    Print("Received: " + a + ", " + b + "\n");

    // Print the current system time
    Print("System has been running for " + GetSystemTime()/1000.0 + " seconds\n");

    // Do the calculation and return the value to the application
    float c = a * b;

    return c;
}

Turns into this:

float calc(float, float)

Temps: 1, 3, 4, 5, 6, 8, 9

Variables:
 000: float a
 -001: float b
 010: float c
 004: string {noname}
 005: string {noname}
 006: string {noname}
 009: string {noname}


- 4,5 -    Print("Received: " + a + ", " + b + "\n");
    0  10 *    VarDecl  0
    0  10 *    VarDecl  1
    0  10 *    SUSPEND
    1  10 *    STR      2         (l:1 s:"
")
    2  12 *    CALLSYS  33           (const string& _string_factory_(const int, const uint8&in))
    4  10 *    PshRPtr
    5  11 *    PSF      v9
    6  12 *    CALLSYS  35           (string::string(const string&in))
    8  10 *    ObjInfo  v9, 1
    8  10 *    VAR      v9
    9  11 *    PSF      v6
   10  12 *    CpyVtoV4 v1, v-1
   12  12 *    fTOd     v8, v1
   14  12 *    PshV8    v8
   15  14 *    PSF      v4
   16  15 *    STR      1         (l:2 s:", ")
   17  17 *    CALLSYS  33           (const string& _string_factory_(const int, const uint8&in))
   19  15 *    PshRPtr
   20  16 *    PSF      v5
   21  17 *    CALLSYS  35           (string::string(const string&in))
   23  15 *    ObjInfo  v5, 1
   23  15 *    VAR      v5
   24  16 *    PSF      v6
   25  17 *    CpyVtoV4 v1, v0
   27  17 *    fTOd     v3, v1
   29  17 *    PshV8    v3
   30  19 *    PSF      v4
   31  20 *    STR      0         (l:10 s:"Received: ")
   32  22 *    CALLSYS  33           (const string& _string_factory_(const int, const uint8&in))
   34  20 *    PshRPtr
   35  21 *    CALLSYS  51           (string string::opAdd(double) const)
   37  17 *    ObjInfo  v4, 1
   37  17 *    PSF      v4
   38  18 *    GETREF   2
   39  18 *    CALLSYS  41           (string string::opAdd(const string&in) const)
   41  15 *    ObjInfo  v6, 1
   41  15 *    PSF      v5
   42  16 *    CALLSYS  36           (string::~string())
   44  15 *    ObjInfo  v5, 0
   44  15 *    PSF      v6
   45  16 *    PSF      v4
   46  17 *    CALLSYS  36           (string::~string())
   48  16 *    ObjInfo  v4, 0
   48  16 *    CALLSYS  51           (string string::opAdd(double) const)
   50  12 *    ObjInfo  v4, 1
   50  12 *    PSF      v4
   51  13 *    PSF      v6
   52  14 *    CALLSYS  36           (string::~string())
   54  13 *    ObjInfo  v6, 0
   54  13 *    GETREF   2
   55  13 *    CALLSYS  41           (string string::opAdd(const string&in) const)
   57  10 *    ObjInfo  v6, 1
   57  10 *    PSF      v9
   58  11 *    CALLSYS  36           (string::~string())
   60  10 *    ObjInfo  v9, 0
   60  10 *    PSF      v6
   61  11 *    PSF      v4
   62  12 *    CALLSYS  36           (string::~string())
   64  11 *    ObjInfo  v4, 0
   64  11 *    PopPtr
   65  10 *    PSF      v6
   66  11 *    CALLSYS  72           (void Print(string&in))
   68  10 *    PSF      v6
   69  11 *    CALLSYS  36           (string::~string())
- 7,5 -    Print("System has been running for " + GetSystemTime()/1000.0 + " seconds\n");
   71  10 *    ObjInfo  v6, 0
   71  10 *    SUSPEND
   72  10 *    STR      4         (l:9 s:" seconds
")
   73  12 *    CALLSYS  33           (const string& _string_factory_(const int, const uint8&in))
   75  10 *    PshRPtr
   76  11 *    PSF      v5
   77  12 *    CALLSYS  35           (string::string(const string&in))
   79  10 *    ObjInfo  v5, 1
   79  10 *    VAR      v5
   80  11 *    PSF      v4
   81  12 *    CALLSYS  73           (uint GetSystemTime())
   83  12 *    CpyRtoV4 v1
   84  12 *    uTOd     v3, v1
   86  12 *    SetV8    v8, 0x408f400000000000           (i:4652007308841189376, f:1000)
   89  12 *    DIVd     v3, v3, v8
   91  12 *    PshV8    v3
   92  14 *    PSF      v6
   93  15 *    STR      3         (l:28 s:"System has")
   94  17 *    CALLSYS  33           (const string& _string_factory_(const int, const uint8&in))
   96  15 *    PshRPtr
   97  16 *    CALLSYS  51           (string string::opAdd(double) const)
   99  12 *    ObjInfo  v6, 1
   99  12 *    PSF      v6
  100  13 *    GETREF   2
  101  13 *    CALLSYS  41           (string string::opAdd(const string&in) const)
  103  10 *    ObjInfo  v4, 1
  103  10 *    PSF      v5
  104  11 *    CALLSYS  36           (string::~string())
  106  10 *    ObjInfo  v5, 0
  106  10 *    PSF      v4
  107  11 *    PSF      v6
  108  12 *    CALLSYS  36           (string::~string())
  110  11 *    ObjInfo  v6, 0
  110  11 *    PopPtr
  111  10 *    PSF      v4
  112  11 *    CALLSYS  72           (void Print(string&in))
  114  10 *    PSF      v4
  115  11 *    CALLSYS  36           (string::~string())
- 10,5 -    float c = a * b;
  117  10 *    ObjInfo  v4, 0
  117  10 *    SUSPEND
  118  10 *    VarDecl  2
  118  10 *    MULf     v10, v0, v-1
- 12,2 -    return c;
  120  10 *    SUSPEND
  121  10 * {
  121  10 * }
  121  10 *    CpyVtoR4 v10
  122  10 * 0:
  122  10 *    RET      2