(some back-story) After my current project got bigger, I decided to split script files across different modules, like one global module 'session' for everything that is permanent for entire time program is run, 'world' module for map/level specific scripting and other compiled-on-demand modules for dynamic objects (ie. for weapons, enemies). Soon I realized it won't work, since many objects need to query game state form 'session' module in order to work. 'shared' keyword come in handy here, but it requires me to append almost every 'session' file whenever I compile new (dependent) module (and doesn't work for global variables yet). Another solution would be iterative module compilation (I remember someone mentioned this some time ago), with adding new script files on-demand to a module. This would effectively defer loading script files for later time, when they're needed. (again, not supported yet)

My idea is to create something between current implementation and iterative compilation, similar to DLL system:

• When module is compiled, if something is declared as 'shared', store any additional data required for subsequent compilation
• Introduce new keyword 'using', in global statement like: using MyModule;
• If module is using another one, it gains access to any shared information stored on that module (and any information from modules it is using)
• If module we depend on is not found/compiled, emit compile error (and stop)
• Every time module is used by another one, increase some usage counter (Modules have to keep track on what they depend on, to decrease this usage counter when discarded)
• If module has usage counter > 0, then it cannot be discarded or recompiled (so it cannot be discarded until all depended modules are)
  Such method would reduce number of compiled files (no redundancy) and also guarantee that shared global variables are always valid.
  It would require new 'using' (or similar) keyword, and some structures (and possibly interface methods) to keep track of dependencies.
  
  So, what do you think?

It is a good idea. I've been thinking along similar lines myself. Until now I've been mostly thinking that a script that wants to use another would simply do a forward declaration, e.g. 'shared class MySharedClass;' and would automatically get the implementation that way, if the class is already implemented and shared by another module.


However, before I get to that I need to decide how to protect the security of the application, so a malicious script cannot simply 'share' memory in other modules without the application knowing it and possibly screw up the whole application. Sharing classes and even functions isn't a problem, but it becomes an issue when sharing the global variables.

It is a good idea. I've been thinking along similar lines myself. Until now I've been mostly thinking that a script that wants to use another would simply do a forward declaration, e.g. 'shared class MySharedClass;' and would automatically get the implementation that way, if the class is already implemented and shared by another module.


However, before I get to that I need to decide how to protect the security of the application, so a malicious script cannot simply 'share' memory in other modules without the application knowing it and possibly screw up the whole application. Sharing classes and even functions isn't a problem, but it becomes an issue when sharing the global variables.

Dumb idea, but maybe allow them to be decorated with the 'shared' keyword too? Alternately, does it even make sense to share globals? I don't think forcing them to go through a virtual property or some such is that drastic an imposition, and could conceivably improve security.

EDIT: Also, if you're doing other things I'd be very happy to throw some development effort this way as this problem's been something of a thorn in my side as of late too. Implementation ideas welcome, I'm still banging around with the module system.

They will be decorated by 'shared'. A 'shared' class or function will only be able to access global variables that are also 'shared'.

The problem is that the way that the 'sharing' is done is that any module that declare the same 'shared' entity will always 'share'. There is currently no way to say that module A cannot share entities with module B, but can share with module C.

While not an immensely huge problem I do want to provide ways for the application to say that for example an entity meant for GUI scripting is not supposed to be shared with AI scripting.

The way I'm thinking about giving this control to the application is through namespaces, and then allow the application to set access masks on specific namespaces. If a module doesn't have access to specific namespace it will simply not be able to share entitites in that namespace.

This would also fit nicely with behc's suggestion. A script could for example say 'using namespace X;' and all shared entities in that namespace will automatically be made available, without having to explicitly name each of them.

Namespaces would be very good solution. It would allow greater granularity of what we want to share, without forcing you to write tons of declaration for every single class. Also namespace could default to module name if not explicitly declared. Technical question: Could one namespace be declared over several modules or should they have hierarchical structure with each module having its own global one?

About global variables, can you explain a little more about security threat it poses? I thought that as long as we can guarantee variable is not destroyed it is safe to use it directly? On the other hand, if it's destroyed (module discarded), does accessing through a function or getter/setter help in any way?

[color="#1C2837"]

They will be decorated by 'shared'. A 'shared' class or function will only be able to access global variables that are also 'shared'.

[/quote]
Is it really necessary to limit 'shared' functions to work on 'shared' globals only? That would defeat the purpose of having public interface working on private data because that data would have to be shared too. As long as all argument types are known (and not module specific), importing module shouldn't bother what is inside function body. (I think application side registered function pointers can already cross module boundary without any restrictions)

A shared entity is exactly that, shared, in all aspects. The modules that share the same entities actually refer to the same bytecode, and global variables when that is implemented. It is not meant for exposing a public interface that accesses private members behind the scene. The necessity of a shared function/object only accessing other shared entities, including global variables comes from the fact that there is no specific order in who will compile the code first. The result should be the same regardless what module is compiled first.

With the feature 'using' (if we can call it that) that you suggested, there would be a specific order that the modules need to be compiled. An in this case one can imagine having public/private interfaces like you mentioned. It would have to be called something different than 'shared' though. Perhaps the keyword 'export' or even 'public' can be introduced for this purpose.

If I'm going to use namespaces to control access, then the namespace has to be shared across modules as well. Though, obviously if the code isn't shared then code declared in two different modules but in the same namespace won't interfere with one another.

The security issue is not about potentially crashing the application, though you do have to be extra careful with shared entities if you use scripting from multiple threads. The security issue is more about unwantedly exposing information to scripts that weren't meant to have access to the information. Cheaters are just as creative as hackers in finding vulnerabilities in the games to exploit, so you will definitely want to have full control over what the users have access to.

Oh, now I get it. I think I was trying too hard to fit 'shared' system into my c++ habits. I still think that 'shared' globals are important feature though. Without them 'shared' functions could operate only on their arguments, effectively making them normal functions with little memory saving. But maybe this part would be better suited for 'public'/'using' system?

As for cheaters, they always find their way in, no code run locally is safe, so I wouldn't bother much about that aspect.

It's true, cheaters have a knack of finding the holes, but that doesn't mean we should just ignore potential security issues, especially when they are known and can be fixed.