[EDIT: I have changed the title of this topic since it the enhancements now include more than native method declarations.] Sorry it's taken me so long to get back to you about the native declarations. Between the holidays and recent events I haven't had much of a chance to reply. If you prefer I can merge the changes into the latest in source control to make it a little easier. I did however put together some information that will accompany the zip file (which you should expect in a day or so). Right now I need to update the test files and make a few minor changes but those won't take long.

[DRAFT] - Edited

Introduction:

	This discusses the enhancements made to AngelScript to support native
	function declarations, meta data, build interaction, and several other
	additions. Keep in mind that this is currently an experiment so every
	effort is being made to prevent dependencies between enhancements but
	allow them to be used as ancillary support if necessary. Only those
	enhancements thought mature enough to be bug free are included in the
	patch files.

	The current implementation is fully function but incomplete. There are
	several support components that I would like to add as well as beefing up
	the error reporting. The enhancements included with the patch files consist
	of the minimal requirements for testing and validation of current design
	goals. I expect that several aspects will change once I have received some
	feedback. The attached set of patch files includes the following
	enhancements:

		*	A minor addition to the language to provide meta data as ancilliary
			support for native declarations.
		*	meta data can be applied to interfaces, classes, methods, and
			functions.
		*	The introduction of the 'native' keyword.

	The following enhancements are not included:

		*	Easier import syntax with the ability to import all functions of
			the module. Includes support for meta data.

				import "Module Name";

				import "Module Name" {
					... function declarations ...
				}

		*	Alternate import semantics with meta data support.
		*	Native variable declarations.
		*	Addition of a 'deprecated' keyword.
		*	Allow access to meta data from within scripts.
		*	Built in management for native implementation (See below).


Language::The Native Keyword:

	The native keyword is applicable to function and class method declarations
	only. Native data is currently not supported. As in Java native
	declarations must include the 'native' keyword and do not have a body:

		native void function(int param);

	If an implementation for a function or class method cannot be found an
	error is issued. The native keyword is only supported in declarations
	appearing in scripts (e.q. only in scripts).

	

Language::Meta Data:

	When considering all of the aspects of managing native implementations I
	kept hitting one case I was unhappy about; there will be times you may not
	know the where the implementation is before compilation. I wanted an
	approach that had minimal language requirements, retained a familiar
	syntax, and was extensible without changing the parser for each host
	implementation. I also wanted to retain the ability to use meta data even
	if support for native declarations is not available. In order to meet those
	goals and maintain compatibility I chose a syntax similar to that of IDL
	attributes.

	Meta Data blocks are enclosed in brackets and precede interface, class,
	method, and function declarations. Within the data block is a comma
	seperated list of identifiers. These identifiers may optionally include a
	list of arguments simiar to that of a function call. Currently this only
	accepts constants as arguments (identifier, string, integer, etc). When
	parsing meta data all reserved words are interpreted as identifiers.
	Identifiers may also be used more than once.

	No assumptions about the content or validity of any meta data element are
	made during compile time. This is left up to the script host developer
	and only reports syntactical errors.

	Examples:

		// Tell the script host to set up a timer object to call this function
		// once every 1500ms.
		[onTimer(1500)] void OnTimer() {
			//	do something
		}

		// Tell the script host to call this function during initialization
		// indicating a dependency upon another initializer
		[initializer, depends(Initializer1,Initializer2)] void Initializer() {
			// Do something
		}

		// Tell the script host to call this function during initialization
		// This initializer required that the method Initializer2() be called.
		[initializer, depends(Initializer2)] void Initializer1() {
			// Do something
		}

		// Tell the script host to call this function during initialization
		[initializer] void Initializer2() {
			// Do something
		}

	The design has taken into consideration applying meta data to the 
	following script elements but currently does not:

		*	Variables.
		*	Function parameters.
		*	Modules.
		*	Modules using an alternate syntax.
		*	Package declarations (if every introduced)
		*	Import declarations

	These require some further investigation as they significantly impact the
	internals either in performance, scope of implementation, or actual
	usefulness. They also rely on some additional enhancements that are in the
	works.


Language::Meta Data::Recommendations:

	Although the implementation of the meta data components is intended to
	use any identifier it's always good to make some suggestions from the very
	beginning. The following is a list of recommendations:

		friends		-	One or more modules that can import the element.
		restricted	-	Specifies that the class, method, or function can only
					be called by the script host.
		string		-	Specifies a string.
		version		-	includes version information for the element.
		singleton	-	Specified that only one instance of a particular class
					may be instantiated at any given time.
		warning		-	Enable and disable specific warnings
		error		-	Issue specific error as warning


API Enhancements::Introduction:

	The API Enhancements are intended for management of native implementations
	and to provide access to meta data. In doing so several new interfaces
	have been introduced and minor changes were made to existing ones.
	
	In making changes to the current API there were several considerations that
	had to be made in order to maintain flexibility. I based many of the design
	requirements on several use cases revolving around extensibility and
	security. The primary use case for testing retrieval of native
	implementations has the following requirements:
	
		*	The implementation is located in a shared library.
		*	The name of the shared library is not known until compile time.
		*	The name of the import function is not known until compile time.
		*	Ability to allow or deny access to the shared library or function.
		*	Specify a file containing a digital signature
		*	Allow platform independent requirements
		*	Allow platform specific requirements
		*	Access to meta data to determine initialization function

	Given the above requirements the test script looks like the following:

		//	Platform dependent
		[
			library(win32, "cruntime.dll"),	
			library(linux, "cruntime.so"),
			function("_rand"),
			signature(file, "cruntime.sig")
		]
		native int rand1();

		//	Platform independent
		[
			library("cruntime"),	
			function("_rand"),
			signature(file, "cruntime.sig")
		]
		native int rand2();

		//	No meta data necessary
		native int rand3();


		[initializer]
		void callrandomfunctions()
		{
			rand1();
			rand2();
			rand3();
		}

	By nature of the meta data itself many of the requirements are already met.
	The declarations are generic enough that the script host developer
	maintains significant control over accepted directives (as will be
	discussed later). This is not to say additional semantics could not be
	introduced to provide a wider range of declarations especially for named
	arguments and other niceties.


API Enhancements::Batteries Not Included:

	There are several changes and additions pending on both feedback and
	whether the changes are added to the official distribution or not. Most of
	these changes involve internal compiler values. At this stage they are
	not required for testing and I didn't feel it necessary to start moving
	things around.


API Enhancements::Interfaces:

	The patch files include several new interfaces some of which are intended
	for accessing to internal compiler elements. The use of an object
	implementing of these new interfaces is explicitly temporary.

	New Interfaces:

		asIScriptEngineHost	-	Script engine host.
		asICustomData		-	Base class for custom data that can be assigned
						to a script engine element (object, function).
		asIMetaDataCollection	-	A collection of data elements.
		asIMetaDataEnumerator	-	An enumerator that can be used to step through
						all elements of a data element collection.
		asIMetaDataElement	-	A data element.
		asIMetaDataValue	-	Value applied to the data element. Possible
						data types are doubles, 64 bit integer, quoted
						string, or identifier.
		asIObjectType		-	Interface to an object representing an
						interface or class.
		asIScriptFunction	-	Interface to an object representing a global
						function or interface/class method.

		Of special note is asIObjectType and asIScriptFunction which expose
		internal objects. These have been included for simplicity and
		experimental use only and are not guaranteed to appear in the final set
		of patches.

		Other types such as asIModule and asIVariable are being considered and
		would include similar functionality to asIObjectType.

		
	Changed Interfaces:

		asIScriptEngine		-	Includes access to new engine components
						and basic native implementation management.
		asIScriptStruct		-	Now provides access to the associated
						asIObjectType.


API Enhancements::asIScriptEngine:

	The scripting engine provides basic management of native implementations
	(for advanced management see notes on asIScriptEngineHost). This is
	accomplished through two new API called:

		RegisterNativeMethod()
		RegisterNativeFunction()

	Each of these methods require at least a module name, function declaration and
	a pointer to the implementation. All native functions must be registered before
	building any script that declares them. The functionality for this is currently
	a WIP.


API Enhancements::asIScriptEngineHost:

	The asIScriptEngineHost interface provides a set of functionality the host
	developer must implement to fully support meta data and management of
	native implementations. Implementing this interface is optional. The
	functionality defined by asIScriptEngineHost falls into several categories.

	The first being the retrieval of native implementations. This includes the
	function pointer and calling convention. These methods are called before the
	compiler searches the list of registered implementations. This allows for
	late binding and enforcement of custom security policies. When the compiler
	retrieves a native implementation the following steps are taken:

		1.	Programatically check to supply an implementation or enforce a
			security policy (host callback).
		2.	Check the list of registered implementations (internal).
		2a.	Programatically check to enforce a security policy (host callback).

	The second group is used for validation of meta data elements including
	argument types. These methods have the option for issue an error or
	act on an error at a later time.

	The last group includes finalizing declarations. These methods are called
	after the creation of compiler elements for interfaces, classes, methods,
	and functions. Methods are also called after an element has been completely
	compiled. This allows custom data to be associated with those elements. The
	primary motivation for supporting custom data is to allow caching of object
	and function id's.

	Implementing a script engine host is optional. If a host is not provided
	default values to maintain backwards compatibility.


API Enhancements::asICustomData:

	asICustomData is simply a forward declaration for compatibility between C++
	and C environments. The asICustomData object can be associated with the
	following types:
	
		asIObjectType
		asIScriptFunction
		asIModule (if implemented)
		asIVariable (if implemented)

	This data is defined and managed by the script host developer. This
	includes full memory management. This approach was taken to allow custom
	data objects to be exposed to the scripting engine including reference
	management.


API Enhancements::Special notes for native declarations:

	Every effort has been made to allow the importing, retrieval, and use of
	functionality implemented through native declarations. This includes the
	ability to call a native declaration using a script context. Note that this
	is a WIP and only minimal support and error reporting is provided. Every
	effort is also being made to apply this update to calling functions
	registered through asIScriptEngine.


API Enhancements::Meta Data

	The interfaces provided to support meta data include access to all elements
	and data associated with a particular object type. This includes the
	ability to retrieve the meta data by index or by implementing an
	enumeration object. Meta Data is currently READ-ONLY and is accessible
	by only the script host.
	

API Enhancements::Bytecode Streams:

	All enhancements are fully supported when saving and restoring a module
	with a bytecode stream. To fully support enumerating base interface
	meta data the restore functionality now pulls in the asCObjectType for
	all specified interfaces. This is considered a WIP as it does not
	report errors and and it's quite likely all parents should be pulled
	in immediately rather than after processing the class or interface. At the
	very least I see that as a possible requirement to support more than one
	class level of inheritance.


Test Code:

	Included are several new files for testing and validation of the included
	enhancements. Of special note are the following two areas:

	Meta Data:

		*	Currently tests only for success
		*	Demonstrates implementing a script host
		*	Demonstrates processing meta data and assigning it to a custom
			data object.
		*	Demonstrates accessing attributes from within the script host
			including enumerating the attributes of all implemented interfaces
			in a class.


	Native Functionality:

		*	Currently tests only for success
		*	Demonstrates implementing a script host
		*	Demonstrates processing meta data and assigning it to a custom
			data object.
		*	Demonstrates accessing attributes from within the script host.
		*	Demonstrates backwards compatibility with imports.
		*	Demonstrates backwards compatibility with script host.
		*	Demonstrates calling a native function from within a script.
		*	Demonstrates calling a native function from the host application.
		*	Demonstrates calling a into a script from within a native function.
		*	Demonstrates accessing object data from within a native method.

[Edited by - Digital_Asphyxia on February 3, 2008 8:45:42 PM]

I find this very interesting. But I'm not sure I want to include this in the core library. I believe this is something most users of the script library won't need or want to use, thus I do not want to burden them with this extra code needlessly.

I think that most of this should be possible to implement as an add-on rather than built-in feature. The imports of native functions for example could probably be implemented in a preprocessing step, where the application reads the import tags and registers the functions requested for use by the compiler.

The meta data tags will probably need some kind of hooks in the script library itself, so that the library simply stores the meta data and then allow the application to scan this the way it wants.

I have sent the source files so if you haven't received them yet please let me know.

Quote: Original post by WitchLord
I find this very interesting. But I'm not sure I want to include this in the core library.

[snip]

I think that most of this should be possible to implement as an add-on rather than built-in feature.

In some respects this was taken into account with the inclusion of the optional script host object. For acquisition of native implementation a callback to the host object is done. Likewise the allocation, release, and assignment of custom user data is handled through a callback. This leaves a lot of the management up to the host developer and I expect to have default implementations available through the addon folder.

Quote: The imports of native functions for example could probably be implemented in a preprocessing step, where the application reads the import tags and registers the functions requested for use by the compiler.

I had considered that and it works for functions but wouldn't work well for class methods. For class methods the closest I could come to was implement the method that simply called a native (or other) function. The changes to support only native declarations are actually quite minimal and are optional based on as_config.h.

Quote: The meta data tags will probably need some kind of hooks in the script library itself, so that the library simply stores the meta data and then allow the application to scan this the way it wants.

This is currently how it is implemented. You can retrieve a meta data collection from any function or object type and then access each element via index or through an enumerated callback. Callbacks are also made for functions, interfaces, classes, and class methods each time one has been declared and then once it has been compiled. Right now I'm not satisfied with the callbacks at the moment and will probably be making a few changes for greater flexibility and easier support for C.

I haven't found the time to look at this yet, but I haven't forgotten about it. I'll let you know when I have looked closer at the code.

Quote: Original post by WitchLord
I haven't found the time to look at this yet, but I haven't forgotten about it. I'll let you know when I have looked closer at the code.

No problem. I'm in no hurry as I've got some clean up to do and need to get enumerations merged into my experimental branch.

Quick update. I have made significant changes and the patches will now include the following features:

    *   IDL like attributes    *   Enumerations.    *   Simple typedef's (applies to integers, reals, and booleans)    *   Alternate method of importing using 'extern "module" {}' like declarations.    *   API and script level support for native declarations in scripts.    *   API and script level support for hybrid object types.    *   Ability to import classes and interfaces from other modules.        Includes support for typedef like aliasing when declaring the        import statement.    *   Patch to allow objects shared across modules to access the global        variables declared in the module the object belongs to.

Most of these features are not supported in save/restore yet as I'm looking at the issue of dealing with dependencies during restore. I'm hoping to have a public release of the patches in the next few days.