I suppose this is the second 'official' release of the preprocessor. I would appreciate it if the angelscript website was updated. The source can be snagged at http://www.omnisu.com/preprocessor.zip Changes include - New method for loading scripts from file. You will need to supply a class derived from Preprocessor::FileSource. The class has the interface

		virtual bool LoadFile(const std::string& filename, std::vector<char>&)

which must load the file 'filename' into the vector. It returns true if sucessful, false if not. - Worse error reporting. Sorry, it is not a priority. Specifically, the error reporting by AngelScript will be flawed because the preprocessor no longer directly calls AngelScript. It is your responsibility to pass the finished code stream to AngelScript, which makes it impossible to extract usefull line and file information. Additionally, this change means you must also pass a functor to receive the code stream. The class Preprocessor::VectorOutStream is already implemented to serve this purpose. - A much more stable recursive descent tokenizer to replace the FSM one. I actually had no idea what a recursive decent model was until I looked into the angelscript compiler code. It is, IMO, a much better design. And it was a lot easier to write, too. - Fixed the hexadecimal constant bug. They are now correctly identified as single tokens, instead of two seperate tokens. - The other reported bug which I have not investigated of leaking file handles is no longer an issue as the preprocessor no longer opens files itself. Things I'm working on - My current project would greatly benefit from the ability to preprocess code already in memory.

Thanks, for the new release Deyja. I'm glad to see that you're still improving on this tool.

I'll update the AngelScript site as soon as I can. Hopefully later in the day.

Sounds nice. I'm going to have a look at it.

The fix i did for the hexadecimal bug was rather hackish.

One way you can fix the error codes (i had a giant trouble with it) problem with line numbers being off is storing the starting point of the script and just counting the number of CRLF's you run into.

I then modified the as error code output function to place the error line# and column into a vector<pair<int,int> > and the actual error code into a vector<string> (i suppose i could have gotten more creative and done a vector< pair<pair<int,int>, string > >) (this made it so that I did not have to parse the actual error string for the line#/column but is not required )and then do some calculations with the error line and combine them back to make a correct judgement of where the error is. With a little more work it would be quite reasonable to be able to say "Error line 57, 5, file 'myfile.txt': cannot convert int to MyStruct" by keeping track of beginnning and ending line #'s of a script that was included into the file.

The #defines on the other hand would pose a different situation should they expand to more than 1 line, but it would be reasonable to also account for that.

The simplest solution to line numbers in errors is of course the way I originally did it. Just pass each file to the script engine as a seperate script section. This unfortunatly has it's own drawbacks. Mainly, it means that include guards are no longer adequate to prevent multiple inclusion. The file might expand to nothing the second time, but then it replaces the script section that is already registered. Additionally, I need to parse the filename into a true absolute path. Currently, if you use the ../ directory name, it stays in the filename as in. You could end up with something like scripts/level1/../general/scriptname.asc. I didn't feel like tackling the problem of actually parsing filenames so I didn't bother. Also keep in mind that I don't want to couple this that tightly with angelscript. As it stands it can preprocess any language.

What I propose instead is to simply save the preprocessed script and examine it instead of the original scripts. This is easily accomplished in the current system by writing a Preprocessor::OutStream that saves to disc, or by just grabbing the contents of the Preprocessor::VectorOutStream you will probably be using otherwise.

Another thing that came up is that it does not support the syntax 0.000f, with the trailing f to signify a float constant instead of a double constant. Fixing this will neccesitate changing how the numbers are tokenized so the problem of 0.0.0.0 and such being a valid number token will also be fixed.

Here is a quick example of how to work with the new interface. I'm working on some docs for the preprocessor, and am going to add it to my webpage at www.omnisu.com Be warned that I am one of the worse html writers ever and that that simple webpage represents the pinnacle of my design adventures. Any help is appreciated.

The static cast is only neccessary to silence a compiler warning. VectorOutStream returns it's size as a size_t, but AddScriptSection wants an int. size_t as of vc2003 (IIRC) is a 64bit type.
You will of course want to add proper error checking.

Apparently I am stupid. I neglected to recall that hex constants can also include the characters ABCDEF. I've fixed it. Also; it now properly handles the #.##f format.