I can't do that because all my script exported functions are in native conventions only. Refactoring or adding the support for the generic calling convention will take me quite a long time because there are a lot of them.
However, I can try to help as much as I can with compiler options, code fixes. Tell me if you prefer me to try things directly with you (fix/compile/test etc.).
EDIT : also it already seems to handle the case for big endian in as_config.h : #if _XBOX_VER >= 200
// 360 uses a Xenon processor (which is a modified 64bit PPC)
#define AS_XBOX360
#define AS_XENON
#define AS_BIG_ENDIAN
#else
I'll see if I can reproduce the problem on my Mac where I can run the rosetta simulator for a PPC processor. If the problem is not in the as_callfunc_xenon.cpp code I should be able to find it.
Okay, let me know if you need more information on the crashes I have like specific values of variables. I had crashes happen in other cases of the ExecuteNext switch than the one I put in example.
EDIT : Maybe there were modifications to as_callfunc_ppc.cpp that were not applied to as_callfunc_xenon.cpp (the comments say it was based on ppc). I can also give you some hardware information if necessary.
I couldn't reproduce the problem on my mac with ppc, but I took a closer look at as_callfunc_xenon.cpp and believe I discovered the problem. When the native function was returning a bool as_callfunc_xenon.cpp did an endian swap. The problem was that it didn't check if the bool type was returned by value or by reference, so it messed up the reference to the bool returned by the index operator.
As I was reviewing the code I also fixed a couple of other bugs, such as the code not handling int8 and int16 types correctly. There are also a couple of more bugs that I saw but didn't fix, because it would require more testing, e.g. the code doesn't support int64 parameters, nor passing objects by value.
Obviously I haven't been able to test these changes, so I may have broken some things while changing the code. Please let me know if everything is working as before and if the bug you reported has been fixed too.
As the problem occured in the compiler it shouldn't really matter how you have registered the various types.
Have you checked the compiler settings you use on PS3? Maybe there is something that goes wrong if you use high level of optimizations for the angelscript library? Or maybe there is a limit to the number of recursive calls on the stack on the PS3?
Is there anything else you can give me on this problem?
Sorry for the delay, we were a little busy. The problem came from a too small stack size for our threads that were loading the script. It is now resolved.