Here is nice document that lists ABI for various compilers (including MSVC and GCC) for various platforms (including 16-bit, 32-bit and 64-bit PC): http://www.agner.org...conventions.pdf (page 16)

Thanks a lot for this article. This clearly explains how the classes are handled.

Yes it seems to use the SSE registers for return value. I found a function attribute sseregparm for GCC, but unfortunately I can't find a complementary attribute to it.

Indeed. As the ABI said, the structure is returned in the XMM0:XMM1 registers rather than the RAX:RDX. I'm not sure why the structure is not returned only in XMM0 as it would fit nicely in there, but the other article that Martins posted describes that only the lower 64bits of the XMM registers are used.


Anyway, I may be able to add a special case for this kind of structure, say by adding the flag asOBJ_APP_CLASS_ALLFLOATS, or something like that. It would then be possible to have the code in as_callfunc_x64_gcc.cpp use that flag to handle the type correctly. However, this will have to wait for a future version, unless you feel up to implementing it yourself.

For now I'll just make sure the library properly checks for the return of simple types and gives an error since there is no way code can predict the way it should be handled.


The unfortunate final verdict is that on Linux 64bit you will only be able to return this type by value in registered functions if you use the asCALL_GENERIC calling convention via function wrappers. However, don't feel too bad about this, because I believe the asCALL_GENERIC calling convention is actually faster than the native calling convention on Linux 64bit, as there are less dynamic decisions that the library has to make.


Thanks for the help in clearing out the doubts.


Regards,

Andreas

asOBJ_APP_CLASS_ALLFLOATS doesn't sound like a particularly good idea but it's definitely better than the other option: we've recently dropped support for generic calling convention from our AS classes completely. it was just too much of a burden.. doubling the amount of code for no practical reason.

Unfortunately there is no other way for AngelScript to be able to support this type on Linux 64bit. The code in as_callfunc_x64_gcc.cpp needs to know that the gcc compiler would classify the type as floats and return it in XMM0:XMM1 instead of RAX:RDX.

The information may be given in a different form, but right now I think it is the way that will give the least amount to work. With the flag it should be possible to quite easily change the code to retrieve the return value from XMM0:XMM1 similarly to how GetReturnedFloat() is done.

Since you do not want to use the generic calling convention (even with autowrappers) and for some reason cannot change the type to include a copy constructor, perhaps you would be interested in adding the support for ALLFLOATS?

Unfortunately there is no other way for AngelScript to be able to support this type on Linux 64bit. The code in as_callfunc_x64_gcc.cpp needs to know that the gcc compiler would classify the type as floats and return it in XMM0:XMM1 instead of RAX:RDX.

The information may be given in a different form, but right now I think it is the way that will give the least amount to work. With the flag it should be possible to quite easily change the code to retrieve the return value from XMM0:XMM1 similarly to how GetReturnedFloat() is done.

Since you do not want to use the generic calling convention (even with autowrappers) and for some reason cannot change the type to include a copy constructor, perhaps you would be interested in adding the support for ALLFLOATS?

Vector is in C side so no constructors there. In C++ this would be no problem since returning by value works with explicit copy constructors.
But I was able to, at least temporarily, fix the issue by unioning the 3 float components with 3 integer components

Just out of curiousity, what is ithat you're doing in pure C? Why have you chosen to use C instead of C++?

C is the language our game was originally written in. Converting to C++ would simply be a waste of time for no real gain..

You don't need to convert it. C++ is supposed to be backwards compatible with C. You just need to compile the C code as if it was C++, then you'll be able to add the copy constructor to the vec3 struct.

Anyway, I decided to try implement the ALLFLOATS myself. It should hopefully be only a few minor changes that I can do without the need for debugging.

You don't need to convert it.

That's another option but it's more of a half-assed effort I mean, is it worth the trouble of altering project and makefiles, compiler settings, etc just to add one(!) constructor to a single(!) class?

Done!

I added the flags asOBJ_APP_CLASS_ALLINTS and asOBJ_APP_CLASS_ALLFLOATS, to allow these two alternatives to work properly on Linux 64bit. The tests I included in test_feature, seems to work properly on the buildbot so hopefully it should work just fine for you as well.

You'll need revision 980 if you wish to try this.

The only change to your code is that the asvec3 type should be registered with the flags asOBJ_VALUE | asOBJ_POD | asOBJ_APP_CLASS | asOBJ_APP_CLASS_ALLFLOATS.

Don't forget to remove the union with the ints, that were added as a work around. If these are there then the flag must be asOBJ_APP_CLASS_ALLINTS instead.


Let me know how it works.