It has been my default assumption that I may have done something wrong. After a few days of looking into the issue I came at an impass that might need info from people who are more familiar.

The issue is failrly simple. I have a registered reference type, which in the script I am calling a method on. In c++ the code asserts as the pointer of the object is not correct. Below is the information I've gathered about the crash.

System: Mac Os X 10.8.2

IDE: Xcode 4

Achritecture: 64-bit

Compiler: Apple LLVM 4.2 (default) also occurs with (LLVM GCC 4.2)

Config: Appears in both Debug and Release builds

AngelScript: 2.26.1

- The issue appears to be related to functions which return a specific registered value type (Vector)

- Other method calls on the same object works fine

- Multiple object types suffer from this crash

- Occurs with both direct method calls and method wrapper functions

- The pointer to the object stored in angelcode is correct

- I have also tracked the object pointer up until the assebly code in x64_CallFunction code, and it appears to be fine until then

- The same codebase works fine on Windows VS2010. I'm currenting porting the code to OSX.

- I've attempted to re-create the issue with a new class value type and new methods, but those work fine.

- Issues also occurs if I use the "LLVM GCC 4.2" compiler

For reference, example configuration of the classes in question. (not full config) Again, this code works fine on the Windows side so I think the issue is deeper.

//This is the suspect value type
ret = engine->RegisterObjectType("Vector", sizeof(GtVector), asOBJ_VALUE | asOBJ_APP_CLASS_CDAK); assert( ret >= 0 );

//Example of object method that fails
ret = engine->RegisterObjectType("Entity", 0, asOBJ_REF); assert( ret >= 0 );
ret = engine->RegisterObjectMethod("Entity", "Vector getOrigin()", asMETHOD(Entity, getOrigin), asCALL_THISCALL); assert( ret >= 0 );

//Another example of object method that fails
ret = engine->RegisterObjectType("Level", 0, asOBJ_REF | asOBJ_NOCOUNT); assert( ret >= 0 );
ret = engine->RegisterObjectMethod("Level", "Vector getBlockOrigin(const Vector& in)", asFUNCTION(_Level_getBlockOrigin), asCALL_CDECL_OBJFIRST); assert( ret >= 0 );

//Example of code that will break
Entity@ entity = CreateEntity("test");  //Works
entity.setOrigin(Vector(1,1,1));  //Works
Vector vector = entity.getOrigin(); //Fails: Entity::getOrigin() is called, but obj pointer is not right

At this point I believe the issue may deal with my compile configurations and/or the as_callfunc_x64_gcc.cpp. I'm not versed well in assembler however, so I'm kind of stuck on what to do. Any information anyone might be able to provide is welcomed. If there is more information I can provide about my issue please let me know.

I believe the problem is with the flags you used when registering the Vector type. You're telling AngelScript that this type has a default constructor, destructor, assignment operator, and a copy constructor, which is most likely not the case. These flags are extremely important to get right, as these are what AngelScript will use to know how the type is to be handled in the native calling conventions.

The way you've registered the type will tell AngelScript that values of this type must be returned in memory by a function, this means that AngelScript will reserve space for the value, and pass an extra hidden pointer to the function as the first argument. Since the C++ function isn't expecting this hidden argument, it ends up using that pointer as the object pointer which is not correct and thus causes the application to crash.

Even without seeing the C++ implementation of the GtVector class, my bet is that the correct flag to use is asOBJ_APP_CLASS_C. You should most likely also add to that the flag asOBJ_APP_CLASS_ALLFLOATS so AngelScript will know the type is composed of only floats or doubles.

Reference: Value types and native calling conventions

The crash doesn't happen on msvc, because it has a different ABI, than llvm/gnuc on does. On msvc it's enough for the type to have the default constructor for it to be returned in memory, so even though CDAK is not correct, msvc only looked at C and ignored the rest. On llvm/gnuc it is the existance of the destructor or copy constructor that tells if the type should be returned in memory or not.

The configuration you suggested does solve the issue, below is the new method to register.

ret = engine->RegisterObjectType("Vector", sizeof(GtVector), asOBJ_VALUE | asOBJ_APP_CLASS_CA | asOBJ_APP_CLASS_ALLFLOATS); assert( ret >= 0 );

Looking at my vector class, I did have a default constructor and assignment operator, but no explicit copy constructor or deconstructor.

I believe part of my problem was a confusion about what I was passing in. I had assumed the the types being passed in for constructor/deconstructor/ect... dealt with what I was going to register on the class with the engine, not the actual class itself. I now understand the separation of these two concepts.

I apprecaite the help, thank you very much!

I did the exact same thing for Ogre::Vector3, etc.

The all floats flag is a godsend since I can't (or rather: won't) modify core Ogre code to add copy constructors. :)