I have a function which works correctly with asCALL_CDECL_OBJFIRST. self is the correct pointer, and passed via rcx as specified in the x64 calling convention:

static void ScriptTextureTest(OverlayTexture* self)

Now I also have this function:

static glm::vec2 ScriptTextureGetSize(OverlayTexture* self)

This one is broken - "self" is pointing to the rdx value passed in rather than the rcx value. MSVC seems to treat this function as though rcx is making room for 8 bytes of return value (2 floats in glm::vec2), and so self is assigned rdx, and thus the wrong pointer.

Microsoft's calling convention documentation mentions the following:

struct Struct1 {
   int j, k, l;    // Struct1 exceeds 64 bits.
};
Struct1 func3(int a, double b, int c, float d);
// Caller allocates memory for Struct1 returned and passes pointer in RCX,
// a in RDX, b in XMM2, c in R9, d pushed on the stack;
// callee returns pointer to Struct1 result in RAX.

Specifically, see the first comment. This seems to indicate that room has to be made (on the stack, presumably) and then passed in through rcx, but only if the size exceeds 64 bits.

Personally I haven't seen this calling convention before, AND it is strange that it's even doing this, considering sizeof(glm::vec2) is 8, so it should fit in 64 bits just fine. Maybe this is also a bug in msvc? Not sure.

Here's the disassembly of the GetSize function:

static glm::vec2 ScriptTextureGetSize(OverlayTexture* self)
{
00007FFA10DA5F80  mov         qword ptr [rsp+10h],rdx  
00007FFA10DA5F85  mov         qword ptr [rsp+8],rcx  
00007FFA10DA5F8A  sub         rsp,28h  
00007FFA10DA5F8E  lea         rcx,[00007FFA110A5B4Ah]  
00007FFA10DA5F95  call        00007FFA10EE8BA4  
    return glm::vec2((float)self->m_width, (float)self->m_height);
00007FFA10DA5F9A  mov         rax,qword ptr [rsp+38h]  
00007FFA10DA5F9F  cvtsi2ss    xmm0,dword ptr [rax+0Ch]  
00007FFA10DA5FA4  mov         rax,qword ptr [rsp+38h]  
00007FFA10DA5FA9  cvtsi2ss    xmm1,dword ptr [rax+8]  
00007FFA10DA5FAE  movaps      xmm2,xmm0  
00007FFA10DA5FB1  mov         rcx,qword ptr [rsp+30h]  
00007FFA10DA5FB6  call        00007FFA10C18180  
00007FFA10DA5FBB  mov         rax,qword ptr [rsp+30h]  
}
00007FFA10DA5FC0  add         rsp,28h  
00007FFA10DA5FC4  ret 

Note the [rsp+38], which is 8 bytes ahead of where the first parameter is stored, which is rsp+30. You can see that in the disassembly of the correct test function:

static void ScriptTextureTest(OverlayTexture* self)
{
00007FFA10DA5FD0  mov         qword ptr [rsp+8],rcx  
00007FFA10DA5FD5  sub         rsp,28h  
00007FFA10DA5FD9  lea         rcx,[00007FFA110A5B4Ah]  
00007FFA10DA5FE0  call        00007FFA10EE8BA4  
    WriteLog(log_Overlay, "Test: %d", self->m_width);
00007FFA10DA5FE5  mov         rax,qword ptr [rsp+30h]  
    WriteLog(log_Overlay, "Test: %d", self->m_width);
00007FFA10DA5FEA  mov         r8d,dword ptr [rax+8]  
00007FFA10DA5FEE  lea         rdx,[00007FFA10F815C8h]  
00007FFA10DA5FF5  mov         ecx,5  
00007FFA10DA5FFA  call        00007FFA10C169B0  
    return;
}
00007FFA10DA5FFF  add         rsp,28h  
00007FFA10DA6003  ret 

This behavior seems to have started after I switched from my own simple vec2 structure to the more advanced (but memory-compatible) glm::vec2, as it worked fine before with the smaller vec2 structure.

It's all really strange to me. Open to suggestions of things to try.

The C++ ABI is full of gotcha's. It is rarely as simple as just checking the size of the structure. Things like the existence of constructor, destructor, copy constructor, assignment operator, or the structure containing unions may all affect the ABI. (what's worse, every C++ compiler has its own way of doing it)

This is the reason for the asOBJ-APP-XXX (underlines removed by forum software) flags when registering the type. These flags are what tell AngelScript the particularity of the type, so that AngelScript can make the right choice when calling the functions with the native calling convention.

This glm::vec2 doesn't happen to be a template, possibly with a union as a member, would it?

If I'm not mistaken I've seen this glm;:vec2 before in a previous topic, and I believe the correct way of registering it is:

engine->RegisterObjectType("vec2", sizeof(glm::vec2), asOBJ_VALUE | asOBJ_POD | asGetTypeTraits<glm::vec2>);

Yep, glm::vec2 is indeed both templated/generic with 2 unions. We actually use glm::vec2 in a 32 bit environment (different project) with the same flags (except we add asOBJ-APP-CLASS-ALLFLOATS as well) and that works great.

Removing the ALLFLOATS flag does not seem to fix the problem. Here's how vec2 is registered:

engine->RegisterObjectType("vec2", sizeof(glm::vec2), asOBJ_VALUE | asOBJ_POD | asGetTypeTraits<glm::vec2>() | asOBJ_APP_CLASS_ALLFLOATS);

Hmm.

Because of the union, the asOBJ-APP-CLASS-ALLFLOATS shouldn't be used. However, it seems this is not the cause of your problem since you tested without it already.

I'll need to do some tests, but I suspect the asGetTypeTraits<glm::vec2> is not giving the correct flags for some reason, probably due to the type being templated.

Alright, so I looked at the code and added asOBJ-APP-CLASS-CDK and that seems to have fixed it! So asGetTypeTraits isn't correctly detecting the constructors and destructor.

Which version of MSVC are you using? Do I recall correctly that you use an older version?

I use MSVC 2017 and I didn't face any problems with glm::vec2 and native calling conventions when registering the type with:

		r = engine->RegisterObjectType("vec2", sizeof(glm::vec2), asOBJ_VALUE | asOBJ_POD | asGetTypeTraits<glm::vec2>()); assert(r >= 0);

The test I made is checked in here:

https://sourceforge.net/p/angelscript/code/2635/

It also works perfectly on 32bit as well as with MinGW 64bit. (I haven't tested other platforms/compilers yet)

I had this issue on VS 2019. (We're still using 2017 for Heroes of Hammerwatch, but for my own projects I use 2019.)

Hmm. I'll see if I can upgrade to MSVC2019 and try the test again.

Would you mind giving it a try on your version of MSVC2017 as well?

I downloaded and tried it with MSVC2019 over the weekend.

I'm still not able to reproduce the problem.

Have you by any chance made any customizations to angelscript or glm that may have affected the behaviour?

I have not, it's an unmodified Angelscript from the latest SVN revision. I'm not entirely sure what version of GLM this is though.

asGetTypeTraits<glm::vec2>() returns only asOBJ-APP-CLASS.

I just updated GLM to 0.9.9.7 (the latest on the website as of writing this), and it gives the same result.