Well, I do I want to continue with this, however I'm a bit frustrated at the moment and completely out of ideas.

I suggest taking a step back, and start over with smaller steps.

Let's make it work to call functions without any arguments, once we get that working, we can add the support for non-float arguments. Only after that we add in the support for float args again.

Start by commenting out the code that uses the args, floatArgs, and paramSize. The input will then just be "m" (ret), "r" (func). Don't forget to update the index where the func argument was used, i.e. %4 to %1.

"# Move function param to non-scratch register\n"
" mov %1,%%r14 # r14 = function\n" // Copy func into r14
" call *%%r14\n"
" lea %0, %%rbx\n" // Load the address of the ret variable into rbx
" movq %%rax,(%%rbx)\n" // Copy the returned value into the ret variable.
: // no output
: "m" (ret), "r" (func)
: "%r14", "%rbx"


results in:

-- TestExecute passed
Failed on line 249 in ../../source/test_cdecl_return.cpp[/quote]

The debugger shows register %rax contains value 102030405 at the moment of exiting the first X64Call..hm..

The function that was called in this test, TestRect(), is returning a structure containing 4 floats. As this is too large to be returned in a register it means the structure is returned in memory. In order for that to work the function needs to receive the pointer to where the value should be stored. The actual value of %rax in this case is irrelevant, though I believe it should be giving the address where the return value was stored, i.e. the same that was given as the first parameter.

If you look at paramSize it should hopefully be 1 for this function, and the first argument in the args array is the pointer.

I'd say you've successfully gotten the code to call native functions without arguments, and even the return value is properly captured.

Now you can add the support for non-float args again.

okay, i was able to proceed further while you were writing your post

-- TestCDeclReturn passed
-- TestExecute1Arg passed
-- TestExecute2Args passed
-- TestExecute4Args passed
-- TestExecute4Argsf passed
-- TestExecuteMixedArgs passed
-- TestExecute32Args passed
-- TestExecute32MixedArgs passed
-- TestCDecl_Class passed
-- TestCDecl_ClassA passed
-- TestCDecl_ClassC passed
-- TestCDecl_ClassD passed
-- TestCDecl_ClassK passed
-- TestReturnString passed
-- TestNegateOperator passed
-- TestReturnWithCDeclObjFirst passed
-- TestExecuteThis32MixedArgs passed
TestNotComplexThisCall: Failed to assign object returned from function. c3.a = BAADF00D
Failed on line 147 in ../../source/testnotcomplexthiscall.cpp
TestNotComplexThisCall: Failed to assign object returned from function. c3.b = BAADF00D
Failed on line 153 in ../../source/testnotcomplexthiscall.cpp
TestNotComplexThisCall: Failed to assign object returned from function. c3.c = BAADF00D
Failed on line 159 in ../../source/testnotcomplexthiscall.cpp[/quote]

It looks like you got the assembly routine to work properly.

Now I think its more about tweaking the as_config.h. What is your current configuration in as_config.h?

The test that failed tells me that the configuration for how non-complex classes larger than 8 bytes are returned is not correct. For MSVC 64bit I have two defines in particular that I think you need:

#define AS_LARGE_OBJS_PASSED_BY_REF
#define AS_LARGE_OBJ_MIN_SIZE 3

#elif defined(WIN32) || defined(_WIN32) || defined(_WIN64) || defined(__CYGWIN__)
...
#undef COMPLEX_MASK
#define COMPLEX_MASK (asOBJ_APP_CLASS_DESTRUCTOR | asOBJ_APP_CLASS_COPY_CONSTRUCTOR)
#undef COMPLEX_RETURN_MASK
#define COMPLEX_RETURN_MASK (asOBJ_APP_CLASS_DESTRUCTOR | asOBJ_APP_CLASS_COPY_CONSTRUCTOR)
...
#else
#define AS_X64_MINGW
#define AS_CALLEE_DESTROY_OBJ_BY_VAL
#define AS_LARGE_OBJS_PASSED_BY_REF
#define AS_LARGE_OBJ_MIN_SIZE 3
#define COMPLEX_OBJS_PASSED_BY_REF
...
// STDCALL is not available on 64bit Linux
#undef STDCALL
#define STDCALL
..
#endif


That's the config I'm using, and ye, it fails with that error above.

Hmm, then MinGW64 is not following MSVC64 exactly in this case.

Can you show me the disassembly of the Class3 ClassNotComplex::notComplex3() method in the file testnotcomplexthiscall.cpp? That will help understanding the configuration should look like.

Ok, here you go:

push rbp
mov rbp,rsp
sub rsp,10h
mov rax,rcx
mov qword ptr [rbp+18h],rdx
mov dword ptr [rax],0DEADC0DEh
mov dword ptr [rax+4],offset image00000000_00400000+0xe34567 (00000000`01234567)
mov dword ptr [rax+8],89ABCDEFh
leave
ret

This disassembly shows me that the class is really returned in memory. The return pointer is received in rcx, and the this pointer in rdx.

It looks like MinGW64 is using a different order for the arguments than MSVC64. For MinGW64 the return pointer comes first, then the this pointer. (this is consistent with how it works on gnuc)

I think you need to change the code in CallSystemFunctionNative to the following:

if( sysFunc->hostReturnInMemory )
{
// The return is made in memory
callConv++;

// Set the return pointer as the first argument
allArgBuffer[paramSize++] = (asQWORD)retPointer;
}

if( callConv == ICC_THISCALL ||
callConv == ICC_THISCALL_RETURNINMEM ||
callConv == ICC_VIRTUAL_THISCALL ||
callConv == ICC_VIRTUAL_THISCALL_RETURNINMEM )
{
// Add the object pointer as the first parameter
allArgBuffer[paramSize++] = (asQWORD)obj;
}
You'll find these two conditions in the very beginning of the function. Just switch the order of them.