I've discovered some sort of bug in the "CallSystemFunction." I have only encountered it in x64 Windows builds (Visual Studio 2013).
It looks like the code cleaning up the arguments incorrectly attempts to clean up the return variable. To trigger the bug, I register a class with a destructor and register a function that returns an instance of this object and takes one as an argument, so that the argument needs to be cleaned up. The cleanup code will then attempt to free the argument, but it will mistakenly pass a pointer to the return variable to be freed. This pointer is not a valid heap pointer, so there is a crash.
The attached file can be added to the test project to demonstrate the problem.
The fix may be something as simple as adding the following code the the cleanup section, but I am not familiar enough with how the AngelScript compiler manages its memory and sets up function calls to make that judgement.
as_callfunc.cpp(820)
// Clean up arguments
const asUINT cleanCount = sysFunc->cleanArgs.GetLength();
if( cleanCount )
{
args = context->m_regs.stackPointer;
if( callConv >= ICC_THISCALL )
args += AS_PTR_SIZE;
//** New code to address bug **/
if( descr->DoesReturnOnStack() )
args += AS_PTR_SIZE;
asSSystemFunctionInterface::SClean *clean = sysFunc->cleanArgs.AddressOf();
for( asUINT n = 0; n < cleanCount; n++, clean++ )
{
void **addr = (void**)&args[clean->off];
if( clean->op == 0 )
{
if( *addr != 0 )
{
engine->CallObjectMethod(*addr, clean->ot->beh.release);
*addr = 0;
}
}
else
{
asASSERT( clean->op == 1 || clean->op == 2 );
asASSERT( *addr );
if( clean->op == 2 )
engine->CallObjectMethod(*addr, clean->ot->beh.destruct);
engine->CallFree(*addr);
}
}
}