Hello! It seems that if a variadic function returns an object on the stack, it causes a crash when calling SetReturnObject in asIScriptGeneric object.

If I'm not mistaking, the data for the variadic function on the stack is like this:

[{return object address}, {arguments count}, {arguments values}]

stackPointer in the asCGeneric object points to the beginning of args values, but receiving the address of returning object is calculated as “(void*)*(asPWORD*)(stack Pointer - AS_PTR_SIZE)”. This expression does not take into account that before the values ​​of the arguments there is also the number of arguments themselves. To solve the problem, it seems that it is enough to simply subtract an additional 4 bytes if the function is variadic. Here is the code for the changed functions:

int asCGeneric::SetReturnObject(void *obj)
{
	...
	else
	{
		int variadicSizeParameterOffset = sysFunction->IsVariadic() ? 1 : 0;

		// If function returns object by value the memory is already allocated.
		// Here we should just initialize that memory by calling the copy constructor
		// or the default constructor followed by the assignment operator
		void *mem = (void*)*(asPWORD*)&stackPointer[-AS_PTR_SIZE - variadicSizeParameterOffset];
		engine->ConstructScriptObjectCopy(mem, obj, CastToObjectType(dt->GetTypeInfo()));
		return 0;
	}

	objectRegister = obj;

	return 0;
}

void *asCGeneric::GetAddressOfReturnLocation()
{
	asCDataType &dt = sysFunction->returnType;

	if( (dt.IsObject() || dt.IsFuncdef()) && !dt.IsReference() )
	{
		if( sysFunction->DoesReturnOnStack() )
		{
			int variadicSizeParameterOffset = sysFunction->IsVariadic() ? 1 : 0;
			// The memory is already preallocated on the stack,
			// and the pointer to the location is found before the first arg
			return (void*)*(asPWORD*)&stackPointer[-AS_PTR_SIZE - variadicSizeParameterOffset];
		}

		// Reference types store the handle in the objectReference
		return &objectRegister;
	}

	// Primitive types and references are stored in the returnVal property
	return &returnVal;
}

Thanks for bringing this to my attention.

I'll have a closer look at this later, but I think you're correct with your analysis and fix.

Regards,
Andreas

Author of the variadic function feature here. (Original thread: https://www.gamedev.net/forums/topic/717619-support-for-registering-a-variadic-function/ )

The variadic function use a special asCGenericVariadic class that derived from asCGeneric. I think the problem is that I forgot to override the SetReturnObject. I only tested returning object by GetAddressOfReturnLocation in the string format(const string&in, const ?&in …) of string add-on.

Maybe the following fix will work. (I'm quite busy currently, so I haven't tested it yet)

In as_generic.cpp:

int asCGenericVariadic::SetReturnObject(void *obj)
{
	...
	else
	{
		// If function returns object by value the memory is already allocated.
		// Here we should just initialize that memory by calling the copy constructor
		// or the default constructor followed by the assignment operator
		void *mem = (void*)*(asPWORD*)&stackPointer[-(AS_PTR_SIZE + 1)];
		engine->ConstructScriptObjectCopy(mem, obj, CastToObjectType(dt->GetTypeInfo()));
		return 0;
	}

	objectRegister = obj;

	return 0;
}

And in the header as_generic.h:

// In class asCGenericVariadic, add
int     SetReturnObject(void *obj);

I noticed another bug. The function int asCContext::CallGeneric(asCScriptFunction *descr) doesn't calculate popSize correctly for variadic functions. The expression "int popSize = sysFunc->paramSize;" should get the size of the arguments on the stack, however, this is not quite correct for variadic functions, since the number of arguments (and therefore their size) is not known for the function. A possible solution would be to add variadic argument size calculation to the CallGeneric function.

int asCContext::CallGeneric(asCScriptFunction *descr)
{
	...
	asDWORD varArgCount = 0;
	if (descr->IsVariadic())
	{
		varArgCount = *args;

		args += 1;
		popSize += 1;

		// Calculate the size of variatic arguments on the stack
		if (varArgCount > 1)
		{
			asUINT idx = descr->parameterTypes.GetLength() - 1;
			// Size of first variadic argument is included in sysFunc->paramSize value,
			// So, using (varArgCount - 1) * variadicParamSize for calculating the size of variadic arguments
			popSize += descr->parameterTypes[idx].GetSizeOnStackDWords() * (varArgCount - 1);
		}
	}
	...
}

Thanks. I'll review this.