I tried assigning a registered function to the function pointer and call that instead of the script function. This didn't work as there was a bug in the VM when executing the asBC_CallPtr instruction in that it failed to recognize that the function was a registered function instead of a script function.

However, after fixing that bug, in revision 1117, I'm still not able to reproduce the problem you have.

Thanks, I'll update our svn copy to include the fix from rev 1117 and report back on the issue.

Not sure what's changed but now I'm getting a crash in asCScriptEngine::CallObjectMethod:

void (asCSimpleDummy::*f)() = p.mthd;
obj = (void*)(size_t(obj) + i->baseOffset);
(((asCSimpleDummy*)obj)->*f)();


Unwinding the stack a bit:

// Release previous object held by destination pointer
if( *d != 0 )
engine->CallObjectMethod(*d, beh->release);
// Increase ref counter of wanted object
if( s != 0 )
engine->CallObjectMethod(s, beh->addref);


The crash is triggered by setting object's function pointers to global functions in the AddReference behavior:

obj->asThinkFunc = asEntityCallThinkFuncPtr;
obj->asTouchFunc = asEntityCallTouchFuncPtr;
obj->asUseFunc = asEntityCallUseFuncPtr;
obj->asStopFunc = asEntityCallStopFuncPtr;
obj->asPainFunc = asEntityCallPainFuncPtr;
obj->asDieFunc = asEntityCallDieFuncPtr;


All of those asEntityCall*FuncPtr are of type void * and hold return values of asGetGlobalFunctionByDecl.

Ah, if I change this part of the code:

obj->asThinkFunc = asEntityCallThinkFuncPtr;
obj->asTouchFunc = asEntityCallTouchFuncPtr;
obj->asUseFunc = asEntityCallUseFuncPtr;
obj->asStopFunc = asEntityCallStopFuncPtr;
obj->asPainFunc = asEntityCallPainFuncPtr;
obj->asDieFunc = asEntityCallDieFuncPtr;


to


// obj->asThinkFunc = asEntityCallThinkFuncPtr;
// obj->asTouchFunc = asEntityCallTouchFuncPtr;
obj->asUseFunc = asEntityCallUseFuncPtr;
// obj->asStopFunc = asEntityCallStopFuncPtr;
// obj->asPainFunc = asEntityCallPainFuncPtr;
// obj->asDieFunc = asEntityCallDieFuncPtr;


I then hit the assertion from the first post. I'm quite sure the use function is never called though.

That's the second time you mentioned setting the function pointers in the AddRef behaviour. What exactly do you mean with this?

The method you register with asBEHAVE_ADDREF is not supposed to change anything but the internal reference counter of the object.

The crash you're getting looks like it is caused by a previous invalid pointer cast, e.g. you do a C style cast (or reinterpret_cast) where a static_cast/dynamic_cast is needed to re-evaluate the pointer.

Try setting a break point in the function where you're setting the function pointers and verify that the object pointer is really pointing to what you think it is. I believe you'll find that it is pointing to something else. If I'm right you'll need to track down where the object pointer was misinterpreted as something it was not and fix that.

Hm... In release build program execution simply stops with 'Pure virtual function call' message. I'll try to investigate this further.

Ok, nvm, fixed this. The assertion failure is still there though. It seems to hit several functions and behavoirs, involving several classes.

I now believe this is somehow related to setting a funcdef property in the script in this manner:


void trigger_capture_area_think( cEntity @ent )
{
}

void trigger_capture_area( cEntity @ent )
{
@ent.think = trigger_capture_area_think;
}


when both 'trigger_capture_area' and the think function are called from the application.

In the aforementioned case, in ValidateNoUsage, asCObjectType *type is going to reference the 'cEntity' class and 'asCScriptFunction *func', trigger_capture_area_think.

Maybe trigger_capture_area_think isn't removed because it is still being referenced by an object through a funcdef?

It's possible. I'll make a test and see if I can reproduce the problem with this new information.

Yes! Finally I was able to reproduce the problem with the following code:



{
asIScriptEngine *engine = asCreateScriptEngine(ANGELSCRIPT_VERSION);

engine->BeginConfigGroup("gr");
engine->RegisterObjectType("type", 0, asOBJ_REF);
engine->RegisterObjectBehaviour("type", asBEHAVE_ADDREF, "void f()", asMETHOD(Type,AddRef), asCALL_THISCALL);
engine->RegisterObjectBehaviour("type", asBEHAVE_RELEASE, "void f()", asMETHOD(Type,Release), asCALL_THISCALL);
engine->RegisterFuncdef("void fun(type @)");
engine->RegisterObjectProperty("type", "fun @callback", asOFFSET(Type,callback));
engine->EndConfigGroup();

asIScriptModule *mod = engine->GetModule("mod", asGM_ALWAYS_CREATE);
mod->AddScriptSection("s",
"void func(type @) {} \n"
"void main(type @t) \n"
"{ \n"
" @t.callback = func; \n"
"} \n");

int r = mod->Build();
if( r < 0 )
TEST_FAILED;

Type *t = new Type();

// Call the function that sets the callback on the object
asIScriptFunction *m = mod->GetFunctionByName("main");
asIScriptContext *ctx = engine->CreateContext();
ctx->Prepare(m);
ctx->SetArgObject(0, t);
r = ctx->Execute();
if( r != asEXECUTION_FINISHED )
TEST_FAILED;
ctx->Release();

// Release the engine, while the object holding the callback is still alive
engine->Release();

t->Release();
}


The problem happens because the cEntity object is still alive when the engine is released. If you release the cEntity object before releasing the engine then you should be able to avoid the problem in your game.

I'll investigate what needs to be done to fix the bug in the library.


Thanks for helping me figure out how to reproduce the problem.

Yeah, I had the same conclusion in my mind. Thanks for being so patient with me

However, releasing objects prior to AS shutdown is somewhat problematic due to objects cross-referencing each other, I'm pretty sure that'd totally fuck up AS reference counting.