Running AngelScript 2.31.1 WIP revision 2319, I have a script method registered in the following way:
ASengine->RegisterObjectMethod("Object", "func@ opCast()", asMETHOD(Object, operator asIScriptFunction*), asCALL_THISCALL);
where Object is a class and func is a funcdef. The method is implemented in such a way that it may return both null and non-null handles, but all results are considered null by the following script:
void main() {
Object o;
if (cast<func@>(o) is null)
print("null");
}
I attempted to reproduce this using only a script, but the bug seems to occur only in application-registered functions. I spent some time debugging, and from what I understand, the function handle is erroneously placed into valueRegister instead of objectRegister because line 726 of as_callfunc.cpp doesn't predict function handles as returned types. Modifying the line to
if ( (descr->returnType.IsObject() || descr->returnType.IsFuncdef()) && !descr->returnType.IsReference() )
appears to fix the problem in this case.