Introduction

This whole adventure started a couple of weeks ago, when a question about using Boost.Python was posted on a Swedish game development forum where I usually lurk. I hadnAEt used Boost.Python before, and I wasnAEt even using Python in my current project, but I decided to take a quick look anyway. A really cool thing that I saw was that they could automatically deduce the types of the parameters for functions they were exporting, but I didnAEt give it that much more thought. A little while later I started looking around for a C++ library that would let me export functions to Lua, and I stumbled upon Luabind which used a lot of the same techniques as Boost.Python. Once again, the library could automatically deduce types, and now I was really curious.

I downloaded the Luabind code and started rummaging through it. I quickly realized that the project was way beyond the scope of what I was looking to do (which was just the most basic automatic exporting of functions from C++ to Lua). But in skimming the code, I noticed that Luabind used Boost.FunctionTypes, which was a library that I'd never paid much attention to before, so I decided to check it out. And when I did, it literally blew my mind.

Boost.FunctionTypes gives you functionality to generate typelists containing the types of the parameters to any function you specify (and more!). My Spidey sense was telling me that this was all I needed to automatically generate a type-safe interface between C++ and Lua, so I decided to see were this would take me

Breaking It Down

The problem of exporting C++ functions to Lua can be broken down into a couple of distinct parts:

• Better dispatching of the Lua callback. The Lua C interface allows you to connect a string (the function name) to a static function with a given prototype, and from this callback I had to be able to call the correct instance of the class that would handle the real executing.
• Type checking and value extraction. Given the Lua stack, I needed to be able to traverse it, and for each value on the stack verify that it was of the correct type, and if it was, store it away somewhere until it was time to call my function.
• Execution. At this stage, I would be holding some kind of function object that represented the C++ function I was going to call, and I also had a list of the parameters, so the problem was invoking the functor with the parameters.
• Handling the return value. Functions that don't return anything aren't really that fun, so the final part of the puzzle was getting the return values from my C++ function back to Lua via the stack.

Better Dispatching of the Lua Callback

When dealing with function pointers to unknown types, I usually begin with a template class holding with a member variable representing the function pointer, so I started with that.

Lua states that all C callbacks we expose must have the following signature:

typedef int (*lua_CFunction) (lua_State *L);

Now this didnAEt exactly match anything I had written yet, but I added the static method inside my template class, thus guaranteeing that IAEd at least have a unique static method for each type of function I was exporting.

Lua allows for C closures, which enable you to associate variables, integers in this case, with a specific callback. As I had a unique static callback for each type, I knew that it would be safe to store my this pointer as an integer in the closure, and cast it back when my callback got called. This gave me the following structure:

template< class T >
struct CommandT : public Command {
  CommandT(const string& name, const T& fn) 
    : Command(name)
    , fn_(fn) 
  {}

  int Execute(lua_State* lua_state) const {
    return 1;
  }

  static int callback(lua_State* lua_state){
    const CommandT* this_ptr = reinterpret_cast(lua_tointeger(lua_state, lua_upvalueindex(1)));
    return this_ptr->Execute(lua_state);
  }
  T fn_;
};

I also added a simple base class, Command, and a class to manage some state, like the Lua state, and the pointers to the Command classes that IAEm creating, and gave the manager class a Register method to call where I could register function names and pointers. Register would create an instance of the CommandT class, register the callback, store a pointer to the class in the C closure of the function, and also put the pointer in an internal list so it could be deleted when we shut down.

  template< class T >
  void Register(string function_name, const T& t) {
    CommandT* ptr = new CommandT(function_name, t);
    // Store the this pointer in the c-closure
    lua_pushinteger(lua_state_, reinterpret_cast(ptr));
    lua_pushcclosure(lua_state_, &CommandT::callback, 1);
    lua_setglobal(lua_state_, function_name.c_str());
    commands_.push_back(boost::shared_ptr(ptr));
  }

Type Checking and Value Extraction

Ok, this is where it starts to get exciting! By applying the function_types::parameter_types metafunction1 we get a typelist that contains the types of the parameters of the function we want to export2. WhatAEs even better is that this typelist is in boost::mpl format, meaning that we can manipulate the typelist using the tools available in the metaprogramming library. One of these tools is for_each, which applies a user-specified functor on each type in the typelist.

IAEve experimented a bit with functors mapped over typelists before, so I had a pretty good idea of what I wanted to do here. For each application of the functor, weAEre basically walking one step down the Lua stack, and seeing if the type we get from the typelist matches the type that Lua says is on the stack. If it is, we want to get the value from the stack and store it; otherwise we signal an error in some way.

One way to implement this is by having the functorAEs operator() call a method that weAEve overloaded with all the types we want to be able to handle. The overloaded method can then use LuaAEs own lua_isXXX family of functions to ensure that the parameter has the correct type. The functor also holds a stack pointer, keeping track of where we are in the Lua stack, and increments this on each iteration.

class Extractor
{
public:
  Extractor(lua_State* lua_state, AnyParams& params) : lua_state_(lua_state), params_(params), stack_index_(1) {}

  void Extract(const int&) {
    if (!lua_isnumber(lua_state_, stack_index_)) {
      throw exception("Expected parameter of type interger");
    }
    params_.push_back(lua_tointeger(lua_state_, stack_index_));
  }

  void Extract(const std::string&) { 
    if (!lua_isstring(lua_state_, stack_index_)) {
      throw exception("Expected parameter of type string");
    }
    params_.push_back(string(lua_tostring(lua_state_, stack_index_)));
  }

  template< typename T >
  void operator()(T t) {
    Extract(t);
    stack_index_++;
  }
private:
  lua_State* lua_state_;
  vector& params_;
  int stack_index_;
};

The Extract method is overloaded for each type we want to be able to extract (double and bool have been omitted above). IAEve actually rushed ahead a little here, and also shown how I extract the values from the Lua stack. This turned out to be trivial, just adding this little guy:

typedef vector AnyParams; 

So if the value on the stack is of the correct type, we just store it in our boost::any vector. Our Execute method has been extended a little to apply the Extractor functor as well

   typedef typename parameter_types::type ParamTypes;

  int Execute(lua_State* lua_state) const {
    AnyParams params;
    boost::mpl::for_each(Extractor(lua_state, params));
    return 1;
  }

Execution

This is moving along nicely! WeAEve got a function object, and a vector of values of the correct types, so now all we need is a way to call the function with the parameters.

I banged my head against this problem for a little while, until I remembered a trick that IAEd used a while back, that involved partial template specialization and a one line superstar called Int2Type.

template struct Int2Type {};

So, what does Int2Type do? Well, you create an instance of it with an integer, and youAEve got yourself a unique type for that integer value. ?Um, yeah? I hear you saying, hoping that IAEll show you where IAEm going with this. How about if we combine this with another metafunction from function_types, namely arity? This will give us unique types for each arity, and specializing a template on these types will allow us to create unique methods to invoke for each number of parameters.

Hopefully some code will be less confusing than that description!

/**
* The executor is a class with operator() specialized on the class Int2Type, where N is the
* arity of the function we're calling. By specializing on the arity, we know the number of parameters
* the function expects, and can extract these from the vector and cast to the correct type
*/ 
template
struct Executor;

// 0 parameters
template
struct Executor, ParamTypes> {
  template void operator()(Fn& fn, const vector& /*params*/) {
    fn();
  }
};

// 1 parameter
template
struct Executor, ParamTypes> {
  template void operator()(Fn& fn, const vector& params) {
    fn( boost::any_cast >::type>(params[0]) );
  }
};

// 2 parameters
template
struct Executor, ParamTypes> {
  template void operator()(Fn& fn, const vector& params) {
    fn( 
      boost::any_cast >::type>(params[0]),
      boost::any_cast >::type>(params[1]) );
  }
};

To be able to get the ?unboxed? value from the boost::any value, we need to call boost::any_cast to cast it to the type we want. Getting the correct type for a parameter at any given index is done using the ?at? metafunction which returns the type at the specified index in a typelist (using the int_ construct to specify the index)3. This type is passed as the template parameter that any_cast wants.

And once again we update our Execute method slightly to create the proper instance of Executor and call it.

// Typedef for the executor (based on the arity of the function)
typedef Executor< Int2Type::value>, ParamTypes> Executor;

int Execute(lua_State* lua_state) const {
  AnyParams params;
  boost::mpl::for_each(Extractor(lua_state, params));
  Executor f;
  f(fn_, params);
  return 0;
}

WeAEve now got type safe extraction and execution of functions with an arbitrary number of parameters (well, as many as we specialize the executor for), and only one piece of the puzzle remains.

Handling Return Values

The final problem, pushing eventual return values back on the Lua stack, actually consists of two smaller problems. First, we have to determine if the function weAEre exporting has a return value at all; the reason for this is that we need separate code paths for handling the void and the non void return value cases. Pseudo code would look something like:

if (function_has_return_value) {
  return_value = function();
  push_on_lua_stack(return_value);
} else {
  function();
}

Ideally we want this resolved at compile time, as all the information needed is known then. The second problem is knowing that we have a return value, to push it on the stack.

Finding out if we have a return value or not is solved by combining function_types magic to get the type of the result, and using the is_void metafunction found in boost.type_traits to determine if itAEs void.

typedef typename result_type::type ResultType;
enum { IsVoidResult = boost::is_void::value };

If we view the pseudo code above as a case of overloading, with the if branch implemented in one function body, and the else branch in another, we just need to figure out how to call the correct branch, depending on the value of IsVoidResult. ItAEs time for Int2Type again!

// Typedef for the executor (based on the arity of the function)
typedef Executor< Int2Type::value>, ParamTypes, ResultType > Executor;

// Executer for void return type
int Execute(lua_State* lua_state, Int2Type) const {
  AnyParams params;
  for_each(Extractor(lua_state, params));
  Executor f;
  f(fn_, params);
  return 0;
}

// Executer for non void return type
int Execute(lua_State* lua_state, Int2Type) const {
  AnyParams params;
  for_each(Extractor(lua_state, params));
  Executor f;
  AddReturnValue(lua_state, f(fn_, params));
  return 1;
}

static int callback(lua_State* lua_state){
  const CommandT* this_ptr = reinterpret_cast(lua_tointeger(lua_state, lua_upvalueindex(1)));
  return this_ptr->Execute(lua_state, Int2Type());
}

We create two versions of Execute, one for the void case, and one for the non void case. We then use Int2Type to create an instance of either Int2Type or Int2Type, depending on the value of the enum IsVoidResult, and overloading takes care of calling the correct version.

Once this was solved, pushing the correct value to the Lua stack was a simple matter of overloading AddReturnValue for the types we wanted to be able to return, and a call to the correct lua_pushXXX function.

void AddReturnValue(lua_State* lua_state, const bool value) {
  lua_pushboolean(lua_state, value ? 1 : 0);
}

void AddReturnValue(lua_State* lua_state, const int value) {
  lua_pushinteger(lua_state, value);
}

And finally, weAEre done (Yata!)

Future Work

Any time you start working on something fun, a million ideas for improvements pop into your head. Here are a couple of things that I might experiment with in the future.

• Named parameters, and a single dispatch function. A friend at work was working on a system where he just exported a single function from C++ called dispatch, and that function took two parameters: the name of an internal function to call, and a table containing pairs of parameter names and values. This would require adding support for tables, amongst other things.
• Support for member functions. Right now, we just handle global functions, but it should be fairly simple to extend to support member function pointers as well. This will require handling objects going out of scope and unregistering their callbacks when they do.
• Multiple return values. Might be more fun than useful, but IAEm interested in seeing if I can get Boost.Tuple to interact with this system, and allow for multiple return values on the Lua stack.
• Use the Preprocessor Library for generating the execute code. That code is just repetitive and error prone to write for a greater number of parameters, and seems like a great fit for the Preprocessor Library.
• Understand whatAEs going on under the hood of FunctionTypes. I think this can be summarized with *gulp*.

Sample Code

The sample code is a single .cpp file along with a .lua file that calls the exported functions. To be able to compile, you need to have Lua and Boost installed, with correct include paths set up. IAEm using Lua 5.1 and Boost 1.36.

References

Lua
Boost, various libraries
Luabind
Modern C++ Design, Andrei Alexandrescu
C++ Template Metaprogramming, David Abrahams, Aleksey Gurtovoy

1 IAEm not sure of the proper name for this type of construct, and I realize that metafunction is probably wrong, as it returns a type and not a value, but it will have to do.
2 This isnAEt 100% accurate, as the typelist may contain references to types, which we donAEt want, so we apply a transform to the list to remove references. See the code for more details
3 Using boost::any is a bit of overkill, as we know when we extract the parameters that weAEre getting the correct types, so we donAEt need the extra type safety/overhead that any_cast gives us