I've used it a long time ago so maybe can't remember well. Basically, Open and close are callback functions which get invoked when compiling your shaders with the D3DCompile API. When entering the Open function, you will receive a filename string in pFileName parameter, while also a file type in the include type parameter. The file type will be D3D_INCLUDE_SYSTEM if your include looks like this:
#include <myHeader.h>
The file type will be D3D_INCLUDE_LOCAL when your include looks like this:
#include "myHeader.h"
In both cases, you must load the file by hand, but the file type gives you a hint where to look for the file. What the system path or the local path will be depends on you, usually the local path would be the current working directory and any folder in there which was specified if you have something like "Additional include directories". The system path also depends on you, probably your engine source directory, or anything... So you load your file with eg. std::ifstream, read all of the data and if everything is successful, you will return the data in the ppData function argument and the data size in bytes in the pBytes function argument and the function should return S_OK.
Else, let your function return E_FAIL if you couldn't load the file.
If everything was successful, the Close function will be invoked at the end of shader compilation and you can free the data.
Apologies but I don't have code example of this, but should be fairly straight forward.
Btw, maybe it's not what you want, but you can also just call fxc.exe and include paths will be handled for you, you can specify additional include directories with -I command line argument.