(Warning: Rambling ahead!)

I met a new client in the beginning of November through a friend. The client asked my friend if he knew how to use Leap Motion. He said, "No, I don't, but I know the just guy who does! Eric is probably one of the best Leap Motion developers in the country.", and a referral was sent my way! I need to remember to buy my friend a nice dinner as a show of thanks.

Anyways, this project was interesting for a variety of reasons. First off, the bulk of development was being outsourced to Ukraine. The outsourcing of work meant that the project owner needed to work late into the night to coordinate efforts with his Ukraine team. They were having some problems with the wrist joint causing the mesh to twist (candy wrapper effect) and nobody was making much headway. So, I was brought into the project. I downloaded the full project, tried it out to see where they were at, saw the problems, and then I started looking at the implementation. Pardon my unprofessional trash talk here, but the implementation looked like it was being held together with bubblegum and duct tape. I told them that I was going to redo the project. It was such a simple project that it took me about two days to rebuild. I was able to simplify, condense and cut about 75% of whatever those Ukrainian guys did, and it was a "correct" implementation which was maintainable. It was valuable as a throw away prototype, but for production work, it needed to be redone. I told them, "If you can't tell the difference between what I did and the original, it means I did my job right." This is always a tough sell for non-technical people, but you see the real value as the project progresses. When everything is done right from the very beginning, you skip over lots of problems you would have otherwise had (which costs time, energy and money).

As far as project implementation goes, this was a super easy project (for me) because I had already done several similar projects and had built out a template library to use. The hardest problem was actually dealing with the wrist and elbow joints with leap motion. So, I'll describe some of that pain here:

The initial problem was that when you rotate your wrist (roll along arm axis), the wrist bone rotates but you start to get some pretty bad deformation of the arm mesh, to the point where the wrist gets pinched to look like a candy wrapper. The problem was a combination of technical implementation, incorrect rigging, incorrect mesh weighting, and a bit of fighting the leap motion plugin. Fixing the problem can be sort of described as trying to straighten wet spaghetti noodles, because so many different factors contributed to the problem. There was no single silver bullet fix that magically fixed it all. My greatest secret weapon was my strong debugging skills and approach to debugging. Although my client liked to call other experts with industry experience, they could only do vague hand waving and guess work at the causes of the problem. The true, best way to figure out what's going on is to dig into the problem, get deep into the weeds, add a ton of debug scaffolding code and visualization helpers, and really spend the time and effort to look at what's going on. A slow, methodological, scientific approach is tried and true for me and far more illuminating than the guesswork of experts unfamiliar with the project.

As far as skeletal rigging goes, I ended up downloading a freely available Paragon character with exposed arms with rotating animations. I looked at the bone setup. To fix the candy wrapper effect, you need two twist joints in the arm, parented off of the elbow joint. The wrist twist joint needs to be about 1cm away from the wrist. You also need a forearm twist joint in the middle of the arm. When you apply roll rotation to the wrist, you only apply roll to the wrist twist joint but yaw and pitch are handled at the wrist joint itself. You also need to take about 50% of the wrist roll and apply that to the forearm twist joint as well. Otherwise, you start to get bad mesh deformation and loss of volume. You also need to be very robust with your mesh weighting on each of the bones. It's good to use a working reference model to get an approximation on the best vertex weights.

There were also some challenges with the out of the box leap motion plugin as well. The problem is that you get the wrist transform, but the rotation values are for the wrist joint itself, not for arm twist joint roll values. I discovered that I needed to derive my own arm aligned wrist rotation, relative to the elbow position. The rotational value needs to be independent of wrist pitch and yaw. Anyways, I won't get into the vector math required to do that, but I ended up having to derive these rotational values myself. It's important to note that the wrist joint is being driven by the user (via IK) and the elbow is an intermediate joint, with transforms being set by the IK solver. The user can put their arm into any position and contortion, and the virtual arm you present needs to match as closely as possible with the physical arm location. This is really hard to get right because the animations need to support *anything* the user does -- any popping or glitches are obvious and break immersion. Doing good elbow placement with limited data has been one of the harder challenges facing most VR developers, so it hasn't been a well solved problem. Most VR developers will skip it all together and just show people a pair of disembodied hands at the motion controller locations, but if you are trying to use a full body avatar for the player character, it's a problem you eventually need to solve. I think I did a pretty good job on the elbows.

I also made a discovery about the hardware capabilities of leap motion: Though this shouldn't be a surprise, the leap motion has great trouble finding elbow placement when people are wearing coats or sleeves. If you want reliable elbow tracking, you need to take off your coat and roll up your sleeves. You also need to be very aware of bright light sources in your environment because those light sources will interfere with leap motion. Keep in mind, the leap motion uses an IR camera to to track hand gestures, and the HTC Vive uses two laser base stations which emit IR lasers, so there is a slim chance that you can get laser base station interference, depending on placement and what direction you're looking. And... if it's not obvious, sunlight causes bad tracking environments for leap motion.

Anyways, I nailed it on this project. You can pretty much do anything with your arms and the digital avatar will do the same thing. I'd show a video of the result, but I think the NDA restricts me from showing that. I don't think I'm the "best" leap motion developer in the country, but I'm firmly in the top 10%. My bigger worry about Leap Motion right now is the health of the company and what that might mean for their product. Are they going to be in business two years from now? Who knows. But, the library of content built for their hardware platform is slowly growing.

On a VR production side note, I made an amazing discovery in this project. If you setup an Oculus to work from your chair, you can develop in VR using the built-in desktop virtualization app. As long as you can type without looking at the keyboard, you can develop in VR. Instead of having a 17 inch monitor, you can have a 5 foot screen in VR. And if you want, you can drag out windows and surround yourself with as many screens as you want. This taxes the GPU a little bit, but who cares? The beauty about this is that if you're developing VR content, you don't have to keep on putting the headset on and off between development iterations. Since you're already in VR, switching between your IDE and your VR app is reduced to a single button press. I worked like this for several days to see how I liked it and if it was feasible. I thought I'd get sweaty face, eye strain, or have some other related draw back issue. Aside from not being able to see my keyboard, it was pretty solid! There are small preference issues I'd like to change. I prefer to listen to music on youtube while I work, and when I work in VR, I am stuck with the ambient music in the oculus home environment, and switching between IDE and project switches the audio (this should be a preference). What's extra magical about working in VR is that it has a pair of headphones and a microphone embedded into the hardware, so if you wanted to do a VoIP call over Google Chat and do screen sharing, you can do it all in VR without taking off the headset! I truly believe this is going to eventually become the future of how people work and interact with computers and each other. I feel like I'm on the bleeding edge here and I saw a glimpse of the future, and damn, it looks awesome!

In closing, this project was relatively small and simple. I was instrumental in shipping it. I feel that I have a fantastic relationship with my client, they are extremely happy and impressed with the work I did, and I will be their go to guy for future VR work. This is how you get repeat business and build a financially self sustained company. Now, if only I knew how to market myself... There's another potential upcoming VR project with a much bigger budget that I am extremely excited about. I can't go into details, but I will say... if we get this and pull it off, it's going to create a new industry and there will be lots of magazine articles written about what we did. My experimental research work in AI will not be wasted.

Spellbound:
As far as Spellbound work is concerned, I still work on it on the side, between projects. Since this is going to be mostly a story driven VR game, I've been focusing a lot on the story writing. It feels weird to open up a google doc for my game story and say I'm going "game development". It's creative story writing, not code writing. And... it's just as hard. I've gone through probably 10 different iterations of the story so far and scrapped most of them. I'm finding that the story writing for VR is extra hard and takes a LOT of time. You need to write it like a movie manuscript, but you also need to think about presentation and the fact that the story is not consumed linearly. If you're an indie on a budget (like me), then you need to consider that every word costs money and player patience, so if you're excessively verbose, it's going to be expensive and nobody wants to sit through long, boring performances. You also have to consider the localization angle: If your scripts have lots of sophisticated word play and language based puns, you are going to hate yourself when you need to localize for different audiences. Short, simple, sweet, to the point, well worded. A story isn't about how flowery and eloquently you can write your sentences, it's about retelling a sequence of emotional events which leaves an impact on your audience...

I recently realized that the story writing doesn't need to be done from an omniscient point of view where you tell every important detail, it can be done from the point of view of the story writer (an in-game historians perspective, if you will) who has a limited point of view. This let's me skip details up front, hide them a bit, and do a big reveal later on in the plot as a twist in the story. With this style, you can experience the story twice and see the plot structure and make new sense of various setup clues (similiar to how you can watch the movie "The Sixth Sense" twice and get two completely different viewing experiences).

Here is the dressed down storybook intro for Episode 1 in Spellbound:

[This is the story book intro. It is presented as a series of illustrated pages in a book and is read by a narrator. We have accompanying voice over lines and sound effects to go along with the story, so that it feels almost like a cinematic.]

Page 1:
Thousands of years ago, evil wizards and witches opened the gates to hell. Swarms of demon spawn entered into the mortal realms, lead by the arch demon Asmodeus.

Page 2:
No army could stand against the demon spawn. Cities were reduced to rubble and ash, tens of thousands of people fell to their blades like wheat to a scythe...

Page 3:
When all seemed hopelessly lost to darkness, the eastern horizon erupted in a blinding glow of shimmering light. Asmodeus was no more, and his leaderless demon spawn retreated to hell.

Page 4:
Nobody quite knew what happened or how they won, but one thing the remaining kings agreed upon was the outlaw of magic and the execution of all its practitioners.

Page 5:
People were finally safe again... all but for wizards and witches. For thousands of years, those who even *looked* like they could be magical, were hunted down and killed in every kingdom.

(Short pause / transition break for establishing shot)

Page 6:
On the edge of the Blackwood Forest, in a small hamlet named Halfordshire, a humble pair of farmers welcomed a new boy into the world. After seeing his fire red hair, Father named him "Rupert".

Page 7:
Rupert grew up, as all young boys do, and trouble followed naturally, as it does with all young boys.

Page 8:
But the natural troubles of boyhood soon turned into supernatural troubles which seemed to haunt Rupert everywhere he went. In a freak magical accident, his home was utterly incinerated. Only Rupert survived. 

Page 9:
As he stumbled out of the smouldering cinders of his former home, the villagers could see that not even one red hair on Rupert was burned. Was it a miracle?

Page 10:
As the initial shock wore off, villagers began to shout that he was a witch. Fearing for his life, Rupert ran, and he ran, and he ran, deep into the Black Forest. As day turned to dusk, the village hunters abandoned the hunt.

Page 11:
Rupert survived through the night, wandering through the forest for days, getting hungrier and hungrier. He stumbled upon an old tower of crumbling stone, and made it his new home and lived off of whatever edible bark, berries and mushrooms he could find in the forest.

Page 12:
Rupert lived in the crumbling stone tower for years, completely alone. Whatever was haunting him, seemed to follow him everywhere he went. What he didn't realize is that the next morning, his life would be changed forever...