This update is going to show you guys what an idiot I can be. Be warned, this could happen to you too!

For the past month, I have been focusing almost exclusively on developing an artificial intelligence system which uses machine learning to play the characters within my game. This is a primary objective. I have spent several weeks learning more about artificial neural networks, reinforcement learning, and a few other AI methodologies. I am by no means an expert at any of this -- I'm just a novice/beginner. I have also spent considerable time working on developing my own AI system which combines the best elements from the existing methodologies but also introduces a model for actual intelligence. Digging into this has felt like a series of intellectual epiphanies exploding in my head and has been extremely rewarding. I feel like I have a strong grasp on intelligence, learning, consciousness, and sentience, and am on the verge of creating a successful conceptual model to emulate intelligence.

Okay, that's a bold claim to make. I'm going to just cut to the chase and discredit myself. This is like saying, "I found a compression algorithm which compresses anything by 97%". Strong claims like this, require proof to be believed, and I have not implemented this. Instead, I'm going to share my design progress on this:
AI 2.0- Reinforcement Learning.docx
(~8 pages, 10 min read)

Within Spellbound, I have spent a lot of time refactoring the AI and preparing it for a machine learning system. An AI agent now has memory and gets signal inputs about the state of the world through its senses. I have implemented sight, hearing, and smell. Sight is a cone of vision which is oriented to the characters head position and rotation. I collect a list of all objects which overlap the cone, and then I do a line trace from the eye to the object to see if there is a line of sight. If an object passes the cone test and the line of sight test, then it is registered as a visible object. Hearing works a bit differently than sight: Sight is looking out into the world to perceive objects, hearing is waiting for object noises to come to us. So, when an object creates a "noise", I create a sphere at that location and set an intensity value to be proportionate to the decibel value of the sound. Then, I change the radius of the sphere at the speed of sound and stop when the intensity of sound attenuation reaches zero (via inverse square law). If this sphere overlaps an ear, and the ear hearing threshold is capable of hearing the current intensity of the sound, then we register the object as being "heard".
The sense of smell is a bit different. Some objects emit odors over time, and the odor of an object slowly radiates outward (probably following an inverse square law for intensity as well). The important note to make distinct about odor is that an object is continuously emitting odors over time. Again, an odor is going to be represented by a sphere which grows over time and only overlaps noses. As an odor emitting object moves, it emits more odor spheres, and we get an "odor trail". A smart creature with a nose can detect an odor, and then find what direction the odor gets stronger, and then it can follow the odor trail to the odor source. So, a hungry zombie can smell living flesh and follow the smell to a living person.
All of the sensory inputs are stored in the short term memory of a brain. If a sensed object is no longer sensed, the lack of sensory information doesn't mean the object stopped existing -- it's sill persistent in memory. Instead of using raw sensory information as our input stream, I use short term memory as the input stream to drive behavior. We can think of memory as a representation of world state for the AI agent, and then operate based off of the state. We can either use this state information as inputs into a state machine based expert system, or we can feed this into a machine learning system. Either way we handle the inputs, should not matter: The output of both systems should be the most optimal behavior for the given state.

Currently, Spellbound uses the state machine based expert system for AI. It works. It makes a believable illusion of intelligence. The code for each characters behavior is roughly 200 lines. That's kind of manageable, right? Keep in mind, the behavior is currently scripted only to suit what is necessary for the prelude chapter of the game, so every time I want to add new game mechanics or capabilities, I will have to script out more behavior. This is where I become an idiot.

My line of reasoning:
"Okay, I am probably going to eventually have up to 50 different characters. They will all need AI scripts to drive their behaviors. My game is constantly changing, so that means every time I make a significant change to the game, I will have to update each AI script. That sounds like a lot of work! Okay... I also don't even know what the most optimal behavior for every character will be, so that may mean that some expert systems are not going to be very good. Wouldn't machine learning be able to handle this gracefully? Let's do that."

Okay. Why am I an idiot? Because machine learning is a trap for engineers. I recently learned about this new term called "Nerd Sniping". Let's take a reality check on what I'm trying to do here. I'm trying to create a generalized AI system which is so good that it learns how to play any character in my game as an expert, without any coaching or training from me. To date, the DeepMind team funded by Google, has been able to create AI systems which play Go so well that they can beat world champions, they can play atari games perfectly, etc. Now, I'm asking that same type of intelligence to play any character in my game? It's possible, yes, but it's not easy. After spending a month doing R&D on this, I realize how hard this would be to accomplish, and if I were to put an estimate on how long it would take for a novice/beginner like me to implement this, I'd be looking at a minimum of 3-6 months. That's 3-6 months in developer time, which means the realistic estimate would be multiplied by any number between two and ten. Just as I have the capability to write my own game engine from scratch (which I spent 12 months on!), I also have the capability to create this kind of general AI system. The hard question is, "Is this really the most important thing for me to work on for the next six months?". Let's put this into a different context: "Is the alternative approach cheaper and faster and less risky? (yes)" and "Do you want to ship games today or build technologies for 6-12 months away?"

You know that "Always be closing" scene from "Glengarry Glen Ross"? The equivalent for game development is "Always be shipping!". The focusing question for everyone on the team should be, "What am I doing to make this game ship as soon as possible?" And the other question: "What am I doing to increase sales?"
If you don't sell, you don't make money.
If you don't make money, like it or not, you are on your way out of the industry.
If you don't ship, you don't sell.
If you ship garbage, you don't sell.

Therefore, by hypothetical syllogism: If you don't ship or you ship garbage, then you are on your way out of the industry.
If you want to stay in the industry, then ship fast, ship quality, ship often.

My AI system, while interesting, does not help me ship my game faster. It's a trap. It's a premature optimization to solve a problem I don't have yet (and may not ever have!). There's a bigger problem to solve: I need to ship quality content asap. I could continue working on this AI system and I could develop it to work and be so good that it can be reusable within any game, and I could turn it into its own product/technology and license it out to other companies, even outside of the game industry. I eventually plan on doing this. However, the right time is not right now. I need to ship my game, and whether I have machine learning AI or expert system AI, won't matter to 99% of the customers.

So, I'm going to focus on wrapping up the production of the Prelude episode, shipping it asap, and then switching gears to working on the content for Episode 1. I'm going to design the content for Episode 1 to keep AI behavior relatively simple for now. If I ship Episode 1 and sales completely suck despite my best marketing efforts, then there's no point in creating Episode 2. If Episode 1 is a success, then I can build Episode 2, and *that* may be the right time to build out my machine learning AI. Certainly by Episode 3. The key is to build a customer base first so that releases actually have an audience to see them.

I've recently been talking by email with a developer relations rep from Oculus. He wanted to drop the price of my game for their online store to better reflect the content. Initially I resisted the idea, but started thinking about alternatives. 

I currently offer my game on Steam for $20 in early access, which has Episodes 1-3 included, whenever the production finishes. I like the simplicity of that structure because it means I can just release builds which have content updates included. Another option I am now considering is releasing each episode as a separate purchase. I could release the prelude for free to act as a teaser/loss leader, and then have in app purchases for each episode. It might be a good way to build out an customer base, because people like free shit and if the content for the free episode is great and I leave a huge cliff hanger at the end, people will want to buy the next episode. The problem is that the current build on Steam only contains the prelude, so if I give the prelude away for free, nobody has any incentive to purchase the full set of content for $20. So, my current solution is to set the current build to the "premium" version when Episode 1 is released, and then make the prelude free. Existing customers won't feel cheated because they'll get all of the future episodes included. If people want to buy episodes individually, they'll cost a little more per episode ($7.99?) than a bulk purchase of $20. This also solves a future problem as well: What if a future customer isn't interested in the "Red Wizards Tale", but they do want to experience the content for "The Sorceress of Light"? Instead of spending $20 for content they don't want, they can spend it on what they do want.

Anyways, I need to refocus my efforts on shipping the final update for the Prelude. Get it done! The next step is to write the script for episodes 1-3. I've been reading a lot of fantasy books lately to get a better feel for writing fantasy, but I now just feel like an amateur writer in comparison to J.K. Rowling, Brent Weeks and George R.R. Martin. I guess it's important to remember that writing a good story is like constructing a skyscraper. If you only look at the finished product in wonder, you won't see any of the scaffolding it took to build it. The scaffolding of a writer is 20+ iterations of the story? So, I have to write my script about 20 times. The first five drafts will probably be garbage, and will mostly be about trying to find the story I want to tell. The remaining 15 drafts will be refining the story I found and polishing it to perfection. And writing for VR games is more like writing a movie script than writing a book. The reason I need to write out the full story in advance is that I need to know where the story is going so that I can go back to earlier sections of the plot and drop foreshadowing hints, cliff hangers, and sharpen plot twists.