Hello dear AI folks,

 now that I finished my Research Project, it is time to tackle the Bachelor thesis. Again I would like some feedback, as it worked out pretty damn well the last time. So here is what I want to do:

0. EDIT - Too Long; Didn't Read
It was brought to my attention, that the first few paragraphs are mostly rambling. Sections 1 gives only vague background to the motivation. Feel free to skip. Here is a shorter description of the project:

Dominus (working title) is a Sandbox game all about the player teaching multiple AI agents, one at a time, to perform better at gathering resources, building a town and getting happier.
The player does so by briefly taking control (dominating) over an AI agent, showing the agent how to act by example. When control is released, the influence of the player’s behavior should be noticeable in the agent’s behavior henceforth.

1. Repetetive Tasks in Sandbox Games

I don't know how much time exactly I spent in Minecraft, shoveling through dirt and punching trees. I do know, however, that I spent over a year (online time > 400 days) in World of Warcraft mindlessly grinding for better gear. Hell, even now that Season 14 in Diablo3 began, I spend more time flinging colorful pixels over the screen to see some counters go up (gold, xp). But I am oh-so-lazy. Can't the game do that for me please?
Also, I miss my friends. I guess, I just hit that time of my life, when everybody is busy doing something new with their life in far away places. Alas, the Internet! We can game online, building something together in Terraria, Factorio, Space Engineers or 7 Days to Die. But not long after the server is set up and I am pretty much alone again. There is that one friend, that (as soon as he finds a shovel) sticks his head in the ground, only to be seen hours later with precious minerals. Another one is building a base camp on his own. Yet another friend is off screen fighting monsters. I am typically the one fiddleing around with redstone.
I know, I know. It is a cooperation-thing. Everyone assumes a role and wants to shine in that role. But in DnD we do those things together, in the same place. We have direct impact on each other, rather than doing the same thing on our own for hours and hours to complete some greater goal.
Maybe this is just the wrong kind of game for us. If we want action, why not just stick to action games? Well, after a great quest we want to return to something that is ours, that we made ourselves, home. If there is an action game with that sandbox element, like player created towns as in Ultima Online, we would probably play that. To be honest though, UO does not satisfy my friends visual needs anymore.

2. Bringing the band back together
The dream of my Sandbox experience is to, at some point, reap the spoils of the mindless tasks all of us did before. This harvest of action packed Ender Dragon slaying is often days in the future, if happening at all (no, i haven't even seen the Minecraft endboss yet). I would wish for some automated process, that I can set up in a way to my liking, as in Factorio, and let it do its thing, while I take out my friends having fun storming the castle. In my case, I prefer NPCs over conveyer tubes. I want "Hey, there is the blond guy with that yellow shirt. I shall name him Bob, and I will build him a house now!" But differently from Dwarf Fortress, I also do want direct control over what Bob is doing. I couldn't stand seeing him eaten by a Wear-Mammoth, while he is crying in the woods over his dead cat. Also, I would like to be able to show him, that he can actually carry more than one log of wood. So when he is out there lumbering away, he can bring some for his NPC-friends. He does not have to do it in the perfect way, but in my way.
And while Bob is getting firewood, and some Charlie (NPC) is baking bread, Darla (NPC) is forging armor, I can take Anna (NPC) out to a trip to the dungeon, together with a friend of mine, who took a liking to Fenya(NPC). The repetetive tasks is left to the common town folk, while my friends and I are out there having fun.

3. Games, that do the job not quite as I want them to
I already mentioned Minecraft, the origin of the problem. Everyone is doing something somewhere else.
Then there is Factorio, which tries to fix that problem with conveyer belts and automated workshops. Still, you spend the most time building more of the same. A direct cooperative effort can lead to chaos in the clockwork.
In Dwarf Fortress you can see NPCs with unique character, goals and needs. There are short-, mid- and long-term goals keeping you, as the player, busy until the inevitable Dwarfocalypse. But those Dorfs can be pretty frustrating at times.
The creature from Black and White was, at times, equally frustrating. But given enough blood, sweat and tears you were able to teach it to help you out quite a bit. And while it was doing its thing, I had fun throwing rocks the size of a beluga whale at my enemy.
In Dungeon Keeper you were able to dominate a creature, enabling direct control, turning the strategy game into an action game for a while. 

4. Occam's Razor - The holy grail is carved out of wood
This is just a Bachelors degree. I actually have to remind myself of the fact multiple times. I will simply not be able to create the one Game of games. On my own I will probably not even be able to do more than a prototype.
I looked at the Maslow Pyramid and thought "I want all of that!".
I looked at the Dwarf Fortress wiki about the production chain and though "I want all of that!"
Well, no. Just no. For the purpose of a prototype, there will be an apple-tree, a wooden hut, an axe and three or four characters that want to sleep and eat. That makes them happy, which in turn makes them pick apples, chop trees and build houses faster.
The prototype should be able to add more content later on, though. Ideally, this would happen with a wizard in the editor.

5. Technical stuff - The road so far
With the requirements seen above, a Goal-oriented Behavior-approach seems rather fitting: Give the AI system some goal and some action and it figures out how to use them on its own. One could even argue that I probably read the book Behavioral Mathematics for Game AI first, and only then decided to build a game for that. Well, okay, you got me. For me it was like seeing some cool Magic - The Gathering-card and deciding to build a deck around it. I know that, at this point, a simpler AI structure would suffice, but I am stubborn in applying what I learned in the book.
So, as Ian Millington suggests in Artificial Intelligence for Games (p403), actions are "generated by objects in the world" using a Command pattern.
As I am most comfortable with Unity3D, Coroutines are already built in the system, so I can spread out the computation of action scores over several frames.

I have sensor to determine the distance to objects. I will have sensors to determine the satisfaction of needs and for how much satisfaction an action provides.
My utility functions are eagerly waiting to have more than a Linear Interpolation.
All actions are already in the game: you can eat, sleep, pick up and drop.

There is already a pig running about, eating the closest apple it can find (including the ones in your inventory)

6. My Problem - Suggestions welcome
The working title for the game is Dominus. You do not have a player avatar, that can directly interact with the environment. Instead, you have something like a mindcontrolling Blob, that can sit on someones head and dominate a NPC, giving you direct control over the NPC and thus allowing you direct interaction. What is supposed to happen is this:
While dominating a NPC, the AI system is supposed to record what you are doing, as long as the NPC is still happy (fed and rested). When released from domination, the NPC is supposed to recreate that behavior in a similar fashion without neglecting their needs.

Example:
The base AI set up by the developer is not perfect, not even close. So, Bob is hungry and moves for 20 seconds to get an apple from a tree. He eats the apple. Bob is now sleepy. He walks 15 seconds to a hut, then goes sleeping. He is now hungry again.
What the player knows, is that it would probably have been smarter for Bob to collect some more apples, to later consume them from the inventory. So the player dominates Bob, walks to the tree, collects apples until Bob's inventory is full, walks back to the house and goes sleeping, then releases Bob.
When Bob wakes up, he will of course eat apples from his inventory. But the next time he goes apple picking, he fills the inventory to a degree, similar to what the player has shown him to do.

My Concerns:
I see no way of implementing this with Utility-based AI. It would probably need some system that automatically changes weights or functions on runtime. But how should it know, why the player did what he did? I heard that the author of Behavioral Mathematics for Game AI pretty much loathes Machine Learning, so I am interested in alternatives to the following:

Imitiation Learning seems to be capable of doing what I want. My idea would be to create the Utility-based AI first, then let the Imitation Learning algorithm imitate that behavior to not being able to distinguish one from the other. Then the Imitation Learning algorithm takes into account what the player does.

My professor is absolutly hyped about Machine Learning and wants me to do it that way. But I am sceptical. In my Research Project I even wrote (shamelessly citing myself now ...)

Quote

As fads come and go, there is great temptation to just pick the most popular model at the time. But it might be ill-fitted for the needs of a particular soon-to-be game.

I don't want to make the mistake, which I warn beginners about. So what do you guys think?
Imitation Learning? "Enhanced" Utility-based AI? Something else?

To me this sounds a bit like a discussion around the AI Wish List. Maybe there are some other resources for me to look into?

I am interested in your oppinion!

Cheers,
Rockroot

P.S. If you are interested, here is the git repository. I have to update/commit soonest. Warning: bug infestation!
Some pictures are attached.
 

My first thought is that I read the first three paragraphs and I still have no idea what it is you actually want. You're rambling a bit, trying to be conversational, but it would probably help to be concise and succinct. If you had to describe the game you want to make in a single paragraph, can you do it? Or maybe even just 1 sentence?

Section 5, "a Goal-oriented Behavior-approach seems rather fitting: Give the AI system some goal and some action and it figures out how to use them on its own" - sure, if you like. If the aim here is to have autonomous characters doing interesting things (and like I said, it's not clear to me what the aim is, given that half of your preamble seemed to be about cooperative play and grinding), then some sort of planning might work.

You mention using Coroutines to calculate action scores over various frames, but I doubt that would be necessary or beneficial. Probably the opposite.

Section 6: "While dominating a NPC, the AI system is supposed to record what you are doing, as long as the NPC is still happy (fed and rested). When released from domination, the NPC is supposed to recreate that behavior in a similar fashion without neglecting their needs." Okay, this is now straying into masters or PhD level territory. You might get away with it if you can simplify the model in which you work, and use some off-the-shelf tools to help you. It could be a machine learning classification task where it examines the data you provide it with and classify that as one of several pre-defined behaviour types. It could be a machine learning regression task where it examines the data and learns to score the needs or utility differently.

The main problem is that machine learning expects to learn based on a lot of input data, spotting the bits that are common and the bits that are 'noise', allowing it to make better predictions when it sees future data. The situation where you control an NPC is unlikely to produce anywhere enough unique samples to be able to generalise effectively, so you'd probably need something simpler.

4 hours ago, Rockroot said:

I see no way of implementing this with Utility-based AI. It would probably need some system that automatically changes weights or functions on runtime. But how should it know, why the player did what he did?

This goes back to what we were saying in the previous thread, in that it's not about choosing 'An Architecture', but about joining the tools together to do what you want. Personally I see this as a very strong candidate for utility-based AI, which by no means excludes the ability for weights to change at runtime - although deciding how to weight things, and how to change those weights, based on which criteria, is the hard part of this project.

There is no AI system in existence that can know "why the player did what he did". All you can do is look for patterns. That happens in a different part of the system to the system that chooses the next action to take.

4 hours ago, Rockroot said:

I heard that the author of Behavioral Mathematics for Game AI pretty much loathes Machine Learning, so I am interested in alternatives to the following

First up, the author of that book is the moderator of this forum, so it's probably best to let him speak for himself if he chooses rather than ascribe opinions to him!

Secondly, it's not that machine learning is something to be "loathed" - far from it - it's just that it is a tool that solves specific problems, most of which are not the problems that game AI developers have. And even when they do have a problem that potentially resolves one suited to machine learning, like yours, it's not necessarily the case that they have the right situation or data in which to use such methods. Much machine learning needs to be done "offline", outside of the game or the program in which it gets used, typically because it takes a long time to learn and requires a large data set to operate. Finally, even when machine learning is appropriate and selected, far too many people jump to "deep learning / neural networks" when there are many simpler approaches which often work.

The Imitation Learning link you provide leads to quite a complex set-up, which appears to be a neural network system running in Python/Tensorflow that communicates with Unity. So there's potentially a lot for you to have to understand there. They have a video showing that after 5 minutes of training, they can have a vehicle steering much like the player's vehicle. Great - but there are 2 big caveats here.

Firstly, they had to set up the inputs and outputs adequately so that the system can learn an optimal mapping from the former to the latter. In the Unity ML system you have to decide on all the observations, what type of data they return, when to gather them, and so on. Gather too many things, especially if some of them are irrelevant, and learning will be slow. Gather too few relevant things, and learning won't happen at all.

Secondly, the vehicle example is gathering useful data every frame, presumably the distance from and direction to obstacles, current speed, the player's current steering, etc. After 5 minutes that could be tens of thousands of useful data points. Compare that with your situation where a character action only changes every few seconds, if that. There's not much useful information there. You might be looking at needing to play for hours, or even days, before the AI could learn enough about the pattern to take over.

Personally, I wouldn't use the Unity ML system for the situation you have in mind. I don't think it's impossible to do so, but I don't think it's well-suited.

But is it possible to have some sort of AI that changes based on example? Sure. You just need to think it through and plan it out on paper before deciding how to approach it. For example, your Bob/apple/hunger/tired scenario is underspecified. You say "it would probably have been smarter for Bob to collect some more apples" - but why? He met both his needs. He was sleepy. Why is it better to pick apples when you're already sleeping, even though you're not hungry? Here, you're confusing a player's belief system with the ability of the computer to mimic a player, and these are totally separate. Ignoring the former, you might be able to get away with a very simple system where, if Bob The Player picks apples when he's tired, then it increases the importance of the Hunger drive and decreases the importance of the Tired drive. So when the AI takes over, the Eat tasks score higher utility than you might otherwise expect and the character should collect more apples.

TL;DR - not entirely sure what you want to make, but you probably need to simplify it a bit, ditch the planning aspect, and probably settle for self-adjusting weights on utilities.

1 minute ago, Kylotan said:

My first thought is that I read the first three paragraphs and I still have no idea what it is you actually want. You're rambling a bit, trying to be conversational, but it would probably help to be concise and succinct. If you had to describe the game you want to make in a single paragraph, can you do it? Or maybe even just 1 sentence?

I actually didn't read much further than that. If you want my help, you shouldn't torture me with a wall of text and stories about your difficulties staying connected to your friends. Get to the point.

19 hours ago, Kylotan said:

If you had to describe the game you want to make in a single paragraph, can you do it? Or maybe even just 1 sentence?

Dominus is a Sandbox game all about the player teaching multiple AI agents, one at a time, to perform better at gathering resources, building a town and getting happier.
The player does so by briefly taking control (dominating) over an AI agent, showing the agent how to act by example. When control is released, the influence of the player’s behavior should be noticeable in the agent’s behavior henceforth.

19 hours ago, alvaro said:

If you want my help, you shouldn't torture me with a wall of text and stories about your difficulties staying connected to your friends. Get to the point.

Sections 1 and 2 describe the motivation, why I want to make this bachelor thesis in this way. I see now that you, as helping people, do not need that information. I understand that reading this wall of text is frustrating and cumbersome. I did not mean to be disrespectful and thank you for pointing that out.
My actual problem at hand: Choosing the right technology to achieve what is aimed for.

19 hours ago, Kylotan said:

it's not clear to me what the aim is

The aim here, next to the teaching of behavior mentioned above, is to have a prototype able to scale well with new content created either by a developer or (wishful thinking, so optional) by a presumable player community.
Another reason for a Goal-oriented approach is that fulfilling a need might be several steps away.
(Grow Crops -> Make Flour -> Make Bread -> Eat)

19 hours ago, Kylotan said:

You mention using Coroutines to calculate action scores over various frames, but I doubt that would be necessary or beneficial. Probably the opposite.

Learner’s question: Why would it be maleficial? What’s so bad about Coroutines?

19 hours ago, Kylotan said:

Okay, this is now straying into masters or PhD level territory. You might get away with it if you can simplify the model in which you work, and use some off-the-shelf tools to help you.

I get that a lot. I already assumed the project to be overscoped, but I thought I might personally learn more that way. Also, I went away from trying to create a huge economy to an Apple, a Tree, a House. Is that the kind of simplification of model you speak about?
Off-the-shelf tools was UnityML Imitation Learning, but you already criticized it, too.

19 hours ago, Kylotan said:

The main problem is that machine learning expects to learn based on a lot of input data

For this reason, I intended to build a Utility-based AI as a Teacher, and make the Imitation Learning algorithm learn from that. That way I would have theoretical infinite data for the basic behavior.
But it does not fix this problem:

19 hours ago, Kylotan said:

The situation where you control an NPC is unlikely to produce anywhere enough unique samples to be able to generalise effectively

19 hours ago, Kylotan said:

Compare that with your situation where a character action only changes every few seconds, if that. There's not much useful information there. You might be looking at needing to play for hours, or even days, before the AI could learn enough about the pattern to take over.

This is exactly the reason why I was feeling skeptical about ML as a solution for the problem, but I could not lay a finger on it. Also, this insight of yours on the matter is the reason for this thread. Thank you!

19 hours ago, Kylotan said:

Here, you're confusing a player's belief system with the ability of the computer to mimic a player, and these are totally separate.

I am not sure, where my mistake here is? When do I confuse the two? And how? I was sure that the player’s belief system would be paramount to the project, so the “ability of the computer” has to be created in dependence to that.
I do see, however, that I am indeed confusing something.

 

19 hours ago, Kylotan said:

not entirely sure what you want to make

I hope I was able to clarify that.

19 hours ago, Kylotan said:

but you probably need to simplify it a bit

Where do you see potential for simplification?

19 hours ago, Kylotan said:

ditch the planning aspect

I can do that (is that the simplification?)

19 hours ago, Kylotan said:

probably settle for self-adjusting weights on utilities

Do you have any resources, you could spontaneously provide? I know, that is part of the work I am supposed to do, but it can’t hurt to ask, can it? If you shove me to a rough direction, that would awesome!


Now an apology is in order:

19 hours ago, Kylotan said:

First up, the author of that book is the moderator of this forum, so it's probably best to let him speak for himself if he chooses rather than ascribe opinions to him!

I did not mean to be disrespectful. I do see your point. I guess, I read and saw so much of him and about him that I mistakenly felt a more personal relationship than there actually is. That, in turn, tempted me to this impudent and provocative statement. It was very unprofessional and I apologize to him and to any reader. I am sorry.

Should I, in any way, misstep in proper forum etiquette, or should you seriously dislike any styling of my post (e.g. many quotes in there today), please do not hesitate to tell me, so I can learn. 

Don't worry, you're doing fine. Just try to be as concise as you can (because everyone here offers their help for free) and don't mistake people recommending against certain techniques for them actively disliking them.

I'm going to have to keep this short as I'm on the way out, but here's what I think:

Your cut-down example is a lot better. I still think it's slightly too ambitious because there are several unknowns in here, such as what the outputs from the learning algorithm are, and from there, what data you need to gather in order to inform that learning algorithm, and then, what the choice of learning algorithm is. These things aren't insurmountable but they probably involve you doing a lot of reading around the subject very quickly.

You say "Another reason for a Goal-oriented approach is that fulfilling a need might be several steps away. (Grow Crops -> Make Flour -> Make Bread -> Eat)" and this is a reasonable observation. However, you don't strictly need this. You can make the planning implicit by providing each activity separately with priorities that cause a plan to 'emerge', so to speak. A simple prioritised rule list might be:

1. If I am hungry:
   1. If I have bread, eat it, stop.
   2. If I have flour, make bread, stop.
   3. If I have crops, make flour, stop.
   4. If I have seed, grow crops, stop.
   5. Get seed

By following the rules in order, you'll see that a character with nothing in their possession will initially pick the Get Seed activity, and when that succeeds, they'll do the Grow Crops activity, etc etc. The plan is implicit in the priorities of the activities. Is it as flexible as a full planning system? No. Is it as complex as a full planning system? Again, no. I would strongly recommend that you consider whether you can create the system without a planner involved.

The idea to "to build a Utility-based AI as a Teacher, and make the Imitation Learning algorithm learn from that. That way I would have theoretical infinite data for the basic behavior" is quite a good idea, with only one flaw - there's no practical application for it. (Having said that, if you're not doing a PhD, this is less of an issue.) If you can already create the behaviour one way, there's little point trying to make a second system that can mimic it. There is, obviously, merit in making a system that can mimic a player's behaviour and then extrapolate that, but you have to start looking into the "less cool" algorithms because you're working with a lot less data. Things like N-grams, Bayesian classifiers, and Markov Chains come to mind.

The thing about a player's belief system... I don't think I explained myself well there. The point I wanted to make is that the idea that a certain choice of action might be smarter or not is entirely in the player's head and has zero bearing on what the learning system does or should do. The player's assessment might not even be correct. A system that imitates a player is going to imitate the behaviour regardless of whether the player made the 'right' choice or not. Therefore it's important not to confuse "picking the best behaviour under the circumstances" with "picking what we think a player would do under the circumstances" because the former is not relevant.

Regarding further simplification, I think that you just need to draw out the feedback loop that you're hoping to see, and then consider the technology options at each point and how they work together. If you can't answer those questions at least on a rough level then you can't code it. I would probably start with a simple utility system working based on character needs, and one action for each of the needs, and the utility function is something with a value you can tweak to make the action more or less appealing. It might be as simple as a small constant you add to an otherwise linear function which can be increased or decreased. Or it might be something like u(need) = pow(need, w) where need ranges from 0..1 and w is the weight, initialised to 1, but which can go down towards zero or up towards infinity. Your learning algorithm would check all the need levels and any other necessary context when a player chooses a behaviour and adjust the weights to make it more likely the AI would choose similarly in those conditions.

Kylotan has nailed most of it.

Another way of stating part of the problem, however, is identifying the vast amount of shit you would have to have as your knowledge base that the ML algo would have to look at. In your example, certainly one aspect is "inventory room"... i.e. "how many apples can I carry now?" However, one that is relevant but less obvious is "hut space"... i.e. "how many apples do I have back at home?" If you are already well-stocked in apples at home, you wouldn't bother filling your carrying inventory -- especially if that inventory is going to be more than you have room to unload when you get back home. 

That said, if you are hungry and standing next to the tree, you might pick one and eat it rather than returning home with to get one. At that point, the utility of time kicks in... e.g. "I'm hungry, and there is an apple right here."

But then as things get more complicated, you need more and more info. e.g. "I'm hungry and have few/no apples back in the hut. So I should pick one to eat and more to carry home... but that slows me down and I really need to get home soon to [take a pee][start dinner][watch Wheel of Fortune] so I am not going to bother picking/carrying more now. Or what if the house is surrounded by orcs? Going home to eat an apple or drop off a load doesn't seem like a good idea.

So yeah, an ML solution is great at pattern recognition and you could use the player's behavior to train it, but only if your knowledge representation layer is robust enough to keep track of what the player is using to make their own decisions.

One classic example that I've heard over and over is how the military was trying to train a system to identify NATO tanks or ships vs. Soviet ones. When deployed, any objects that were in sunny days and had clear photos were identified as NATO and anything in the rain or on hazy days or poor photo quality were identified as Soviet. Why? Because they used really purdy pictures of the NATO shit and bad spy-taken stuff of the Soviet tanks. The system simply learned how to judge the photo quality rather than the shape of the tank.

So the lesson is that your ML problem has the potential to be a rabbit hole big enough for a school bus sized rabbit. A standard ML algo like a NN will have no way for you to peer under the hood to determine what is going right or wrong. At least a utility system where you are tweaking weights in formulas, etc. would allow you to inspect and ask "so why did you think to do ASDF?". Those formulas are human readable.

My IAUS architecture is good for this (although I haven't tried to deploy it in a ML situation like this because ML is a PITA for an authorial control environment like game designer). For more on my IAUS, see this GDC lecture:

http://www.gdcvault.com/play/1021848/Building-a-Better-Centaur-AI

Hello dear AI folk,

I turned in my Bachelor (of Arts) thesis and received a grade (1.3 , where 1 == best, 5 == worst), that seems a bit odd to me. I guess, in the standard of our university it is rather fitting (a lot of people are artists, not coders), but I don't have a frame of reference in terms of where it ranks among the professionals (that is: you, dear reader)

I would be very honored, if you read (or skimmed) through the 40 pages and gave me any feedback (maybe some grade). Otherwise I hope it is fun to read.

I am not satisfied with what was accomplished in a pragmatic sense (due to time constraints), but also I am sure how I would continue and improve. I would be happy to discuss any possible future work.

The reason for the highly positive grading given by my professor was due to the pioneer character of the topic. Still, I want to see how it holds up to your standards.

I do want to improve. If you have suggestions on where to look next, do not hesitate to advise!

EDIT: I will soon post a short summary on what I believe is my biggest mistake and how I would fix it. But right now, job search has priority.

P.S. Thank you all for your previous feedback. I feel like my thesis benefited a lot from it!

ThomasHaase_ImitationLearning_Bachelor_Documentation.pdf