Hi everyone, I'm programming a robot for SimSpark simulation (check out robocup simulation league at www.robocup.org). Anyway I want to teach the simulated robot how to walk, but I don't know how to use learning algorithms like Q-learning (being relatively new to these fields). I wonder if you could help me, for example how can I formulate a long term reward function to prevent the robot from falling? I really don't know how to do it when a full walk takes at least 20 cycles and I have to update the Q value for 20 times... Yours, KSP

Q-learning is just another type of iteration processing update, by introducing the rewards and discount rate at a given state.

For your case, the goal is not to fall down, so it is up to you how to define a 'fall-down', 'going-to-fall', or even 'steady' state.

Yes and indeed, you need to take 20 cycles for full walk and you need to check it for precision purpose, or perhaps if you wish to deduce the algorithm consumed times, you can make every 5 cycles check for that every-5th state of the movement.

So if your Nth-movement makes your robot's condition good, then you reward it by giving for say '1 point' for the reward, vice versa -1 for punishment.

Thanks danong,
Do you know about any other algorithms that I can use for walking, or other places to look?(google didn't really work!) I'm new to AI and I have to do this programming so Q-learning was the only thing I was aware of...

Yours,
KSP

if you are an engineering student, i'm pretty sure you can apply what's called 'kinematic' law to your movement of the robots.

it really depens on how you want to model your framework,
if you wish some advanced topics such as deterministic walking framework, you would have to dive into some areas such as inverse-kinematic, physics law of retrieving center-of-gravity of the body, calculating balancing point, and many more.

however, this is not always the case for novice programmer,
you could just simply end up modelling your walking structure frame by frame (sometimes refer to as 'forward kinematic') and perhaps applying some adjustment to stabilize the movements.

try google up 'inverse' & 'forward' kinematic, i'm pretty sure there re lots of resources lecturing these topics.