Embodied manipulation: a control scheme for immersive sim gameplay

Published May 06, 2024
Advertisement

I love the controls in the original 1996 Tomb Raider. It feels like I'm touching the world. Holding down the keys for holding objects or holding on to ledges. Using the lower key for the legs and the upper key for the arms. It’s intuitive, consistent and it felt right for me.

Contextual buttons

My main frustration for controls are the contextual actions that depend on proximity to an object. I have been vaulting over campfires in Horizon more often than not just because the quicksave button is the same as the dodgeroll button. I think that the generic “action” button is bad. The action can be radically different depending on what pixel you are looking at in the case of two different targets next to each other. Games with auto-climb often make me climb over tables when I’m just wanting to go to pick something up. It’s another case of immersion-breaking when I’m with a group of companions indoors and trying to talk to them but accidentally climb and jump all over the place.

Keeping control

Most games don’t allow you to cancel, change or modify actions. You should be able to change your mind in the middle of anything you do. Never take away the control from the right stick camera or left stick movement. For example, if you tell the avatar to sit down, and that starts a 1.5 second animation, you should be able to cancel the action at any point, resulting in what would be physically reasonable. If you issue the action to drink, resulting in you reaching in your bag, taking out a flask, opening it and moving it to your mouth, you should be able to cancel that at any time. More important still; If you are knocked down during a fight, you should be able to steer yourself while falling, for the possibility to get back up on your feet faster. You should be able to grab hold of things. You should be able to roll to lessen the damage. It may be hard, but it should be possible to minimize the setback. Never take the control away from the player. Keep the basic things easy while providing lots of things to master.

More inspirations

Death Stranding uses L2 and R2 as grip buttons for the left and right hand. You can throw stuff by swinging while holding something and let go mid swing. The monkey climbing in Astro’s Playroom is wonderful, where you can reach for and grab handholds with L2 and R2, even using the motion sensor for the reaching and also for swing jumps. Uncharted 4 also lets you steer your hands when selecting where to climb. The Last Guardian use circle as the grip button, allowing you to push/pull and move with objects. The playstation symbol buttons should not be used for things where you also want to keep using the right stick. Dying Light has the right idea by using R1 to jump and holding on to ledges. Off-stick buttons should only be used for things that in-world would interfere with moving or looking around. Half Life 2 was influential in not restricting the movement while people were talking with you, even though you actually didn't say anything. Cyberpunk 2077 gives you the power to select things to say while still letting you look around and move at the same time. The quicktime events of Detroit: Become Human hints at something that could resemble embodied actions, but fails miserably. You should never have to state what button to press, other than as a reminder.

The original meaning of the playstation symbols are triangle viewpoint, square document, circle yes and cross no. With time, now they usually are used for triangle Interact, square Action, circle Evade and cross Jump.

Immersive Sim Open World Adventure Game Control Scheme

This is my suggestion for allowing many different actions but in a way that feels natural. Keep the action buttons reachable while keeping the thumbs on the sticks. Slower actions can require long-press.

Your hands

The L2 and R2 trigger buttons represent your left and right hand. The trigger buttons are analog, meaning that it registers gradual impressions from zero to one. The Playstation 5 controller triggers are also adaptive, meaning that they can resist at different positions to different degrees at different points. The adaptive trigger will create a connection between your hands and the hands of your avatar in game. You can feel when you pick something up, the force when you try to pull, push or lift something heavy, the thug when you catch hold of a ledge after a jump or fall, or the resistance while pushing open a door or pulling a lever.

Gradually pushing down the trigger will first raise your arm, ready the item in it and finally squeeze or use the item. Use the right stick to select a target and press down a trigger to reach and pick up that item. Releasing the trigger will put down the item at the place you are looking at, or just in front of you if you are not looking at a place within reach. If an item is just out of your reach, you will reach for it. Use the left stick to move or try to extend your arm towards the item if you're stuck.

The target for your hands is the item you are looking at that's within reach. The default setting would be to first subtly highlight the current target. When a trigger press starts the pickup movement, the target will be more clearly highlighted, and you will be able to cancel the action by releasing the trigger before you actually move the item.

If the target is stuck, you will place your hand on it. Stuck items include things that can’t be picked up, because they are heavy, equipped or connected to something else. You can squeeze the item by pushing the trigger to the max. Squeezing an item while looking at it will enable you to move it with your right stick.

You can move items between your hands. If you are holding R2 with an item in that hand, and then selecting that item by looking at it with the right stick, you may place your left hand on it by holding L2. Releasing R2 will move the item to the left hand. The looking part is optional if you have no other target within reach, the other hand is empty, and the item is not equipped.

Equipment

Hold square to open your inventory as a radial menu of stuff near at hand. That menu will also have options for accessing your bags and other stuff. Select the item and release. Selecting an item from your inventory while not holding up a hand will equip it directly in its place or to the right hand. Selecting an item while holding L2 or R2 will put the item in that hand. Anything already in that hand will be put in your inventory or released if not collectable.

While holding an item, you can press square to equip it. When the item is equipped, you will not drop it by releasing the hand. Equipping something that isn’t wearable will make you hold on to it with the selected hand. Equipping a two-handed item will make you hold on to it with your left hand, if you can carry it with just one hand. Equipping something you wear on your arm or hand will put it on the opposite hand. If you are holding a ring in your right hand and equipping it, you will put it on the left hand. You can select the exact placement by raising your other hand and pointing with your right stick. Equipping other wearable items will put them on your body. You should also be able to look at all parts of your body by looking down on yourself.

Looting

While not holding L2 or R2, pressing circle will make you collect the item you are looking at, and put it in your inventory. You may hold circle as a shortcut for unlocking and opening containers.

Looking at an item within reach and holding L2 or R2 will pick it up in that hand. If the item is stuck, you will place your hand on the item. Having your hand on an item and pressing circle will make you try to unequip it and take it in your hand. For example, you would take your shoes off by looking down at them, hold L2 and R2 to place your hands on them, and then press circle to untie them and place them in your hand.

If not having your hand on an item, press circle while holding L2 or R2 to put the items you are holding in that hand in your inventory. Those items are always quickly reachable from your inventory by holding square.

Shooting

Let's say you see a bow and arrows on a table. Look at the bow and hold R2 to pick it up. While holding R2, you can move it to your left hand by holding L2 and releasing R2. Having an item in your hand means that you use it, if it's the kind of item you use by holding it in your hand. You may also equip it with square so that you will not drop it by releasing the trigger.

Now hold R2 for picking up the arrows. While holding the arrows, press circle to store them in your inventory, or hold L2 to raise your bow while storing all but one of the arrows. Then hold R2 to draw the bow with one of your arrows. Release R2 to shoot an arrow.

You can throw an unequipped item by holding it up, hold L1 to aim and release L1 to throw the item. If the item is equipped, you must also release the hand to actually let go of the item during the throw. You don’t have to do the manual release or even the holding up of the hand if the equipped item has a stash of throwable ammunition. For a stash of equipped rocks, you may just hold down and release L1 to do a throw, as long as you have a free hand.

Having a rifle equipped means that you will raise it with L2 and use it with R2, using the same principles as for everything else. Gradually pressing R2 will raise your arm to the trigger. The adaptive trigger will push back at the position of the trigger, where the squeeze will fire the gun.

Reload the guns by releasing R2, look at the left hand and press square, meaning that you hold the gun in the left hand and use the right hand to re-equip it. If no other target within reach is selected, you may just press square while holding up the gun. Though you still need an empty hand.

This will also allow having one item in each hand, be it single-handed guns, melee weapons, throwables, or any other type of item, like a torch, compass, rope, or a brass key.

Melee

Holding L2 or R2 will block with whatever you have in your left or right hand. Time pressing L2 or R2 to parry. All of these are variations of raising your left or right arm.

Melee weapons can be used by squeezing. Hold squeeze for a strong attack. Modify the type of attack by the direction you move with the left stick and the movement of your pointing with the right stick, for swinging to the left, right, up, down. If your opponent parried your attack, you may follow up with a riposte by pressing L1.

Use L1 to force your movement. L1 while moving towards a target within reach will kick or tackle the target. L1 in a direction with no target will do a short dodge in that direction. L1 while standing still will apply force towards the target without moving. Do L1 without a target for a nice grunt.

Crouching

Press cross to crouch. Hold cross to go prone. Press cross near cover to hug cover. Then cross again to crouch if standing behind cover. Press cross while crouching to duck. Hold cross while prone to hold your breath and play dead.

While prone, you can press R1 to go back to a crouching position or hold R1 to stand up. While crouching and standing still you can press or hold R1 to raise up.

Pressing against an incline while crouching or prone and holding R1 will make you try to climb while crouching or prone.

Moving crouching or prone will be silent while not in sprint mode, meaning that your avatar will try to not do something that would make noises.

Jumping

Pressing R1 will jump. Combine with L1 for different jump-attacks. Hold R1 while standing still to charge a standing jump. R1 + left forward in any direction will jump in that direction. R1 + left sideways or backwards will dodge and duck in that direction.

Press R1 + left while crouching to roll, or R1 + cross + left to roll while running. This will work in any combination. Begin jumping in a direction and press cross before your feet left the ground to modify a basic jump ro a roll, or press cross while in air to roll after you land. Press cross while in air to pull your legs up for clearing obstacles.

Combining moves with cross will take you away from the right stick, but I think that is ok since it's something that should be a bit harder and would limit your awareness of your surrounding, since you would focus on the ground.

Climbing

Hold R1 while running at obstacles for parkouring over them. This will work for everything where you don’t have to hold on with your hands. Bracing or pushing off with your hands can be done even while holding items, as long as they aren’t fragile or heavy. You will parkour the thing you look at as long as you keep up your momentum. This will also work for things like ladders that are especially easy to climb.

For climbing with your hands, make sure to have them free. Look at something you can hold on to and hold L2 or R2 for grepping with the left or right hand. With your free hand, look at the next handhold. Move your body and arm with left and grab it. Release the other hand and repeat for the next handhold. You can use square to lock your latest grip if you want to hang on without holding down the trigger.

Your feet will follow along using the nearest footholds while climbing with your hands. To climb over an edge, hold on to the edge with both hands, move the left stick upward to press your body up and then hold R1 to climb up with your legs.

Press L1 for using your arms to thrust your body to something out of reach. While reaching for a handhold, press R1 to jump in that direction. Jumping while not reaching for or holding on to the environment will let you jump backwards.

You can climb down over an edge by looking at the edge and using an empty hand. Holding the ledge, you can continue to move forward to drop down and turn around.

Balance

You will pause at edges, but can continue for jumping down if it's a comfortable drop. For larger drops, you will have to initiate climbing or switch to sprint mode. This will also help you with balancing on stocks. You will not step of the ledge, but can still lose your balance.

You can center your mass to regain balance by raising your left or right arm. The degree of press on the trigger correspond to how high your arms are raised. Looking down at the stock or ledge and pressing a trigger will take hold on the ledge. You may also crouch or go prone to keep your balance.

Dragging

Use L2 + R2 for putting your hands on the target item in front of you and move forward to push. If you have a grip, you can also drag backward or rotate to the side. Hold L1 to try to lift the object from the ground, or L1 + left to force the item in that direction.

Ropes

Hold square to open your inventory as a radial menu of stuff near at hand. Select your rope with the left stick and release square to place the rope in your left hand. Ready the rope by holding L2. For binding to something, look at the item within reach and hold R2 and press square. Cancel the binding by releasing R2 and/or L2 before finished. Untie the rope by pressing circle.

Pull the item by moving backwards while holding R2. Moving while not holding R2 will just let the rope out til the limit of its reach. Hold L1 + left to pull the rope while standing still.

Hold R1 + left for climbing up/down the rope. While hanging on the rope with R2, use movement direction to run on walls. Press R1 to jump against the wall with your feets. Release R2 to let go of the rope, for a swinging jump.

Look at the rope and press square to lock yourself to the rope, in case you want to use your hands for something else. Press circle to free yourself again.

For throwing a lasso or grappling hook or similar, you can hold a rope with your left hand and throw the part in your right hand by holding and releasing L1. Press circle to collect the rope back to your hand and circle again to put it back in your inventory.

Widgets

While looking at a stuck item in reach, you can hold a trigger to raise your arm and place your hand at the item. While your hand is placed, use right to move it left, right, up and down. left will still control your walking, giving you the option to move forward and backward or to the sides.

Squeeze the trigger to grab hold on the item. Use your sticks to move the item in any direction. Combine with L1 to jank the item with force, or use L1 by itself to just punch your target.

Push buttons by squeezing them. Levers can be pulled by squeezing them and moving with the sticks. Wheels can be rotated by rotating the right stick. You may try to loosen it by holding L1 and loot it with circle.

The shortcut for opening a container is holding down circle. Not using shortcuts, you will open a chest by placing your hand on the lid and move it with the sticks. Unlocking a chest would be done by equipping the key to your right hand, placing your left hand on the lock and then raising your right hand. Insert the key by squeezing R2 and then rotate it with right.

You can open doors in a similar way, or do it quickly by pressing L1 while you are pressing up against the door.

Inspecting

Press R3 to inspect your target. It will mark up situational relevant things in your environment representing the awareness and knowledge of your avatar, lingering for a while like short-term memory.

The thing you are looking at with right is your target. Inspecting will display more in-depth information of the current target, including tips for available actions and how the thing relates to your quests and places you know.

Inspecting your hand will let you rotate it with left, including the item you are holding. Holding the item in your left hand allows you to manipulate it with your right hand.

Crafting

Hold square to get your tool from the menu. Hold the thing you want to modify in your left hand and process it with your right hand by holding R2, and directing your action with your right stick. You may for example put a knife in your right hand and take hold of a tree branch with your left hand, for making an arrow or spear.

Place things together by releasing an item you're holding while looking at the other item. Attach one item to another by having the thing you want to attach equipped to the right hand, place the left hand at the target or hold the target with your left hand, and then look at the target while you raise your right hand and press square.

Talking

The triangle is used for mind-stuff, like dialogue, quest-tracking or magic. Hold triangle to select what to do from a radial menu for fast action or go one step further for a more comprehensive list of options.

Careful or Fast

Press L3 to toggle sprint mode.

While in normal mode, you will behave civilized among people, actively avoiding bumping into things or people and avoiding climbing on stuff just by walking towards things. You will also refrain yourself from stepping over cliff edges and falling to your death.

Tilting left in normal mode will increase the movement to running after about two seconds. Bumping up against obstacles while not running will make you stop for a second and then find a reasonable way to continue to the place you’re looking at, by moving in a direction similar to the direction of left. You will go around obstacles if you can avoid them by one or two steps in either direction. If there's no way to step over or around the obstacle, you will walk over it.

By entering sprint mode, you will have to take responsibility for your own safety. You may still choose your actual speed by how much you move left, but the maximal speed will be much greater and you will also change your speed faster. You will automatically step over or press through anything that comes in your way. The main difference is your speed and lack of safety net and that you don’t care if you break things in your way by pushing them to the side.

You will still have to manually jump or climb over things that reach over your waist. You must still use L2 + R2 to grab hold of edges if needed during a jump. Engaging in combat in sprint mode will trade speed for increased vulnerability and faster stamina drain.

Previous Entry Health in games
0 likes 11 comments

Comments

Aressera

This is all very specific to console games using gamepad controllers. What about mouse/keyboard, which is the optimal input method for first person games?

Have you played any VR games? VR allows the player to directly interact with objects in the scene, since we have direct control over the placement of hands and the view direction.

May 06, 2024 10:26 PM
aigan

@Aressera
This is close to what could be done in a VR game. Expecially with PSVR2 that incorporates the adaptive triggers and haptic feedback.

Mouslook is great and something you especially want if fast precision is wanted. Even more so in a PvP game. But the other aspects of the Dualsense controller is so good that its worh it. So it's not so much about 1st person but rather how important it is to be able to move the camera fast and precise. Example: Thief is a 1st person game that works well with a controller.

Look at the control scheme and try to imagine it. Most game doesn't use the analog rigt and left trigger as well as they could. The mouse is one analoge input. The controler has the two sticks and the two triggers. The dual sense hase a couple of more on top of that.

If you haven't you should really really find a way to play Astro's Playroom to get an idea of what is possible.

May 07, 2024 07:47 PM
JoeJ

What about mouse/keyboard, which is the optimal input method for first person games?

The mouse is optimal, yes.
But the kb is not. Those WASD keys are not ergonomic. Left Hand comes from the left, so the W button should be shifted to the right for a ergonomic angle, if at all, but it's shifted to the right.
World was much better when we still used the cursor keys.
And most importantly: We want an analog joystick for movement, not those shitty binary 4 way buttons.
So i ask you: MS is making more money from games than from Windows, they say.
If so, why can they enforce totally useless new buttons to the world two times (first the windows button which i never ever used, and now the copilot button which i will never use either even less),
but they can not come up with putting a little analog stick on each kb, to make games better?

Or, a very similar question: Why does Sony put a fucking touchpad on their controller, which would work just fine to give us mouselook, but they put it on the center of the controller, so not even Paul Gilbert could reach it with his thumb? Why?

How comes that those companies manage to generate the impression to be actually innovative? They are not. They are stuck, block their own way, and they have no balls. >:D

May 07, 2024 06:47 AM
aigan

@JoeJ
but they can not come up with putting a little analog stick on each kb, to make games better?

You mean like?

Or rather something with just the left part for leaving the right side for the mouse.

The steamdeck has touchpad for an alternative to mouslook, but I prefer a real mouse to touchpads.

The touchpad on the Dualshock and dualsense are obviously not a substitute for the camera control stick. It should be used for navigating menues. Especially for games made for mouse navigation. But it's not been used that much.
There is probably a reason, but games COULD implement look on the right stick the same as for the mouse, where the degree is coupled directly to the tilt rather than having it coupled to turn speed.

It would be good to have the option to use a touchpad on the right side also, but thats another input scheeme that games would have to suport, so they are sort of stuck.

But except for that part, I think my control scheme, for all the other inputs, are good.

May 07, 2024 08:03 PM
JoeJ

@aigan OMG, a kb monster. Fits well into the gaming room in the basement, right beside the big tower with shiny LEDs inside, water cooling, and the box of a 4090 in a glass showcase. : /

I meant just a single stick, but did not think about where to put it so you could press some keys as well.
What i really wanted would be a one handed controller for the left hand with one stick and some buttons.
Or just something like this:

Never tried to play with a trackball, though.
But i have tried a Steam controller. Not as good as a mouse, but acceptable.
Sadly, still many of those new PC gaming handhelds ship without a touch pad.

but games COULD implement look on the right stick the same as for the mouse, where the degree is coupled directly to the tilt rather than having it coupled to turn speed.

No way. The problem is that any stick tilts back to the center, so it can never implement proper mouselook. Whatever you do, it's a level of indirection killing immersion.
We need mouse, touch pad or trackball. Or any game plays like shit.
I may be more pedantic than others, but - no, i'm not. : )

But except for that part, I think my control scheme, for all the other inputs, are good.

Maybe, but i think it's really specific to the game.
On one hand, we really want similar controls for any game, so we don't need to learn how to play each one individually.
But on the other hand, this rules out innovative and new games eventually.
It's bad if people start to ignore those, just because they don't match their habits.

Just look at gamepads. If that's not an example of bad habits ruining everything, then idk. :D

May 08, 2024 06:55 AM
aigan

@JoeJ The problem is that any stick tilts back to the center, so it can never implement proper mouselook.

It would not be the same but something similar can be done. The stick will still have less precision.

Since the stick tilts back, you can do it as in Detroit Become Human where you will look at the sides and when look back to the front when you release the stick. The left stick can be used to actually turn rather than strife.

Another way is to use degree look for most of the positions but then do turning at the edges, so you will continue to turn, making it a hybrid.

Another way is to lock the new position either by the button or by holding the position a third second, and that would then ignore when the stick goes back to the center, allowing you to continue in the chosen direction or again look from the center. Several games has a button for “center camera", that can be used for the inverse thing in this case of settig the new direction.

It will confuse a lot of people being used with “normal” controls. And it's still not as high precision as a mouse. Many people push the stick to the edge instinctively and don't know about how to use gradual movement. Some games take that into consideration and does things like lowering the max speed in indoor settings just because players don't know or care about walking slowely.

It will also be a problem with stick drift. There need to be a per-controller global setting for adjusting dead zones and related settings. Playstation doesn't have that and it's left to each individual game.

May 09, 2024 12:00 PM
aigan

@JoeJ We need mouse, touch pad or trackball. Or any game plays like shit.

It's also a skill issue. Everyone who used both will feel that they can't aim with a stick. But you can get used to it and you will get better at it.

And there are compensations sutch as slowind down the action, lowering the difficulty or using aim assist (by adjusting the direction of the fiering, increasing hit boxes or snapping to positions).

May 09, 2024 12:07 PM
aigan

@joej i think it's really specific to the game

This is for an immersive sim adventure game. The design here is for something that will work for mostly all 1st or 3rd person action-adventure games. Even if it's built to be super-consistent for thousands different things you may want to do in the game, it still match up with things that can feel familiar. Like aiming with left trigger and shooting with right trigger. But that is done not as a specifik aim and shoot mechanic but rather something that comes from the more generic design.

May 09, 2024 05:33 PM
JoeJ

VR allows the player to directly interact with objects in the scene

Haha, no. Real interaction goes both ways.
No haptic feedback - no impression of interaction. : )

May 07, 2024 06:52 AM
JoeJ

Since the stick tilts back, you can do it as in Detroit Become Human where you will look at the sides and when look back to the front when you release the stick. The left stick can be used to actually turn rather than strife.

So you need one stick to orient your head, a second to turn your character, and a third to move your character.
Too bad i only have two hands. But some aliens might have fun.

Another way is to use degree look for most of the positions but then do turning at the edges, so you will continue to turn, making it a hybrid.

But you never know in advance when you hit the edge with the stick.
Beside the constant pullback force, which makes it hard to hold a position precisely, that's a big reason why sticks just suck in general, and PC players are quite fine with their binary 4-way movement.

Another way is to lock the new position either by the button

That's clumsy. You constantly have to pause your looking while locking with the button.
Bit since it's similar for the mouse (we need to lift it up sometimes), that's maybe the best of your proposals.

But if any of this would be better than the standard controls, they would have converged at that already. The problem can't be solved at all with software.

They should have solved it when the iPhone came out. Nintendo did it. On 3DS you can play Metroid using mouselook by moving a pen on a touchscreen. It's as good as Quake with mouse.

The best experience i've ever had with playing a 3D game on a console. And actual the only one at all til yet.

I think the reasoning here is: On consoles, you sit far away from the screen, so your real world fov is tiny, and immersion is not really possible anyway. Thus, shitty controllers which prevent immersion are acceptable, players are used to them, so let's stick at it.
However, screens get bigger and bigger. Zak McCrackens living room is no longer science fiction, but widespread reality.
This should force console makers to rethink their bad decisions about controllers from decades ago.
But obviously they are too busy with buying studios just to close them afterwards, or imitating Hollywood.

May 10, 2024 06:33 AM
aigan

@JoeJ

So you need one stick to orient your head, a second to turn your character, and a third to move your character.

Left stick for walking. Replace strafing with turning. This is similar to what the did in Detroit. … Again. Its unusual, but could work well for adventure games. Not typical FPS.

But you never know in advance when you hit the edge with the stick.

That would take about five seconds to get used to. And could be enhanced with feedback of the turn curve and even more with haptic feedback.

That's clumsy. You constantly have to pause your looking while locking with the button.

Bit since it's similar for the mouse (we need to lift it up sometimes), that's maybe the best of your proposals.

Just remembered that another common method is to have a mode for free-look. So a dedicated button for adjustning the camera is rather common.

But if any of this would be better than the standard controls, they would have converged at that already. The problem can't be solved at all with software.

I don't experience the right stick look mechaninc as a problem. Have gotten used to it.

Don't believe in that the controls converge to what is best because if that was true, the (X) or (A) button would not be used for jumping. Its like qwerty. Something got hold because it was part of an older technology that became popular.

Backward compatability makes it hard to switch to something better.

May 17, 2024 06:50 PM
You must log in to join the conversation.
Don't have a GameDev.net account? Sign up!
Profile
Author
Advertisement
Advertisement