As I've mentioned before, I'm working on a metaverse client for Second Life / Open simulator type systems. The existing viewers have very cluttered interfaces. There's a menu bar at the top, with drop-down menus. Looks just like Microsoft Word from 20 years ago, which is when it was designed. There's also an info bar above the menu bar, with info such as where you are, how much money you have, and what you're allowed to do at your current location (talk, build, etc.)

At the bottom, there's a row of buttons, each of which brings up some pop-up menu. Then there are 2D game objects which appear atop the play area, non-modal dialog boxes, and the occasional modal dialog. Most people keep the “nearby avatars” and “local chat” non-modal dialog boxes open.

I'd like to have a much cleaner screen, so the system feels more immersive. But not change the interface so drastically that it loses existing users. So I'm considering making all the menus disappear much the way YouTube does. After a few seconds of menu nonuse, the menu bars disappear. YouTube has a nice trick you barely notice. There's a greyish gradient behind the bottom controls, which hints at the control area without blanking out the bottom part of the video. That's worth doing.

The trick is how to get the menus back. Yeah, push ESC. But I'd like to set things up so that when the cursor hits the top or bottom of the screen in full screen mode, the menus come back. This is a classic idea that's not used much. (https://en.wikipedia.org/wiki/Fitts's_law#Implications_for_UI_design)​.​

How strange is that in the game UI world? Good? Too weird?

On a related note, what's current best practice for displaying text chat? The current interface is a movable, scrollable window of small type on a light grey background. If that's not turned on, it shows up scrolling atop the main screen, with insufficient contrast. Should text chat today look more cell-phone like?

Assuming the menu is manipulated with the mouse:

• If that's all the mouse does, then blend in the menu when the player moves the mouse.
• If the mouse is also used for other things, then blend in the menu when the player moves the mouse to the menu area of the screen.

Nagle said:
How strange is that in the game UI world? Good? Too weird?

Hmm… personally i hate such automated helping hands. For example, at some point Windows proposed me a half-the-screen layout if i drag a window and touch some edge of the screen.
That's a good idea, and many people may like it. But i hate it, because i can no longer position the window precisely the way i wanted. I want to drag it to the side so it does not block my view, but OS blows it up to the whole half screen instead, and i can't prevent it from doing so. It makes me very angry each single time this happens.

That's surely a problem for many people. It could be avoided by making the helping hand optional, but disabling it by default. So the users long years habits are not broken. But those willing to improve their UI experience can find those new options in the options menu and try it out. Then, if they don't like it, they can disable it again and won't be angry at all. Contrary, i am angry, because now i have to find options to disable helping hands which were forced on me.

Beside that i can't help. My view of my 3D scene becomes smaller and smaller, because more and more UI windows need to be added. But i'm too lazy to double click their title bar to minimize them. Seemingly i prefer to have a tiny view instead.
I guess good GUI design is a problem involving many psychological aspects, like the one mentioned above, which is just one factor out of many. I am also one of the kind who thinks: Games should not need GUI at all. But that's another story.
The only thing which helps without doubt is a good initial layout, and automatically hiding windows which have no effect in the current situation.

Good points. On the other hand, having a zillion customization options creates another set of problems. Also, that sort of thing appeals more to developers than users.

JoeJ said:
automatically hiding windows which have no effect in the current situation.

That's an interesting problem here. One issue with metaverse systems is that the system has no idea what's currently relevant, because the system doesn't own the game logic. That belongs to the creators who make things in the virtual world. They have assets and a scripting language to work with. It's more like a web browser, where the browser has no idea what the web page is about.

Nagle said:
On the other hand, having a zillion customization options creates another set of problems. Also, that sort of thing appeals more to developers than users.

Depends. Designing good options so they can be found is another problem on its own, but solvable.
Bad example: Windows. It's literally impossible to find options at all.
Good example: MacOS (remembering the state 10 years ago). Here all options are in one place, reasonable divided, and pretty easy to find.
Just to point out there are good solutions too.

The big, big risk i see for you: By trying to improve, you break peoples habits. No way around that. Many will not like it, and then all your primary work on a better client is lost for them for silly reasons.
Because of that, i would make the classic layout (something pretty close to that) the default. Then, the advantages (better framerate, etc.) are the only things they see at first. They may discover new options about UI a bit later, to be surprised positively another time.

But you can point out those new options early in some form. If i launch the 3D modeling tool Modo for the first time, it asks me about my preferred kb layout: Maya, 3Ds MAx, etc.
That's very nice. They give to the option to keep my habits from competing applications. And i get comfortable with the new application much easier, without a need to dig into its options settings right at the start.

JoeJ said:
Because of that, i would make the classic layout (something pretty close to that) the default.

At least at launch. Once your client becomes widely adopted, you can change it's default settings to be more modern. Gradually, over time, so the changes are not to harsh between updates.

JoeJ said:
By trying to improve, you break peoples habits.

Right. There are people who are really upset by programs which use the arrow keys instead of WASD for movement.

Any good books to read on this? What I really need is a cheat sheet, “Controls gamers expect on desktop”.

I'm working with Rust→Rend3→WGPU, and Rust game dev is still in a rather raw state. So there aren't off the shelf solutions which have a standard way of doing things. If you use Unity or UE, there are defaults for such things.

Pick a highly popular game in the same genre, and figure out what they do.

If you're an RPG, check out Skyrim, or Witcher 3.

If you're a shooter, check out Overwatch or Apex Legends.

If you're an RTS, check out StarCraft II or Age of Empires 4.

If you're an action-adventure, check out Tomb Raider or Assassin's Creed.