In this first draft, many components are extremely simplistic. For example, a reasonable fireball doesn't "target enemy by life force": it flies straight where the wizards points a finger, up to a certain range. Identifying "enemies" and detecting "life forces" are whole difficult sub-spells.

Learning runes and assembled spells is the heart of the system. The most tactical depth would come from easy spell research, easy access to variants of a known spell in particular. For example, a wizard who normally casts modest fireballs at the expense of his fatigue "should" be able to improvise the free and powerful magma pool fireballs in the example whenever he is in the right place: otherwise a volcano isn't going to be a special location, and the flexibility of the spell system is going to remain theoretical.

I suggest making knowledge of some runes incompatible (in the examples, binding demons vs animating the golem with true life is a good candidate) for flavour and strategy reasons, and maybe making some important runes exclusive to character classes (e.g. a "shaman" using spirits and a "channeler" using elemental forces).

Nice to see people thinking about unconventional spell systems. Pity you rarely see those implemented in games.

Here's my two cents on this matter:
You can think of spells as pieces of a puzzle, where you need to connect them together in order to produce the desired effect. For that to work, spells themselves need to have properties that can be affected by other spells as well as be able to affect other spells. What this scheme would allow is for example to create a "chained fire+lighting" spell out of a basic fire spell that connects to the chained lighting spell.

Obviously the spells aren't the only ones to have attributes, and spell attributes aren't the only thing other spells should affect, so the player character as well as the target can have certain attributes that are affected by the spells. By the way, the target here doesn't really need to be another character or monster.

You don't see them implemented because they're incredibly complex on design first, let alone implementation.
Two additional examples on how to graphically implement programs.
Bureau of steam engineering. Very similar in notion to what we tried to do albeit fairly simplified, especially in terms of output. Please note that our problem is fairly more complicated as we have to figure out much more complex effects.
Manufactoria. Effectively implements nearly arbitrary programs with incredibly simple constructs (relative jump branches limited to +1, write, non-conditional jump +1). Note this is more limited than most assembler instruction sets and the limitation of having to deal with geometric layout without arbitrary jumps is often a limiting factor. It's going to be difficult. Again, this game is limited in establishing output complexity: you have to match its requirement rather than the other way around.

If you're going for high magic, then you might want to consider countermagic and metamagic spells. ;)

I think that anyway if the system is very complex, it could ruin the gameplay.

Also, if you know such magic grammar, and you can create anything (in theory), this also ruins the expectation when you hunt for special spells scrolls.

You may want to look at Drakan on PS2, it uses a dial-a-spell system (four symbols/buttons in certain combinations) to uses magic which would allow a level of skill based play.
They could be something that has to be committed to memory or can be stored in a spell book. Spells for instant use or for storing power. Button combinations based on logic/theory - prefixes or suffices to spells to dictate use.

Also, GTA:San Andreas skill/player system was reliant on constant action to improve skill level. The more you do something, the better you get at it, the less you use it, the weaker it gets. Some skills are directly related-improve one, lose in the other.

I read some of your post OP and it kinda reminds me of Avatar: The Last Airbender

If you haven't seen it, it's a great cartoon series and I've always thought it could make an awesome RPG. I recommend checking it out if you're ever looking for spell system concepts

I think that anyway if the system is very complex, it could ruin the gameplay.

Also, if you know such magic grammar, and you can create anything (in theory), this also ruins the expectation when you hunt for special spells scrolls.

I think the opposite. For more "casual" magic users they could just buy/find spells and use them, e.g. bind a scroll to a hotkey or select from inventory and cast. The depth is intended for people who are interested in the topic and want to experiment, or even just save money by not buying scrolls.

In case I expained it wrongly, you'd start off not knowing any magical runes (the building block of spells). You'd learn them from scrolls etc. There would still be special scrolls out there with runes you'd never seen before which would enable you to build new classes of spells. So you'd maybe need to buy/obtain a strategic 5% of all scrolls to get all the runes you needed, but which 5%?

I read some of your post OP and it kinda reminds me of Avatar: The Last Airbender

If you haven't seen it, it's a great cartoon series and I've always thought it could make an awesome RPG. I recommend checking it out if you're ever looking for spell system concepts

I've seen occasional episodes, the timing on TV isn't good for me. ;) Could you explain their system in broad brush strokes?

I've seen occasional episodes, the timing on TV isn't good for me. ;) Could you explain their system in broad brush strokes?

Sure, in the show there are some people who are born with the ability to control one of the elements (earth, water, fire, and air). They are called "benders" and they develop and hone their skills through both martial arts techniques and sort of spiritual/emotional control. Air is a more graceful, peace with nature kind of bending, while fire is more emotional and energetic. But anyway, they have to use the elements around them to bend like you said with fire spells.