Hi,

I'm designing a new shader system for my engine - so I decided to checkout how others do it... I stumbled uppon Hodgman's response, I saw the Bitsquid's presentation about their Data-driven renderer and finally FrameGraph (D.A.G. applied to yet another thing ). And... now I'm overloaded with informations...

Initially I thought that I've understanded Hodgman's post so I tried to wrap my shader into Passes, and those into Techniques. Since I know Lua pretty well, I also decided to use it for custom "effect" files:

Shader "common" [[
    // GLSL snippet
]]

Shader "simpleVertexShader" [[
    // GLSL snippet
]]

Shader "simpleFragShader" [[
    // Another GLSL snippet
]]


ShaderProgram "simpleProgram" {
  vertex_shader = { "common", "simpleVertexShader" },
  fragment_shader = { "simpleFragShader" }
}

ShaderProgram "simplerProgram" {
  options = { maxLights=2, cheapApproximations=true },
  vertex_shader = { "common", "simpleVertexShader" },
  fragment_shader = { "simpleFragShader" }
}

Technique "normal" {
  { name="opaque_pass", layer="opaque", program="simpleProgram" },
}

Technique "selected" {
  { name="outline_pass", layer="opaque", program="simplerProgram", uniforms={ outlineColor={0.5, 0.1, 0.1, 1.0} } },
  { name="opaque", layer="opaque", program="simpleProgram" }
}

So basically a Technique is a container for TechniquePasses, each Pass has a name, a shading program and is assigned to a Layer. A layer is a special place in a pipeline, and it has its own RenderQueue and optional profiling scope (e.g. "shadow mapping"). My tought process was as follows: I can have multiple Pipelines (Forward, Shadow, Deferred, Postprocess, Raytracing). Pipelines are divided in stages ("layers") that run at different times and they enforce the ultimate render order (the connections between pipelines builds RenderPaths my version of FrameGraph). TechniquePasses are also guaranted to execute in declaration order (not exactly one right after another, but the order won't be changed - I reserved some bits in the sortkey for pass index). During rendering I look up the Material, it gives me a Technique (the Material-Technique connection is actually baked offline), then I iterate over it's passes submitting drawables to respective renderqueues (layers).

In theory a "Toon Shading" rendering technique should be pipeline-independent (it should be possible to use it forward and deferred). So does "Phong" and "BlinnPhong", but there might be issues with Deferred Pipelines (different G-Buffer layouts in some cases)...

I started implementing it, but it went to far for my taste - It feels too much coupled (high-level rendering mixed with low-level) and with high "mental tax". Swapping Techniques at runtime is hard and the overall complexity just skyrocketed... and this should be low-level render wrapper

So taking 2 steps back - what actually is a "Technique"? Should it describe how to render a material in different pipelines (e.g. "DepthOnly", "Deferred", "Forward"), shading methods ("Phong", "BlinnPhong", "PBR" techniques) or more basic variations ("selected", "nightVision", "normal", "glowing" - currently I'm using such variations as shader options). Up to this point I always imagined a "Technique" as a dumb container for passes for either in the same stage (like in toon shading: contour pass + normal pass) or for special effects like some kind of magic shield (opaque + transparent passes), but they are not that popular these days. Well, I believe the last time I've used them was when XNA was becoming popular.

And one more question: should the Shading System allow for technique swaps at run-time? I'm starting to think, that it should be static - in order to make Materials less complicated (no dynamic lookups, single UBO/cbuffer layout, etc.)

Sorry, but I have to bump this thread. I would really love to hear any feedback on whats the good (if not the best) way to define a "Technique". I'm started to think that a Technique should implement a shader stage ("DepthOnly", "Deferred", "Forward") - but I'm still not confident / have mixed feelings about this.
Thanks

There's no right way to structure this, the right answer is whatever abstraction helps you out as a graphics programmer
I can answer how my abstraction works though --

On 8/27/2017 at 7:44 AM, malboro said:

So basically a Technique is a container for TechniquePasses, each Pass has a name, a shading program and is assigned to a Layer.

I've got the same, except my "layers" are called "stages", and they're not visible to the shader author. The shader author simply declares pass names/ids (opaque, alpha, z-only, gbuffer-attrib, etc). Later on pipelines can create a list of stages/layers, which declare which pass ID they will collect. e.g. for cascaded shadow maps, you might create 4 shadow-generation stages, which are all set to collect z-only draw-items.

On 8/27/2017 at 7:44 AM, malboro said:

So taking 2 steps back - what actually is a "Technique"?

They're the shader abstraction that's exposed to artists; artists pick a single technique to use for a material. It then has to be a container of shader programs that covers every possible rendering situation that a bit of geometry may find itself in.
Other abstractions have called this concept an "Effect" and the different rendering algorithms that it contains would be called "techniques"... So my usage of the terminology is a bit wrong if you're used to those other abstractions.

On 8/27/2017 at 7:44 AM, malboro said:

And one more question: should the Shading System allow for technique swaps at run-time?

I don't allow it, and haven't needed it. Artists pick a single technique for a mesh, and then the mesh compilation/import code assumes that the mesh will only be used by that technique, making it unsafe to swap to another technique (e.g. required vertex attributes may not be available / the cbuffer layout required by the material may be different / etc). If you need to change how the scene is rendered, you can use a different set of stages/layers, which in turn selects a different set of passes, or, you can tweak individual objects in the scene by using technique-options, which allow selection of different program permutations.

OMG, look who replied - I'm sorry for this fanboy moment I just want to thank you for ALL of your input here. I've even become a little paranoid - since all those great sites like AltDevBlogADay tend to disappear quickly and wrote a tool that backups all your replies for my grand-grandsons I hope one day I will be as good as you (and several others at this forum).

13 hours ago, Hodgman said:

The shader author simply declares pass names/ids (opaque, alpha, z-only, gbuffer-attrib, etc). Later on pipelines can create a list of stages/layers, which declare which pass ID they will collect. e.g. for cascaded shadow maps, you might create 4 shadow-generation stages, which are all set to collect z-only draw-items.

Yes, this is far more better since some passes might be shared between different stages - allowing us to precompute/optimize the command buffer. Also this is way more decoupled.

13 hours ago, Hodgman said:

Other abstractions have called this concept an "Effect" and the different rendering algorithms that it contains would be called "techniques"... So my usage of the terminology is a bit wrong if you're used to those other abstractions.

Exactly, I'm more familiar with this setup - I believe it was also used in old (deprecated now) DirectX Effect Framework. As a side note - I implemented such approach in my previous engine and it was a nightmare (I even deferred the effect+technique+pass lookup up to the render loop (facepalm)) - this is the reason I wanted to design it from scratch in a more sensible way.

13 hours ago, Hodgman said:

I don't allow it, and haven't needed it. Artists pick a single technique for a mesh, and then the mesh compilation/import code assumes that the mesh will only be used by that technique, making it unsafe to swap to another technique

Now I'm thinking I wanted to give too much flexibility (and try to solve problems that does not exist yet)... I will go the same path as you (as always ) since I like preprocessing/offline baking as much as possible.

Thank You soo much