This is an idea I had.

Let's start by peeking at a simplified pipeline -

First, we render the Viewed From Light Shadow Maps that will contain the Z from the depth buffer. So far, as usual for shadow mapping.

Next, comes the first pass that is part of my idea. It executes a pixel shader, because we need the Zbuffer.

 

Shadow Map is sampled here for the first pass of the Path Tracing. It is explained why, later.
“Hit x,y,z” has simply the visible part of the projected geometry. Instead of saving the color, it saves the x,y,z of the texels that survived the Z ordering.

“Next x,y,z” is generated by using the normal of the survived texels. “Next” takes the “Hit x,y,z”(Texel x,y,z) and mirrors the ray from the camera to the new(next) direction. In the case of a mirror, this is straightforward thing to do. Later you will see how it is done for a regular texture with roughness. The new(Next) ray is saved to VRAM as the unit x,y,z of the vector.

So far, in VRAM we have a hit or a bounce of a ray. Start → center → end.
Meanwhile, the shadow mapping provides in this pass a simple shadowing. This shadowing could be already sent to monitor. It already is ready to see. But in the miliseconds we still have we will try to make it more realistic by tracing more rays around. 

And here comes the compute pass -
 

 It really performs the classical ray tracing operations. On the Start → Hit→ End that the pixel shader computed. It checks if in between Hit and End there is some triangle that obstructs the ray.
We make sure there is a dome in the game and any ray eventually hits something. The x,y,z of the pixel the ray hit is stored in another resource for the next pass. At the same time, using the normal of the new hit we can get in the same shader the direction of the next ray to trace.

The number of compute shaders executed depends of the number of ray to path-trace.

In the case of rough textures, textures with alpha and so on, the compute sahders just generate more “Next rays”. This uses more memory and computation :(((
 

Eventually, we can discard any pixel that falls under a direct light. Most of the times, visually it will make not much difference if adding or not path tracing to already lightened up by direct rays areas. In the case of the first/pixel shader, this is shown here:

 

We can see here the shadow mapping logic from the first/pixel shader pass.
 

The next/compute shader pass adds Ray Tracing:

 

It is the easier place to see RT and SM working together. Right after finding the hit, we check if it is lit directly by a light by asking the Shadow Map. The principle is the same - we transform the real world position of the collision texel to the view of the light and check if the distance to the light is more or less than the Z stored in the Shadow Map texture. Just a regular Shadow Mapping that is executed again, but here - after the first Ray Tracing. If it is directly hit by direct light, we can interrupt the path here too, in order to save computation. If we will keep path tracing, the ray extra ray from SM will contribute to the final color of the texel(the color in the origin of the path).

You can see how for the first two bounces. The first projection in the first pixel shader is checking rays if they intersect with geometry. It is Ray Tracing done by projection. In the middle is the classical Ray Tracing done by iterating over the triangles inside a BHV to find the closest hit. And the last ray is traced by the shadow mapping. If that last ray intersects with something, the Shadow Map Texture will give negative.

Like the next sketch shows, it is basically path tracing, but at every hit, we introduce an extra ray that is for free and comes from shadow mapping. It is just a read and compare. It is a valid Ray Tracing, but precomputed by the Shadow Mapping.
 

 Because the computing is performed in the following order -

 

we need to have in VRAM the information showed next:

 

The layout of this information is distributed over various resources, but virtually, this is the struct of data per path.

The extra light sample is when the max depth(max amount of bounces) is consumed, but we can still get some more light info. The last shader could produce a last ray(or not produce a last ray) and test how much this ray is looking into the light source. It is a trick that gives approximated results. (when there is a next ray generated but no more ray tracing(Ray/Triangle intersection tests) will be performed)

 

The way the light affects this last ray can be tricked too - 
 

When we read an unit Ray(Next x,y,z), we find ourselves upon the situation of very few workload for a lot of threads. To deal with that, we can use the extensions of AMD to pile workload into a flat array or into the corner of a texture. This way the CUs will have more work. Otherwise, more than half of the threads will have not a NextRay to work on(after the second bounce directly lightened texels should be discarded for real. Visually speaking, the deepest in the Path, the less the light contributes(statistically speaking, there could be exceptions of some Death Star laser beam, but for this case, in a game, i personally would tend to skip work)).

 

  
Although it is clear how it works, the real implementation in practice, could differ in the details. A LOT of work needs to be done to take the sketches to the real life implementation. So, many things could change.

If we generate only one ray from each hit, having one extra ray for free from Shadow Mapping is doubling the quality of the result. For free. The more rays we generate from a hit, the less noticeable will be the improvement that Shadow Mapping gives us.

Because of the fact that, after the first shadow mapping we already have an image that looks great enough for many games, we can just interrupt all the process of path tracing at any moment, "blend" the final color of the path and show it to the gamer. If the scene has few triangles and transparencies, the time window will allow us to compute deeper paths, but if the player turns his head around and looks into a forest, after the shadow mapping, we will get close to the end of the time window and we will abort it all and show shadow mapping forest without a single ray traced.

(Notice a serious overkill in the case of using transparent textures. SM + Alpha = a hell of layers. I started to build the pipeline in DX, but the transparencies of textures made me abandon the project(and the lack of motivation).)
(When I say “texel” i mean “point” or "the smallest voxel possible”. I mean a point in 3D space. Not a pixel taken form a texture. A point in space with a meaning, with a texture/material applied to it)

I am sorry for bumping this, but i think the Last Extra ray makes no sense at all, because if it happens inside a closed room it can leak sunlight from the outdoors. Sorry for that. I wanted to edit the OP, but i will just leave it there to hurt me.

And i found a simpler way to explain it:

“Every time in your code you generate rays from a bounce you have a free ray(one per light source) coming from SM. Its direction is arbitrary, out of your control. It depends of the position of the light at that frame of the rendering.”

 

And it doesn't need to be a compute RT. It could combine hardware acceleration RT and SM.

Fetching SMs to shade hit points is common practice for both games but also offline rendering. I guess if one enables RT reflections in UE4, the reflections would show SM artifacts here and there to proof this. Also BF5, the first RTX game probably used its SMs for this purpose. (But not sure.)

So i'm afraid to say your idea - although well thought - is not new. I remember using SMs in offline raytracers 2 decades ago.

It is however not guaranteed using SMs gives a speed up. SMs have a cost to generate, and it's not worth it if you fetch only 5 samples from it later. Another problem is frustum culling lights would no longer work because you would need their SMs to shade out of view hitpoints. So you either need to generate SMs for all lights or fall beck to tracing shadow rays for the lights that miss them. Depending on scene, the latter is more practical ofc.

NikiTo said:
ventually, we can discard any pixel that falls under a direct light. Most of the times, visually it will make not much difference if adding or not path tracing to already lightened up by direct rays areas.

The contribution of a single direct light may be only a fraction of total irradiance, so terminating the path after any direct lighting has been found would likely cause bad artifacts.
To fix this, one would need to adjust probability factors accordingly.

It's an interesting idea, reminding me on the Russian Roulette path termination technique. (That's the kinda opposite, terminating paths if they transport only little amount of light.)
This may be a new idea and worth of further research! :D

I see i reinvented the wheel again :((((

Now that i think it better, You can add 100% more to already 100% of intensity,… if you add a different color. So, nope, no ray cancelling nowhere. Let the hardware win its price!


Terminating paths if they transport little amount of light is doubtful too. Imagine the last ray from the path is just turning around the corner to feed on the direct sunlight. The intense color will back propagate to the camera. How do you know how much a path contribute if you haven't walked the path to the end(it sounded philosophically).


Really, if there were a way to solve transparency impact on performance and on memory and on my nerves, i could try to code it again.
I do want nice adding-up transparency for the edges of hair lashes. I am going nowhere without my precious transparencies!

NikiTo said:
Now that i think it better, You can add 100% more to already 100% of intensity,… if you add a different color. So, nope, no ray cancelling nowhere. Let the hardware win its price!

There is no upper bound on radiance at all. It can be zero or more. But if you know the contribution will bring the pixel to white appearance, terminating the ray is an interesting idea. It would hurt something like bloom filter, but…

NikiTo said:
Terminating paths if they transport little amount of light is doubtful too. Imagine the last ray from the path is just turning around the corner to feed on the direct sunlight. The intense color will back propagate to the camera. How do you know how much a path contribute if you haven't walked the path to the end(it sounded philosophically).

…notice what you say is true in any case, except the material is vanta black :)
If you decide to use constant 10 bounces, it could happen the 11th. bounce would hit a bright light. But this is exactly why probability is important in path tracing - it guarantees the image converges towards the correct solution. And as often this guarantee is enough, even if it may appear weak when considering a special case example like you just did.
The point is: Other paths or sample will capture the direct sunlight you mention, even if you terminate paths.

But it's necessary to weight contributions to leverage early termination. I have not implemented Russian Roulette yet, so i don't know how this works.
Is it as easy as… assuming max 10 bounces, and we terminate after 6, multiply the contribution by 10/6? I don't know - maybe we need to factor in material reflectance as well. Maybe somebody else could help out…
(Probability stuff turns out pretty hard to learn for me, lacking the necessary math background)

NikiTo said:
Really, if there were a way to solve transparency impact on performance and on memory and on my nerves, i could try to code it again. I do want nice adding-up transparency for the edges of hair lashes. I am going nowhere without my precious transparencies!

You can see RTX struggles too here. Transparency is always an issue, but RT can at least handle it 'in an elegant way'.

Though, if it's just about hair lashes… it should not take such a big hit?

Also, alpha mask for lashes or leafs is not really transparency, just a ‘wasted’ triangle test.

However, i think transparency will remain a special case requiring hacks for a very long time.

Hemisphere light contribution is a sorcery that is beyond my understanding.

This is how i handle the hairlashes:

This is how a hairlash opacity texture looks. On the level of a pixel, not triangle. 

 

White is completely transparent, this is completely discarded by an initial shader. The black area is completely solid, it is projected by an initial shader along any solid triangles. But it is all at a pixel level. So far, as usual.

Grey area contains pixels that have an alpha other than 0.0 and 1.0. 

I have seen people to reject alpha > 0.5 and accept alpha < 0.5. But visual tests show that this way, the edge throws a dented pixelated shadow. You can blur that, but it depends of the setup. At far distances, <0.5< approach without blurring is enough.

In order the shadows of a single hairlash to be perfect, the alpha of the hairlash is projected on the target. This way the shadow the edge of the hair throws will look perfect. Simply perfect!

But what if there are two hairlashes on top of the other. Just like it happens with real hair in real life?

 

 The resources of 2GB VRAM and WebGL allowed me to pile the alpha of 4 hairlashes to get a shadow. I think it is enough.

Notice, we are talking here not about what is correct, but what looks nice to the eye. We compile it, run, move it around, look at it, and decide if it looks nice. Math is secondary. The wrong math could look better than the correct math.

For the lower ends of the skirts of princesses we need nice alpha too. If i can not render a beautiful princess skirt properly, i am not gonna lose 6 moths with DX.

Shortly said, transparency doubles the programming effort and rendering times. Doubles or triples or worst.

And tessellated lines for hairs is not even an option for RT.

I agree that, if a GPU becomes more advanced at resources management, brute forcing the rays will be the most logical solution. Like when you see out there blogs like: “Ray Tracer in only 100 lines”. The same concept but for GPU. Copy paste and go! But GPUs are not there yet.

You could solve this stochastically, like pretty much anything with RT.

So when the ray hits the hairlash triangle you sample the alpha. Then generate a random number between 0 and 1.
If the random number is larger than alpha, reflect the ray like on a usual opaque surface. If it's smaller, let the ray go through the triangle and treat it as non existant.
So there is no handling of transparency at all. But if you have a larger number of samples, the average of all of them will correctly resolve the transparency you want to see.

The same works for handling reflection vs. refraction (so full transparency), or diffuse vs. specular.

Another problem that become very easy this way would be a spherical area light.
Here you select a random point on the surface of the sphere each time you test visibility using a shadow ray, and after many samples mix together you get correct penumbra.

The only problem here is to use correct probabilities. For the hair lash it's easy, for the sphere light it's already harder.
A very hard case is then for example a PBR material. You want to generate your ‘random’ rays in a way the sum of all light thy capture correctly represents the reflectance functions of the material, so more uniform or chaotic directions if you hit the surface along it's normal, but mirror like reflection if you hit it at shallow angles to capture fresnel effect. You could also use random hemisphere directions always and weight them accordingly, which is easier, but generating a direction contribution that that already represent the reflectance function is much faster ('importance sampling').

Now you might think ‘nice, but i can not trace 100 paths per pixel to have enough samples for an acceptable result in realtime’. But this is the problem denoising tries to solve, by using samples form nearby pixels and previous frames as well.

Some more examples of seemingly hard problems that become easy to solve with stochastic RT are:
DOF, motion blur (not that easy but still), and even continuous LOD from discrete detail levels of models - achievable by randomly switching discrete LODs while traversing the ray.

NikiTo said:
Hemisphere light contribution is a sorcery that is beyond my understanding.

It's not hard if we restrict ourselves to perfect diffuse materials (lambert).
To get correct results we have two options, as mentioned above:

1. Generate completely random directions in the hemisphere defined from surface normal, and weight the incoming light from all paths by dot(surfaceNormal, rayDirection) ('cosine law').
To be clear, if you imagine to trace many paths form the first primary ray hit at each pixel, it's like this: radiance = surface color * accumulate (all paths irradiance * their individual weight) / number of paths

2. Instead weighting, generate ray directions in a way the direction distribution fits the cosine formula from above.
This is pretty simple: Create a unit circle on the normal plane, generate a random point in the resulting disc, move the point along the normal so its distance from origin becomes one.
This forms a distribution of rays that looks like a half sphere, but there are more rays towards the normal than on the plane - exactly the distribution we want.
Summing up the samples than becomes: radiance = surface color * accumulate (all paths irradiance) / number of paths
The weight is gone. But the real benefit is that we need less rays for the same image quality.

At this point we have a simple path tracer with importance sampling, and it's not hard to do.