Hey guys.

Something weird is going with my Vulkan shadow mapping. I already have it working on DX, and today I finally took the time to implement it in Vulkan. Or so I thought. This is what I see when shining a spot light on a sea buoy model in a test level.

I spent quite some time checking and double checking all values before looking at it in RenderDoc. I verifed that the pipeline state is what it should be and all input and output resources are what I expect. At some point I looked at the texture viewer at the actual output and, lo and behold, it actually renders correctly in RenderDoc. However, it still is broken when rendered by the actual game even with RenderDoc attached and the incorrect result even shows up in the little thumbnail RD generates for the capture. Here you can see it magically working when looked at through the texture viewer:

Any ideas what I might be doing wrong? Anyone recognize the weird artifacts? Any idea why it would show up correctly in RenderDoc?

Cheers!

An idea unsupported by any specific knowledge or experience, I'd be looking carefully at synchronisation, I wonder whether something is clobbering your shadow map before you're done with it?

It could be a synchronization issue I guess. Some of the artifact pixels flicker slightly which gives me the illusion of garbage memory but I honestly don't know.

Before I render to the shadow map I do an image transition with:

barrier.oldLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
barrier.newLayout = VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL;
barrier.srcAccessMask = VK_ACCESS_SHADER_READ_BIT;
barrier.dstAccessMask = VK_ACCESS_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT;
sourceStage = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT;
destinationStage = VK_PIPELINE_STAGE_LATE_FRAGMENT_TESTS_BIT;

And after the shadow map has been rendered I transition it to a shader input with:

barrier.oldLayout = VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL;
barrier.newLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
barrier.srcAccessMask = VK_ACCESS_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT;
barrier.dstAccessMask = VK_ACCESS_SHADER_READ_BIT;
sourceStage = VK_PIPELINE_STAGE_LATE_FRAGMENT_TESTS_BIT;
destinationStage = VK_PIPELINE_STAGE_FRAGMENT_SHADER_BIT;

AFAIK those should do the trick. Am I mistaken?

I also tried adding a second barrier with VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT as the source stage and VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT as the destination stage after each transition, which AFAIK should flush the whole pipeline.

Even after all this, the output looks the same. Ideas? Thanks for your time!

I had a similar issue as well. It wasn't working when I run normally and was working when I launch from renderdoc. As mentioned, it was related to synchronization. It seems like renderdoc inserts some synchronization itself so changes the behavior. 

Well, it took a few nights of pulling out my hair and double, triple and quadruple checking my synchronization code to realize that I had initialized the depth attachment of the shadow map render pass with storeOp = VK_ATTACHMENT_STORE_OP_DONT_CARE, ie. the contents of the depth buffer were discarded as soon as the render pass ended. Changing the store op to VK_ATTACHMENT_STORE_OP_STORE fixed the problem.

RenderDoc apparently changes the store op so that it can visualize the depth buffer, which made it work when RD was attached. It is interesting to see what happens with the depth buffer contents if you don't store them. Rather than being complete garbage you can still clearly see the geometry in the depth buffer.

Here you can see the proper depth buffer contents on the left and what happens on my GTX 980 if you init the render pass attachment storeOP with VK_ATTACHMENT_STORE_OP_DONT_CARE, on the right.

On the bright side, I found a few gnarly things in the code I fixed while looking for this...

Glad you figured it out, I only just had a chance to check back and was about to reply that your image barrier looks okay, but I guess you know that now:)