For debugging purposes, I tried to recreate both the top and bottom-level acceleration structures from scratch every frame so that I can use a custom transformation matrix, but it exhibits a problem with the culling. It's very strange. If I don't change the transformation matrix, and instead move to see the back of the model using the WASD keys, the problem does NOT occur.

The code is at: https://github.com/sjhalayka/sw_partial_reflectivity

A Twitter video showing the problem is:

I had to pass in the transformation matrix to the closest hit shader, which is used to transform the normals.