Pigeon Devlog #3: Ray Tracing + Compute in Vulkan

Hello again! 

Over the last two weeks, I focused on implementing the ray tracing pipeline in Pigeon and converting my shadow mapping feature to ray-traced shadows. I also added another important feature to Pigeon: a compute pipeline. With compute shaders, I blurred the ray-traced shadows so they don’t look like sharp edges. Finally, I added an ambient term to the shadow, which makes the result look more realistic. 

Ray Tracing Pipeline

A ray tracing pipeline is a rendering pipeline where the main “work unit” is a ray, not a triangle. Instead of rasterizing triangles to pixels, we sending rays into the scene and ask: what do we hit first? Then we use that hit information to calculate shading.

In Vulkan ray tracing, the pipeline is built around a few shader stages:

• Ray Generation Shader: This is the entry point. It decides which rays to send (camera rays, shadow rays, reflection rays, etc.).
• Miss Shader: Runs when the ray hits nothing. Usually returns background color, sky, or “no shadow”.
• Hit Shaders: Run when the ray intersects geometry
• Closest Hit Shader: Runs for the closest intersection and is used for shading.
• (Optional) Any Hit Shader: Can run for every intersection (useful for alpha-tested geometry like leaves).
• (Optional) Intersection Shader: Used for custom geometry.

To make ray tracing fast, the scene is stored in acceleration structures:

• BLAS (Bottom-Level Acceleration Structure): built from meshes.
• TLAS (Top-Level Acceleration Structure): built from instances of BLAS (TODO)

At a high level, the flow looks like this:

1. Build BLAS/TLAS.
2. Create the ray tracing pipeline and shader binding table (SBT).
3. Dispatch ray tracing work from RayGen.
4. Rays traverse the TLAS/BLAS, and the GPU calls miss/hit shaders automatically.

The mental model is simple: RayGen send rays → traversal finds intersections → miss/hit shaders decide the result.

Shadow with RT Pipeline

Ray Traced Shadow

In Pigeon, I used the ray tracing pipeline to compute shadows by sending shadow rays. A shadow ray is a ray from a surface point toward a light source. If something blocks that ray, the point is in shadow.

The key idea is:

• If the ray reaches the light without hitting anything → lit
• If it hits any geometry before reaching the light → shadowed

How I structured it in Pigeon

I used these shaders:

1) Ray Generation Shader: This shader runs for each pixel and decides the shadow visibility. The RayGen shader:

• Reads the world position (and sometimes the normal) of the visible surface.
• Computes the light direction:
• For a point light: dir = normalize(lightPos - worldPos)
• Offsets the start point slightly along the normal to avoid self-intersection (shadow acne):
• origin = worldPos + normal * smallBias
• Traces a ray toward the light with a maximum distance given by a constant

Then it writes the result to a shadow texture:

• 1.0 = fully visible (not in shadow)
• 0.0 = fully blocked (in shadow)

2) Miss Shader: If the shadow ray hits nothing, that means nothing blocks the light:

• Set payload to “visible”.

So the miss shader is basically: no hit → light is visible.

3) Closest Hit Shader If the shadow ray hits something, we know the light is blocked:

• Set payload to “occluded”.

So the closest hit shader is: hit → in shadow.

A shadow is literally an occluded light path.

Compute Pipeline

A compute pipeline is a GPU pipeline that runs compute shaders, which are general-purpose programs on the GPU. Unlike graphics pipelines (vertex/fragment) or ray tracing pipelines (raygen/hit/miss), compute is not tied to drawing triangles or tracing rays. It’s just dispatch many threads and do parallel work on buffers/textures.

This makes compute perfect for post-processing steps like:

• blur filters
• denoising
• edge detection
• tone mapping
• particle simulation etc.

In Pigeon, I used compute for a post-process pass after the ray-traced shadow result was generated. I blurred the shadow texture to make the shadows look softer.

Blur with Compute (Gaussian-like blur kernel)

The first issue I noticed was that ray-traced shadows can look too sharp, especially if you use a simple hard visibility result (0 or 1). In real life, shadows often have softer edges because of indirect lighting.

Shadow texture before blur pass

To make the shadows look better, I blurred the shadow texture using a compute shader.

A blur is basically a weighted average of neighboring pixels. For each pixel, we sample nearby pixels and combine them:

• Samples closer to the center should have higher weight
• Samples farther should have lower weight

A Gaussian blur is a common choice because it produces a very smooth natural result. 

I used a simple 3x3 Gaussian kernel with weights:

| 0.0625 | 0.125  | 0.0625 |

| 0.125   | 0.25    | 0.125   |

| 0.0625 | 0.125  | 0.0625 |

and with one pass of this kernel, we can achieve a nice blur effect.

Implementation steps

This shader does the following:

1. It defines weights for the center, sides, and corners of the kernel.
2. It calculates the UV coordinates for the current pixel based on the global invocation ID.
3. It samples the input shadow texture at the 3x3 neighborhood around the current pixel, applying the defined weights.
4. It accumulates the weighted samples into a final color.
5. It writes the blurred result to the output image.

Shadow texture after blur pass

Ambient Light Term for Shadow

After blur, the shadows looked smoother, but there was still one visual problem: fully shadowed areas can become too dark when you only use direct light visibility. In real scenes, shadows are rarely pitch-black because light bounces around (indirect light) and the environment contributes some illumination.

Scene without Ambient Term

To approximate that, I added an ambient term inside my final pass with 0.5 value.

A simple formula is:

• finalShadow = ambient + visibility * (1 - ambient)

So:

Scene with Ambient Term

• If visibility is 0 (fully shadowed), the result becomes ambient (not totally black)
• If visibility is 1 (fully lit), the result becomes full light

This small change makes the scene feel more natural, because the shadow areas still keep

some detail instead of becoming completely black.