Custom shaders

Note

If you haven’t already done so, make sure you’ve completed the steps in Basic before starting this tutorial. This tutorial will be based on SAPIEN assets and rendering.

Setup

For this tutorial, we will start from the following code. Do remember replacing the token with your access token.

import sapien.core as sapien
from sapien.core import Pose
import numpy as np
from PIL import Image, ImageColor
import open3d
from sapien.asset import download_partnet_mobility

my_token = "Your Access Token"
sapien_assets_id = 179
urdf = download_partnet_mobility(sapien_assets_id, token=my_token)

sim = sapien.Engine()
renderer = sapien.VulkanRenderer(False)
sim.set_renderer(renderer)
scene = sim.create_scene()
scene.set_timestep(1 / 60)

scene.set_ambient_light([0.5, 0.5, 0.5])
scene.set_shadow_light([0, 1, -1], [0.5, 0.5, 0.5])
scene.add_point_light([1, 2, 2], [1, 1, 1])
scene.add_point_light([1, -2, 2], [1, 1, 1])
scene.add_point_light([-1, 0, 1], [1, 1, 1])

controller = sapien.VulkanController(renderer)
controller.set_current_scene(scene)
controller.set_free_camera_position(-3, 0, 0)

loader = scene.create_urdf_loader()
loader.fix_root_link = 1
asset = loader.load_kinematic(urdf)
assert asset, "No SAPIEN asset is loaded"

while not controller.is_closed:
    scene.step()
    scene.update_render()
    controller.render()

controller = None
scene = None

Use custom shaders

Vulkan renderer uses the spir-v shader format, which can be compiled from other shader languages. Here we will introduce how to compile glsl shaders and use them in SAPIEN. First download the original shader pack original.zip and unzip.

This renderer uses a deferred shading pipeline with forward transparency pass. Files starting with gbuffer processes opaque geometry to create gbuffer textures. Files starting with deferred computes lighting from the gbuffer textures. Files starting with transparency computes another forward transparency pass after deferred shading. Files starting with axis are for axis drawing in the GUI. Files staring with composite are also for displaying to the GUI window.

To compile glsl to spir-v, you will need a tool called glslc. After installing glslc you just need to run glslc -c * in the shader folder containing glsl files to compile them into spir-v shader files. Now to use the shaders, you just need to set the shader path for the VulkanRenderer immediately after its creation.

renderer = sapien.VulkanRenderer(False)
renderer.set_shader_dir("shader")

Custom shader: world normal

Now let’s try to add the functionality of getting world normal instead of camera space normal. You need to edit the file gbuffer.vert. Change the line objectCoord = pos; as follows.

  // objectCoord = pos;
  objectCoord = normalize(mat3(transpose(inverse(objectUBO.modelMatrix))) * normal);

You can see, previously, we are storing the local coordinates of the object into the shader variable objectCoord. This variable eventually shows up in the custom texture. Now we just change it to show world normal.

Now in the SAPIEN controller GUI, if you switch to the “custom” display mode. You can see world normals on the object. You can compare it with the camera-space normal shown in the “normal” display mode. If you have a mounted camera, you can get this texture by calling camera.get_custom_rgba().

Per-object data

SAPIEN also supports passing per-object constant into the shader. This enables per-object dynamic effects.

First, to pass in custom data, we need to call set_custom_data on an VisualBody. For any Actor, we call get_visual_bodies to get all visual bodies associated with this actor. The function set_custom_data should always send in a list of 16 numbers representing a 4x4 matrix, but you do not have to use it as a matrix.

For example, the following code sends in increasing values from 0 to 1 into every visual body of the loaded articulation.

idx = 0
values = np.linspace(0, 1)

while not controller.is_closed:
    scene.step()
    scene.update_render()
    controller.render()

    for link in asset.get_base_links():
        visual_bodies = link.get_visual_bodies()
        for v in visual_bodies:
            v.set_custom_data([values[idx % len(values)]] + [0] * 15)
    idx += 1

Now let’s adjust the shaders to display some light intensity changes. First we make the following changes to the gbuffer.vert file.

// layout(location = 3) out vec3 objectCoord;
layout(location = 3) out float intensity;

  // objectCoord = pos;
  intensity = objectUBO.userData[0][0];

Here we pass a intensity value from the per-object user data into the fragment shader. Now in fragment shader gbuffer.frag, we modify the albedo computation by

  if (material.hasColorTexture != 0) {
    outAlbedo = texture(colorTexture, inUV) * intensity;
  } else {
    outAlbedo = material.baseColor * intensity;
  }

Now you should see a chair with changing light intensity.