NodeBox for OpenGL » Shader

NodeBox module nodebox.graphics.shader offers functionality to create custom, hardware-accelerated image filters. This functionality might not be supported on older graphics hardware. In that case, nodebox.graphics.shader.SUPPORTED will be False.

Shader

An OpenGL shader is a pixel effect (e.g. blur, fog, glow) executed on the GPU (Graphics Processing Unit). It is therefore very fast. Shaders are written in GLSL, a variant of the C programming language.

The Shader object is a Python wrapper for GLSL source code, which you have to write yourself. It compiles the source code, installs the pixel effect and provides an easy mechanism to set variables in the source code from within Python. It expects two distinct parts of GLSL code: the vertex shader and the fragment shader.

• GLSL vertex shader: determines the coordinates of the current pixel to process.
• GLSL fragment shader: manipulates the color of the current pixel.

shader = Shader(vertex=DEFAULT, fragment=DEFAULT)

shader.source                              # Tuple of (vertex, shader)-source.
shader.variables                           # Dictionary of (name, value)-items.
shader.active                              # True => shader is being applied.

shader.get(name)                           # Returns the variable's value.
shader.set(name, value)                    # Sets the variable's value.
shader.push()
shader.pop()

• Shader.get() sets the value of the variable with the given name in the GLSL source code.
  Supported values: vec2(), vec3(), vec4(), int, float, or a list of int or float.
• Shader.push() activates the pixel effect with the current variables.
  When image() is called between Shader.push() and Shader.pop(), the image will be drawn with the effect applied to it.
• Shader.pop() deactivates the pixel effect.
• ShaderError will be raised if the GLSL source code fails to compile.

For example, below is a basic effect that colorizes the pixels in an image. It only uses a fragment shader (e.g. how the color of each pixel is modified) – the default vertex shader is sufficient for most effects. Note how the source code defines three variables: src, color, bias. The src does not need to be set explicitly, it will contain whatever image is currently being drawn.

from nodebox.graphics.shader import Shader, vec4

colorizer = Shader(fragment='''
uniform sampler2D src;
uniform vec4 color;
uniform vec4 bias;
void main() {
    vec4 p = texture2D(src, gl_TexCoord[0].xy);
    gl_FragColor = clamp(p * color + bias, 0.0, 1.0);
}''')

To make the shader work, values for color and bias need to be set. This is in fact very useful since it allows the shader to be manipulated from the outside, have different visual effects based on different values.

colorizer.set("color", vec4(1, 0.5, 0.5, 1))
colorizer.set("bias", vec4(0, 0, 0, 0))
colorizer.push()
image("creature.png", 0, 0)
colorizer.pop()

GLSL introduction

Below is a short list of essential GLSL syntax. The DEFAULT_VERTEX_SHADER and DEFAULT_FRAGMENT_SHADER offer a glance of how these statements interrelate. Full documentation for the OpenGL Shading Language can be obtained from: http://www.opengl.org/documentation/glsl/

If no value is passed to the vertex or fragment parameter of Shader, a default script is used: 

DEFAULT_VERTEX_SHADER = '''
void main() { 
    gl_TexCoord[0] = gl_MultiTexCoord0; 
    gl_Position = ftransform(); 
}'''

DEFAULT_FRAGMENT_SHADER = '''
uniform sampler2D src;
void main() {
    gl_FragColor = texture2D(src, gl_TexCoord[0].xy);
}''' 

A Shader initialized with these values will have no effect: the fragment shader simply retrieves the color for each pixel and passes it on without modification.

Note: if you plan to mix the pixels of two images (i.e. image compositing), take a look at glsl_compositing and the Compositing filter in the shader.py module. This filter can be subclassed (see the AlphaMask filter for example) and might save you a lot of work.

References: lighthouse3d.com

Filter

The Filter object is a high-level wrapper for Shader with the responsibility to correctly initialize the necessary variables. It can be passed to the optional filter parameter of the image() command. Because GLSL is not as forgiving as Python, setting the variables can be tricky (and tiresome). If you plan to release a custom pixel effect, consider subclassing Filter instead of just writing the Shader. Users will appreciate it.

filter = Filter()

filter.shader                              # Shader used.
filter.texture                             # Texture to apply the shader to.

filter.push()                              # Set variables and call Shader.push().
filter.pop()

For example, here is a wrapper for the above colorizer shader that is more user-friendly (it can be passed to image() and works with Color objects so users don't need to know about vec4).

class Colorize(Filter):
    
    def __init__(self, texture, color=Color(1,1,1,1), bias=Color(0,0,0,0)):
        self.shader  = colorizer           # Create a Shader once and reuse it.
        self.texture = texture
        self.color   = color
        self.bias    = bias
        
    def push(self):
        self.shader.set("color", vec4(*self.color))
        self.shader.set("bias", vec4(*self.bias))
        self.shader.push()

image("creature.png", filter=Colorize(color=(1, 0.5, 0.5, 1)))

You may wonder why Filter has a texture property if it is not passed to the shader. Indeed, the shader will simply use whatever image is currently drawn, but we may need to know what image that is to set the variables correctly (e.g. a blur effect needs to know about the image size, and there is no way to retrieve the size inside GLSL if we don't pass the information as a variable ourselves).

Filter helper functions

The module has a number of helper commands that are useful when defining variables in a Filter.

The vec2(), vec3() and vec4() commands take float numbers and can be passed to Shader.set() to emulate the GLSL vec2, vec3 and vec4 data types.

vec2(f1, f2) 

vec3(f1, f2, f3)

vec4(f1, f2, f3, f4)

ceil2(x)

extent2(texture)

ratio2(texture1, texture2) 

The ceil2() command returns the nearest power of 2 that is higher than x, e.g. 700 => 1024.

The extent2() command returns the extent of the image data (0.0-1.0, 0.0-1.0) inside its texture owner. Textures have a size power of 2 (512, 1024, ...), but the actual image can be smaller. For example: a 400 x 250 image will be loaded in a 512 x 256 texture. Its extent is (0.78, 0.98), the remainder of the texture is transparent. 

The ratio2() command returns the size ratio (0.0-1.0, 0.0-1.0) of two texture owners.

OffscreenBuffer

The OffscreenBuffer object can be used to draw into a hidden texture instead of directly to the canvas. It relies on the OpenGL FBO (Frame Buffer Object) which may not be available on older graphics hardware. In that case, an OffscreenBufferError will be raised.

buffer = OffscreenBuffer(width, height)

buffer.texture
buffer.active 

buffer.push()
buffer.pop()
buffer.clear()
buffer.reset(width=None, height=None)

• OffscreenBuffer.push() activates the buffer.
  Between OffscreenBuffer.push() and OffscreenBuffer.pop(), all drawing is done offscreen into OffscreenBuffer.texture. The buffer has its own transformation state, so any translate(), rotate() etc. does not affect the onscreen canvas.
• OffscreenBuffer.clear() clears the current contents in the buffer.
• OffscreenBuffer.reset() creates a new texture of the given size in which to draw.

Instead of drawing an image with a filter, we may want to apply one or more filters to an image and return it as a new image. A buffer is useful to chain multiple filters together. This is what happens behind the scenes in the render() and filter() commands.

For example, here is a function that uses the above Colorize class to create a function that returns a new colorized image:

def colorize(img, color=(1,1,1,1), bias=(0,0,0,0)):
    buffer = OffscreenBuffer(img.texture.width, img.texture.height)
    buffer.push()
    image(img, filter=Colorize(color, bias))
    buffer.pop()
    return img.copy(texture=buffer.texture)

img = colorize(Image("creature.png"), color=(1, 0.5, 0.5, 1))

Buffers with a transparent background into which transparent images are blended behave poorly. The functionality in the example above is already bundled in the filter() command, together with specific calls to glBlendFuncSeparate() to produce better transparency effects. You are therefore advised to use filter() instead of the above example:

def colorize(img, color=(1,1,1,1), bias=(0,0,0,0)):
    return filter(img, Colorize(color, bias))

img = colorize(Image("creature.png"), color=(1, 0.5, 0.5, 1))

References: gamedev.net (2006), openframeworks.cc (2009)