Optical Flow

To enhance the possibilities to apply external velocity to the fluid solver, PixelFlow contains an optical flow module.

Based on the idea of simple frame-differencing (image1 – image2), the optical flow of subsequent images (Frames-Animation, Movie, WebcamCapture, etc…), can be computed from 3 quantities:

dt – difference of frames (frame_A – frame_B)
dx – sum of horizontal grandients (sobelX_A + sobelX_B)
dy – sum of vertical gradients (sobelY_A + sobelY_B)
The normalized vector(dx, dy) is basically already the flow-vector. dt defines the length and direction (positive or negative).

There are several ways to generate the gradients. I got decent results with using the Sobel-Operator. Since everything can be done on the GPU using shaders the overhead is minimal.

Optical Flow Algorithm
update step
preprocessing (blur, noise-reduction, grayscale/color, etc…) – optional
gradients (sobel, horizontal+vertical)
optical flow (GLSL-Shader)
smooth (gaussian blur + temporal smooth) – optional
Most obviously, the input images can (and should) be preprocessed. It depends a lot on the source-footage if its worth to apply more expensive filters (median, bilateral filter, …) and/or simply a gaussian blur.
A threshold, to omit flow-vectors, of a certain length.
Gauss-bluring of the resulting flow and finally some temporal smoothing. (mixing the current and the previous flow)
GLSL-Shader – Optical Flow

  • PixelFlow | Copyright (C) 2016 Thomas Diewald – http://thomasdiewald.com

version 150

precision mediump float;
precision mediump int;

define SUM_RGB(v) ((v).r + (v).g + (v).b)

out vec2 glFragColor;

uniform sampler2D tex_curr_frame ; // current image
uniform sampler2D tex_prev_frame ; // previous image
uniform sampler2D tex_curr_sobelH; // current gradient horizontal
uniform sampler2D tex_curr_sobelV; // current gradient vertical
uniform sampler2D tex_prev_sobelH; // previous gradient horizontal
uniform sampler2D tex_prev_sobelV; // previous gradient vertical

uniform vec2 wh; // size rendertarget
uniform float scale; // scale flow
uniform float threshold = 1.0; // flow intensity threshold

void main(){

vec2 posn = gl_FragCoord.xy / wh;

// dt, dx, dy
vec3 dt_ = texture(tex_curr_frame , posn).rgb –
texture(tex_prev_frame , posn).rgb;
vec3 dx_ = texture(tex_curr_sobelH, posn).rgb +
texture(tex_prev_sobelH, posn).rgb;
vec3 dy_ = texture(tex_curr_sobelV, posn).rgb +
texture(tex_prev_sobelV, posn).rgb;

// sum rgb-channels
float dt = SUM_RGB(dt_), dx = SUM_RGB(dx_), dy = SUM_RGB(dy_);
// gradient length
float dd = sqrt(dxdx + dydy + 1.0);
// optical flow
vec2 flow = scale * dt * vec2(dx, dy) / dd;

// threshold
float len_old = sqrt(flow.xflow.x + flow.yflow.y + 0.00001);
float len_new = max(len_old – threshold, 0.0);
flow = (len_new * flow)/len_old;

glFragColor = flow;

Leave a Reply

Your email address will not be published. Required fields are marked *