To enhance the possibilities to apply external velocity to the fluid solver, PixelFlow contains an optical flow module.

Based on the idea of simple frame-differencing (image1 – image2), the optical flow of subsequent images (Frames-Animation, Movie, WebcamCapture, etc…), can be computed from 3 quantities:

dt – difference of frames (frame_A – frame_B)

dx – sum of horizontal grandients (sobelX_A + sobelX_B)

dy – sum of vertical gradients (sobelY_A + sobelY_B)

The normalized vector(dx, dy) is basically already the flow-vector. dt defines the length and direction (positive or negative).

There are several ways to generate the gradients. I got decent results with using the Sobel-Operator. Since everything can be done on the GPU using shaders the overhead is minimal.

Optical Flow Algorithm

update step

{

preprocessing (blur, noise-reduction, grayscale/color, etc…) – optional

gradients (sobel, horizontal+vertical)

optical flow (GLSL-Shader)

smooth (gaussian blur + temporal smooth) – optional

}

Noise-ReductioN

Preprocessing

Most obviously, the input images can (and should) be preprocessed. It depends a lot on the source-footage if its worth to apply more expensive filters (median, bilateral filter, …) and/or simply a gaussian blur.

Thresholding

A threshold, to omit flow-vectors, of a certain length.

PostProcessing

Gauss-bluring of the resulting flow and finally some temporal smoothing. (mixing the current and the previous flow)

GLSL-Shader – Optical Flow

/**

- PixelFlow | Copyright (C) 2016 Thomas Diewald – http://thomasdiewald.com

# version 150

precision mediump float;

precision mediump int;

# define SUM_RGB(v) ((v).r + (v).g + (v).b)

out vec2 glFragColor;

uniform sampler2D tex_curr_frame ; // current image

uniform sampler2D tex_prev_frame ; // previous image

uniform sampler2D tex_curr_sobelH; // current gradient horizontal

uniform sampler2D tex_curr_sobelV; // current gradient vertical

uniform sampler2D tex_prev_sobelH; // previous gradient horizontal

uniform sampler2D tex_prev_sobelV; // previous gradient vertical

uniform vec2 wh; // size rendertarget

uniform float scale; // scale flow

uniform float threshold = 1.0; // flow intensity threshold

void main(){

vec2 posn = gl_FragCoord.xy / wh;

// dt, dx, dy

vec3 dt_ = texture(tex_curr_frame , posn).rgb –

texture(tex_prev_frame , posn).rgb;

vec3 dx_ = texture(tex_curr_sobelH, posn).rgb +

texture(tex_prev_sobelH, posn).rgb;

vec3 dy_ = texture(tex_curr_sobelV, posn).rgb +

texture(tex_prev_sobelV, posn).rgb;

// sum rgb-channels

float dt = SUM_RGB(dt_), dx = SUM_RGB(dx_), dy = SUM_RGB(dy_);

// gradient length

float dd = sqrt(dx*dx + dy*dy + 1.0);

// optical flow

vec2 flow = scale * dt * vec2(dx, dy) / dd;

// threshold

float len_old = sqrt(flow.x*flow.x + flow.y*flow.y + 0.00001);

float len_new = max(len_old – threshold, 0.0);

flow = (len_new * flow)/len_old;

glFragColor = flow;

}