Point cloud overlay on video


I would like to use the Projection2 component to display a lidar point cloud on top of a video stream.

I get my point cloud from my lidar which is sent to a Projection2 then to an Overlay component.
I can see the result in the image viewer, however the point cloud is not correctly rotated.

I assume I am doing something wrong with the Projection’s matrix input

Can you help me with this ?


The Projection2 component was made to use OpenGL projection matrixes.

A quick word on how matrixes work in OpenGL and homogeneous coordinates :

Let’s say you have a point A (x, y, z)
In homogeneous coordinates this become x, y, z, w

This allows all transformations to be done in one matrix i.e :

(x', y', z', w') = [scaling + rotation + shear + translation] × (x, y, z, w)

Examples can be found on the khronos website :

Rotation (linear in 3d) :


Translation (not linear in 3D) :


LookAt (used to set the camera position in a “world reference frame”) :


Projection2 input matrixes :

The component waits for a 3x4 matrix instead of a the usual 4x4 because we don’t need the z as it is simply the z-index of the point which is irrelevant here.

This gives a good starting point for a projection perspective from camera parameters :


And finally back to RTMaps, here is a demo which uses the Projection2 component if you want to see it in action :


1 Like