DataLeaper

Perspective Projection, Intrinsics, and Depth

This is Part 4 of a 4-part series:

  1. Understanding Camera Coordinate Transformations
  2. Orthographic Projection? 📸
  3. Viewport Transform for Orthographic LiDAR Projection
  4. Perspective Projection, Intrinsics, and Depth

Table of Contents

Glossary

Assumptions

Intro

In the orthographic projection posts, the useful simplification was this:

A pixel can be treated as a constant-sized square in the real world.

That is why orthographic projection is easier for measurement. Once we know the scale, pixel distance can be converted back into real distance with simple multiplication.

Perspective projection breaks this assumption.

In perspective projection, a pixel is not a fixed-size square in the world. A pixel identifies a viewing ray leaving the camera. To recover metric geometry, we point that ray back into the 3D world and determine where it meets the visible surface.

Parallel train tracks

alt text

This is also how our eyesight works. In real world its not possible to directly obtain ortographic projection.

This is why an RGB-D measurement pipeline becomes important in the real world. It combines

  1. image taken with perspective projection using camera intrinsics
  2. a Z-depth map (this can be derived from LiDAR, photogrammetry, or another depth source)

In essence

Together they let us reconstruct visible points in 3D. We can then estimate the surface plane and calculate physical dimensions in that plane.

1. The Intrinsic Matrix

The camera intrinsic matrix usually looks like this:

$ K = \begin{pmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix} $

For example, a calibrated camera might have this intrinsic matrix:

m = np.array([
    [3003.1174, 0, 2011.17],
    [0, 3003.1174, 1514.9209],
    [0, 0, 1]
])

So:

fx = 3003.1174
fy = 3003.1174
cx = 2011.17
cy = 1514.9209

Definitions:

These four values define how image pixels relate to camera rays.

2. Principal Point: cx, cy

The principal point is:

The pixel where the camera is looking straight ahead.

It is located on the sensor. It is not exactly the same thing as the image center, although it is usually close.

The image center is just the geometric middle of the rectangular image:

image center = (image_width / 2, image_height / 2)

The principal point is physical:

principal point = where the lens optical axis hits the sensor

In a perfect camera, those would be exactly the same. In a real camera, the lens and sensor are not mounted with mathematical perfection, so calibration gives us the actual principal point.

For example, if an image is roughly 4032 x 3024, then the image center is:

(2016, 1512)

This example principal point is:

(2011.17, 1514.9209)

That is very close to the center, but not exactly. It is about 4.8 pixels left and 2.9 pixels down from the image center.

The useful mental model is:

(cx, cy) = the zero point for camera direction

If a pixel is exactly at (cx, cy), then it looks straight forward from the camera.

If a pixel is to the right of cx, then it looks a bit to the right.

If a pixel is to the left of cx, then it looks a bit to the left.

Same for cy vertically.

This is why the code uses:

x - cx
y - cy

It is asking:

How far is this pixel from the straight-ahead pixel?

3. Focal Length in Pixels: fx, fy

The values fx and fy are focal lengths, but measured in pixels.

That sounds strange at first because focal length is often described in millimeters. But for image geometry, pixel units are more practical.

In the ideal geometric picture, focal length is the forward distance from the pinhole to the image plane. In a calibrated intrinsic matrix, however, fx and fy should be treated directly as two independently estimated pixel-coordinate scale parameters.

They may be equal, but the projection equations do not require this. They can differ because of image-axis scaling, resizing, non-square sampling, or the camera-calibration result. No assumption about a known physical focal length or physical pixel size is needed for the reconstruction below.

alt text

What focal length really does

Large fx means:

same pixel offset = smaller angle

That is like a zoomed-in / narrow field-of-view camera.

Small fx means:

same pixel offset = larger angle

That is like a wide-angle camera.

The same applies to fy, but vertically.

In this example, fx and fy are equal:

fx = fy = 3003.1174

That means this example assumes the camera has the same scaling horizontally and vertically. In practical terms, square pixels and symmetric focal scaling. This is commonly assumed.

4. Pixel to Ray

First let's talk about the virtual image plane

In the pinhole camera model, the real image sensor sits behind the small camera hole / camera center.

That real image plane receives an upside-down version of the world, because light rays cross at the pinhole before they hit the sensor.

For geometry, that flipped picture is annoying. So instead of drawing the image plane behind the pinhole, we usually draw a virtual image plane in front of the pinhole, between the camera and the scene.

It represents almost the same thing as the actual image plane, but inverted to the front side:

scene
    |
    |
virtual image plane
    |
camera center / pinhole
    |
real image plane / sensor

The virtual image plane is not a physical surface inside the camera. It is a mathematical helper. It lets us say that a pixel is in front of the camera and that the ray goes from the camera center through that pixel into the world.

So the virtual image plane is basically the real image plane mirrored through the pinhole. Same projection idea, but with the inconvenient upside-down sensor image turned into a forward-facing construction.

Pixel to Ray

Two 2D views

X,Z plane

X,Y plane

Reasoning

$ q = \begin{pmatrix} \dfrac{x-c_x}{f_x} \\ \dfrac{y-c_y}{f_y} \\ 1 \end{pmatrix}. $

5. Back-Projection into 3D

From previous we have

$ q= \begin{pmatrix} \dfrac{x-c_x}{f_x} \\[6pt] \dfrac{y-c_y}{f_y} \\[6pt] 1 \end{pmatrix}. $

The vector $q$ is a camera-space ray direction scaled so that its forward component is $1$. Every point on that ray has the form:

$ P(s)=sq, \qquad s>0. $

A pixel alone therefore does not identify one phyisical 3D point. It identifies infinitely many possible points along one ray.

Thus having Z, based on previous logic above we can create 3D point

$ P=Zq= \begin{pmatrix} Z\dfrac{x-c_x}{f_x} \\[6pt] Z\dfrac{y-c_y}{f_y} \\[6pt] Z \end{pmatrix}= \begin{pmatrix} X \\[6pt] Y \\[6pt] Z \end{pmatrix} $

6. Creating a Point Cloud

Back-project every valid depth pixel to create a point cloud of the visible surface:

def depth_map_to_points(depth, fx, fy, cx, cy):

  z_depth = depth[pixel_y, pixel_x]

  point_x = z_depth * (pixel_x - cx) / fx
  point_y = z_depth * (pixel_y - cy) / fy
  return np.column_stack((point_x, point_y, z_depth))

Each row of the result is a point $(X,Y,Z)$ in camera coordinates.

7. Enjoy your life back in 3D space

References