research_docs/COLMAP_EXPORT_EXPLANATION.md

DA3 COLMAP Export: From Depth Maps to Point Clouds

Overview

Depth Anything 3 (DA3) uses a fundamentally different approach than traditional COLMAP for generating point clouds. Instead of feature extraction, matching, and bundle adjustment, DA3 directly predicts depth maps and camera poses, then converts them to 3D points using geometric projection.

Traditional COLMAP Workflow

Traditional COLMAP follows this pipeline:

1. Feature Extraction: Extract keypoints and descriptors (e.g., SIFT ) from images
2. Feature Matching: Match features across image pairs
3. Initial Reconstruction: Triangulate initial 3D points from matched features
4. Bundle Adjustment: Optimize camera poses and 3D points jointly to minimize reprojection error
5. Dense Reconstruction: Generate dense point cloud (optional, using MVS)

Key characteristic: COLMAP starts with sparse features and builds up to a reconstruction through optimization.

DA3 Workflow

DA3 takes a direct approach:

1. Model Inference: Neural network directly predicts:

   • Depth maps: (N, H, W) - per-pixel depth values
   • Camera extrinsics: (N, 3, 4) - world-to-camera transformation matrices
   • Camera intrinsics: (N, 3, 3) - focal length and principal point
   • Confidence maps: (N, H, W) - per-pixel confidence scores

2. Depth-to-Point Conversion: For each pixel with valid depth:

   • Convert pixel coordinates (u, v) to camera-space ray direction
   • Scale ray by depth to get 3D point in camera space
   • Transform to world space using camera pose

3. COLMAP Structure Creation: Build COLMAP reconstruction structure:

   • Create 3D points from converted depth maps
   • Set up cameras, images, and frames
   • Link 2D observations to 3D points

Key characteristic: DA3 starts with dense depth predictions and directly converts them to 3D points.

Detailed Process: Depth Maps → COLMAP Point Cloud

Step 1: Model Inference

The DA3 model takes images as input and outputs:

prediction = model.inference(images)
# prediction.depth: (N, H, W) - depth maps
# prediction.extrinsics: (N, 3, 4) - camera poses (w2c format)
# prediction.intrinsics: (N, 3, 3) - camera intrinsics
# prediction.conf: (N, H, W) - confidence maps

Step 2: Depth-to-World Point Conversion

File: src/depth_anything_3/utils/export/glb.py (lines 205-252)

The function _depths_to_world_points_with_colors() performs the conversion:

def _depths_to_world_points_with_colors(
    depth: np.ndarray,      # (N, H, W)
    K: np.ndarray,          # (N, 3, 3) - intrinsics
    ext_w2c: np.ndarray,    # (N, 3, 4) - extrinsics (world-to-camera)
    images_u8: np.ndarray,  # (N, H, W, 3) - RGB images
    conf: np.ndarray,       # (N, H, W) - confidence
    conf_thr: float,        # confidence threshold
) -> tuple[np.ndarray, np.ndarray]:  # (points, colors)

For each frame i:

1. Filter valid pixels:

   valid = np.isfinite(depth[i]) & (depth[i] > 0) & (conf[i] >= conf_thr)
   

2. Create pixel grid:

   us, vs = np.meshgrid(np.arange(W), np.arange(H))
   pix = np.stack([us, vs, ones], axis=-1)  # (H*W, 3) - homogeneous pixel coords
   

3. Convert pixels to camera-space rays:

   K_inv = np.linalg.inv(K[i])  # Inverse intrinsics
   rays = K_inv @ pix[valid].T  # (3, M) - ray directions in camera space
   

4. Scale rays by depth to get 3D points in camera space:

   Xc = rays * depth[i][valid][None, :]  # (3, M) - 3D points in camera frame
   

5. Transform to world space:

   c2w = np.linalg.inv(ext_w2c[i])  # Convert w2c to c2w (camera-to-world)
   Xc_h = np.vstack([Xc, np.ones((1, M))])  # Homogeneous coordinates
   Xw = (c2w @ Xc_h)[:3].T  # (M, 3) - 3D points in world space
   

6. Extract colors:

   colors = images_u8[i][valid]  # (M, 3) - RGB colors
   

Result: Dense point cloud with M points (where M = number of valid pixels across all frames after confidence filtering).

Step 3: COLMAP Structure Creation

File: src/depth_anything_3/utils/export/colmap.py (lines 28-127)

The export_to_colmap() function builds the COLMAP reconstruction:

3.1 Create 3D Points

reconstruction = pycolmap.Reconstruction()

# Add all 3D points
for vidx in range(num_points):
    point3d_id = reconstruction.add_point3D(
        points[vidx],      # 3D position (x, y, z)
        pycolmap.Track(),  # Empty track (will be populated)
        colors[vidx]       # RGB color
    )

3.2 Set Up Cameras, Images, and Frames

For each frame:

1. Create Camera:

   camera = pycolmap.Camera()
   camera.model = pycolmap.CameraModelId.PINHOLE
   camera.params = [fx, fy, cx, cy]  # Intrinsics in COLMAP format
   camera.width = orig_w
   camera.height = orig_h
   

2. Create Rig (COLMAP's camera rig structure):

   rig = pycolmap.Rig()
   rig.rig_id = camera.camera_id
   rig.add_ref_sensor(camera.sensor_id)
   

3. Create Frame (camera pose):

   frame = pycolmap.Frame()
   frame.rig_from_world = cam_from_world  # w2c transformation
   

4. Create Image:

   image = pycolmap.Image()
   image.name = os.path.basename(image_paths[fidx])
   image.camera_id = camera.camera_id
   

3.3 Link 2D Observations to 3D Points

For each frame, create 2D point observations that correspond to the 3D points:

# Find which 3D points are visible in this frame
points_in_frame = points_xyf[:, 2] == fidx  # points_xyf: (x, y, frame_idx)

# Create 2D observations
for vidx in np.where(points_in_frame)[0]:
    point2d = points_xyf[vidx][:2]  # (u, v) pixel coordinates
    point3d_id = point3d_ids[vidx]   # Corresponding 3D point ID

    # Add 2D observation
    point2d_list.append(pycolmap.Point2D(point2d, point3d_id))

    # Update track (which images see this 3D point)
    reconstruction.point3D(point3d_id).track.add_element(
        image.image_id, len(point2d_list) - 1
    )

3.4 Export

reconstruction.write(export_dir)  # Writes COLMAP binary format

Key Differences from Traditional COLMAP

Aspect	Traditional COLMAP	DA3
Input	Images only	Images (poses optional)
Output	Sparse → Dense reconstruction	Dense depth maps + poses
Point Cloud Source	Triangulated from matched features	Directly from depth maps
Optimization	Bundle adjustment optimizes poses & points	Poses & depth predicted by model
Density	Sparse initially, dense via MVS	Dense from the start
Feature Matching	Required	Not needed
Bundle Adjustment	Required	Not needed

Advantages of DA3 Approach

1. Dense from the start: Every pixel with valid depth becomes a 3D point
2. No feature matching: Avoids issues with textureless regions or repetitive patterns
3. Consistent geometry: Model enforces geometric consistency across views
4. Faster: No iterative optimization required
5. Works with fewer images: Can work with just 2 images (traditional COLMAP needs more)

Limitations

1. No bundle adjustment: Poses and depths are fixed from model prediction
2. Model-dependent quality: Quality depends on model training, not geometric optimization
3. Scale ambiguity: For monocular inputs, scale may need to be recovered (unless using metric models)

Confidence Filtering

DA3 uses confidence maps to filter unreliable depth predictions:

conf_thresh = np.percentile(prediction.conf, conf_thresh_percentile)  # Default: 40th percentile
valid = conf >= conf_thresh

This means only the top 60% most confident depth predictions are converted to 3D points, reducing noise in the point cloud.

Deep Dive: How Confidence is Calculated

Confidence in DA3 is learned by the neural network during training, not computed from depth errors or geometric consistency at inference time. Here's the complete pipeline:

1. Network Architecture

File: src/depth_anything_3/model/dpt.py and src/depth_anything_3/model/dualdpt.py

The depth head outputs 2 channels when confidence is enabled (output_dim=2):

• Channel 0: Depth logits
• Channel 1: Confidence logits

Configuration (from da3-large.yaml):

head:
  output_dim: 2 # 2 channels = 1 for depth + 1 for confidence
  conf_activation: 'expp1' # Activation function for confidence

2. Confidence Activation Function

File: src/depth_anything_3/model/dpt.py, lines 286-309

The confidence logits go through the expp1 activation:

def _apply_activation_single(self, x: torch.Tensor, activation: str = "expp1"):
    if activation == "expp1":
        return torch.exp(x) + 1

Formula: confidence = exp(logits) + 1

Properties:

• Range: [1, +∞) - Confidence is always ≥ 1, unbounded above
• Interpretation: Higher values = higher confidence
• Minimum: The +1 ensures minimum confidence of 1 (no zero confidence)

3. Forward Pass Flow

File: src/depth_anything_3/model/dpt.py, lines 244-252

# Main head outputs logits with 2 channels
main_logits = self.scratch.output_conv2(feat)  # (B, S, 2, H, W)

# Permute to (B, S, H, W, 2) for channel-wise processing
fmap = main_logits.permute(0, 2, 3, 1)

# Split channels: depth (channel 0) and confidence (channel 1)
pred = self._apply_activation_single(fmap[..., :-1], self.activation)  # depth = exp(logits)
conf = self._apply_activation_single(fmap[..., -1], self.conf_activation)  # conf = exp(logits) + 1

# Output
output["depth"] = pred.squeeze(1)  # (B, S, H, W)
output["depth_conf"] = conf.squeeze(1)  # (B, S, H, W)

4. What Confidence Represents

Based on the DA3 paper (Section 3.3) and architecture:

Confidence is learned to predict:

• Multi-view visibility: Whether a pixel is visible and consistent across multiple views
• Geometric reliability: How reliable the depth prediction is based on:
  • Texture richness (textured regions → higher confidence)
  • Edge alignment (depth edges aligned with image edges → higher confidence)
  • Multi-view consistency (consistent across views → higher confidence)
  • Occlusion handling (occluded regions → lower confidence)

Training supervision (from DA3 paper Section 3.3):

The paper explicitly defines confidence training in the loss function:

L = LD(D̂, D) + LM(R̂, M) + LP(D̂ ⊙ d + t, P) + βLC(ĉ, v) + αLgrad(D̂, D)

Where:

• LC(ĉ, v) is the confidence loss with weight β = 1
• ĉ is the predicted confidence (from network)
• v is the ground truth visibility mask (binary mask indicating pixel visibility across views)

Depth loss with confidence weighting (from paper):

LD(D̂, D; Dc) = 1/Z_Ω Σ_{p∈Ω} m_p [|D̂_p - D_p|/D_c,p - λ_c log D_c,p]

Where:

• D_c,p is the confidence of depth D_p at pixel p
• The depth error is inversely weighted by confidence: |D̂_p - D_p|/D_c,p
• Higher confidence → lower weight on depth error (model trusts its prediction, less supervision needed)
• Lower confidence → higher weight on depth error (model is uncertain, needs more supervision)
• The -λ_c log D_c,p term is a regularization that prevents overconfidence (encourages appropriate confidence calibration)

Uncertainty-aware training mechanism:

This creates a heteroscedastic regression setup where:

• The model learns to predict both depth (mean) and confidence (inverse variance/uncertainty)
• High-confidence regions: Model is certain → small depth errors are acceptable → less gradient signal
• Low-confidence regions: Model is uncertain → depth errors are penalized more → stronger gradient signal
• This allows the model to focus learning on uncertain regions while being more lenient on confident predictions

Example:

• Pixel with conf = 10: Depth error of 0.1m → weighted error = 0.1/10 = 0.01 (small contribution to loss)
• Pixel with conf = 1: Depth error of 0.1m → weighted error = 0.1/1 = 0.1 (large contribution to loss)
• The model learns to assign high confidence to regions it can predict well, and low confidence to challenging regions

Ground truth visibility masks (from paper context):

• Multi-view geometry: Pixels visible in multiple views get v = 1 (high confidence target)
• Occluded regions: Pixels occluded in other views get v = 0 (low confidence target)
• Sparse depth regions: Areas with missing/noisy ground truth depth get lower confidence targets
• Teacher-student alignment: Teacher model's high-quality predictions help identify reliable regions

Training process (inferred from paper Section 4.2):

1. Teacher model generates high-quality pseudo-depth for real-world noisy data
2. Visibility masks are computed from multi-view geometry (COLMAP, SfM)
3. Confidence loss LC(ĉ, v) supervises the network to predict confidence matching visibility
4. Depth loss uses confidence to weight errors: unreliable regions (low confidence) contribute less to depth loss
5. Joint optimization: Confidence and depth are learned together, with confidence acting as an uncertainty estimate

Key insight from paper: Confidence serves dual purpose:

• Training: Acts as a learned weighting for depth loss (uncertainty-aware training)
• Inference: Acts as a reliability score for filtering unreliable predictions

Confidence loss formulation (inferred from paper and DINOv2-style training):

Based on the paper's mention of LC(ĉ, v) and similarity to DINOv2's loss patterns, the confidence loss likely follows:

LC(ĉ, v) = ||ĉ - v||₁  # L1 loss between predicted confidence and visibility mask
# or
LC(ĉ, v) = BCE(σ(ĉ), v)  # Binary cross-entropy if v is binary

Where:

• v ∈ {0, 1} or v ∈ [0, 1] is the visibility mask (1 = visible/reliable, 0 = occluded/unreliable)
• The model learns to predict higher confidence (ĉ) for visible pixels and lower confidence for occluded pixels
• Since confidence uses expp1 activation (range [1, +∞)), the loss likely normalizes or scales the visibility mask to match

Connection to DINOv2 training paradigm:

Similar to DINOv2's teacher-student training where:

• Teacher provides soft targets (high-quality pseudo-labels)
• Student learns from teacher with confidence weighting
• Confidence indicates how much to trust the teacher's supervision

In DA3:

• Teacher model provides high-quality depth pseudo-labels (Section 4.1-4.2)
• Confidence indicates how much to trust these pseudo-labels
• Visibility masks provide ground truth for confidence learning from multi-view geometry
• Joint optimization: Confidence and depth are learned together, creating uncertainty-aware training

5. DualDPT Architecture (Main Models)

File: src/depth_anything_3/model/dualdpt.py

DA3 uses a dual-head architecture with two confidence outputs:

1. Main head confidence (depth_conf):

   • Primary depth prediction confidence
   • Used for filtering in COLMAP export

2. Auxiliary head confidence (depth_aux_conf):

   • Secondary confidence from auxiliary prediction branch
   • Used internally for multi-scale consistency

Both use the same expp1 activation: conf = exp(logits) + 1

6. Confidence Usage in Export

File: src/depth_anything_3/utils/export/colmap.py, lines 35-44

# 1. Compute percentile threshold
conf_thresh = np.percentile(prediction.conf, conf_thresh_percentile)  # Default: 40th percentile

# 2. Filter points
points, colors = _depths_to_world_points_with_colors(
    prediction.depth,
    prediction.intrinsics,
    prediction.extrinsics,
    prediction.processed_images,
    prediction.conf,  # Confidence used for filtering
    conf_thresh,     # Threshold
)

Filtering logic (in glb.py):

valid = np.isfinite(depth[i]) & (depth[i] > 0) & (conf[i] >= conf_thresh)

Only pixels with confidence >= threshold are converted to 3D points.

7. Confidence Statistics

Typical confidence values:

• High confidence: conf > 10-100 (well-textured, multi-view consistent regions)
• Medium confidence: conf ≈ 2-10 (moderate texture, some consistency)
• Low confidence: conf ≈ 1-2 (textureless, occluded, or inconsistent regions)

Percentile filtering:

• conf_thresh_percentile=40.0 means: keep top 60% most confident pixels
• conf_thresh_percentile=50.0 means: keep top 50% (median split)
• conf_thresh_percentile=10.0 means: keep top 90% (very permissive)

8. Key Insights

1. Learned, not computed: Confidence is a neural network prediction, not a post-hoc error metric
2. Unbounded above: No maximum confidence value (can be very large for highly reliable pixels)
3. Minimum of 1: The +1 ensures no pixel has zero confidence (all pixels have some baseline reliability)
4. Multi-view aware: Confidence reflects multi-view geometric consistency (learned during training)
5. Texture-dependent: Higher confidence in textured regions, lower in textureless areas
6. Occlusion-aware: Lower confidence in occluded or boundary regions

9. Comparison with Other Methods

Method	Confidence Source	Range	Interpretation
DA3	Learned by network	[1, +∞)	Higher = more reliable
COLMAP MVS	Photo-consistency score	[0, 1]	Higher = better photo-consistency
Stereo matching	Matching cost/confidence	[0, 1]	Higher = better match quality
Uncertainty estimation	Predicted variance	[0, +∞)	Lower = more certain

10. Limitations

1. No explicit error modeling: Confidence doesn't directly predict depth error magnitude
2. Training-dependent: Quality depends on training data and supervision
3. Relative, not absolute: Confidence values are relative (higher = better), not absolute error bounds
4. No calibration: Confidence values aren't calibrated to actual error rates (unlike calibrated uncertainty)

Summary

DA3's confidence is a learned reliability score that:

• Is predicted by the neural network (not computed from errors)
• Uses expp1 activation: conf = exp(logits) + 1 (range: [1, +∞))
• Reflects multi-view geometric consistency, texture richness, and occlusion
• Is used to filter unreliable depth predictions before point cloud generation
• Default filtering keeps top 60% most confident pixels (conf_thresh_percentile=40.0)

The confidence map is essentially the model's self-assessment of how reliable each depth prediction is, learned from training on multi-view datasets with ground truth visibility and geometry.

Coordinate System Notes

• Extrinsics format: DA3 uses w2c (world-to-camera) format, which is converted to c2w (camera-to-world) for point cloud generation
• COLMAP format: COLMAP's rig_from_world expects w2c format, so the extrinsics are used directly
• Camera model: Pinhole camera model is assumed

Summary

DA3's COLMAP export bypasses the traditional feature-based pipeline entirely. Instead:

1. Model predicts → Dense depth maps + camera poses
2. Geometric projection → Convert depth pixels to 3D points
3. Structure creation → Build COLMAP reconstruction with points, cameras, and observations

This results in a dense, geometrically consistent point cloud that can be directly used in downstream applications like 3D Gaussian Splatting, mesh reconstruction, or further COLMAP processing.