# DA3 COLMAP Export: From Depth Maps to Point Clouds

## Overview

Depth Anything 3 (DA3) uses a fundamentally different approach than traditional COLMAP for generating point clouds. Instead of feature extraction, matching, and bundle adjustment, DA3 directly predicts depth maps and camera poses, then converts them to 3D points using geometric projection.

## Traditional COLMAP Workflow

Traditional COLMAP follows this pipeline:

1. **Feature Extraction**: Extract keypoints and descriptors (e.g., SIFT ) from images
2. **Feature Matching**: Match features across image pairs
3. **Initial Reconstruction**: Triangulate initial 3D points from matched features
4. **Bundle Adjustment**: Optimize camera poses and 3D points jointly to minimize reprojection error
5. **Dense Reconstruction**: Generate dense point cloud (optional, using MVS)

**Key characteristic**: COLMAP starts with sparse features and builds up to a reconstruction through optimization.

## DA3 Workflow

DA3 takes a direct approach:

1. **Model Inference**: Neural network directly predicts:

   - Depth maps: `(N, H, W)` - per-pixel depth values
   - Camera extrinsics: `(N, 3, 4)` - world-to-camera transformation matrices
   - Camera intrinsics: `(N, 3, 3)` - focal length and principal point
   - Confidence maps: `(N, H, W)` - per-pixel confidence scores

2. **Depth-to-Point Conversion**: For each pixel with valid depth:

   - Convert pixel coordinates `(u, v)` to camera-space ray direction
   - Scale ray by depth to get 3D point in camera space
   - Transform to world space using camera pose

3. **COLMAP Structure Creation**: Build COLMAP reconstruction structure:
   - Create 3D points from converted depth maps
   - Set up cameras, images, and frames
   - Link 2D observations to 3D points

**Key characteristic**: DA3 starts with dense depth predictions and directly converts them to 3D points.

## Detailed Process: Depth Maps → COLMAP Point Cloud

### Step 1: Model Inference

The DA3 model takes images as input and outputs:

```python
prediction = model.inference(images)
# prediction.depth: (N, H, W) - depth maps
# prediction.extrinsics: (N, 3, 4) - camera poses (w2c format)
# prediction.intrinsics: (N, 3, 3) - camera intrinsics
# prediction.conf: (N, H, W) - confidence maps
```

### Step 2: Depth-to-World Point Conversion

**File**: `src/depth_anything_3/utils/export/glb.py` (lines 205-252)

The function `_depths_to_world_points_with_colors()` performs the conversion:

```python
def _depths_to_world_points_with_colors(
    depth: np.ndarray,      # (N, H, W)
    K: np.ndarray,          # (N, 3, 3) - intrinsics
    ext_w2c: np.ndarray,    # (N, 3, 4) - extrinsics (world-to-camera)
    images_u8: np.ndarray,  # (N, H, W, 3) - RGB images
    conf: np.ndarray,       # (N, H, W) - confidence
    conf_thr: float,        # confidence threshold
) -> tuple[np.ndarray, np.ndarray]:  # (points, colors)
```

**For each frame `i`:**

1. **Filter valid pixels**:

   ```python
   valid = np.isfinite(depth[i]) & (depth[i] > 0) & (conf[i] >= conf_thr)
   ```

2. **Create pixel grid**:

   ```python
   us, vs = np.meshgrid(np.arange(W), np.arange(H))
   pix = np.stack([us, vs, ones], axis=-1)  # (H*W, 3) - homogeneous pixel coords
   ```

3. **Convert pixels to camera-space rays**:

   ```python
   K_inv = np.linalg.inv(K[i])  # Inverse intrinsics
   rays = K_inv @ pix[valid].T  # (3, M) - ray directions in camera space
   ```

4. **Scale rays by depth to get 3D points in camera space**:

   ```python
   Xc = rays * depth[i][valid][None, :]  # (3, M) - 3D points in camera frame
   ```

5. **Transform to world space**:

   ```python
   c2w = np.linalg.inv(ext_w2c[i])  # Convert w2c to c2w (camera-to-world)
   Xc_h = np.vstack([Xc, np.ones((1, M))])  # Homogeneous coordinates
   Xw = (c2w @ Xc_h)[:3].T  # (M, 3) - 3D points in world space
   ```

6. **Extract colors**:
   ```python
   colors = images_u8[i][valid]  # (M, 3) - RGB colors
   ```

**Result**: Dense point cloud with `M` points (where `M` = number of valid pixels across all frames after confidence filtering).

### Step 3: COLMAP Structure Creation

**File**: `src/depth_anything_3/utils/export/colmap.py` (lines 28-127)

The `export_to_colmap()` function builds the COLMAP reconstruction:

#### 3.1 Create 3D Points

```python
reconstruction = pycolmap.Reconstruction()

# Add all 3D points
for vidx in range(num_points):
    point3d_id = reconstruction.add_point3D(
        points[vidx],      # 3D position (x, y, z)
        pycolmap.Track(),  # Empty track (will be populated)
        colors[vidx]       # RGB color
    )
```

#### 3.2 Set Up Cameras, Images, and Frames

For each frame:

1. **Create Camera**:

   ```python
   camera = pycolmap.Camera()
   camera.model = pycolmap.CameraModelId.PINHOLE
   camera.params = [fx, fy, cx, cy]  # Intrinsics in COLMAP format
   camera.width = orig_w
   camera.height = orig_h
   ```

2. **Create Rig** (COLMAP's camera rig structure):

   ```python
   rig = pycolmap.Rig()
   rig.rig_id = camera.camera_id
   rig.add_ref_sensor(camera.sensor_id)
   ```

3. **Create Frame** (camera pose):

   ```python
   frame = pycolmap.Frame()
   frame.rig_from_world = cam_from_world  # w2c transformation
   ```

4. **Create Image**:
   ```python
   image = pycolmap.Image()
   image.name = os.path.basename(image_paths[fidx])
   image.camera_id = camera.camera_id
   ```

#### 3.3 Link 2D Observations to 3D Points

For each frame, create 2D point observations that correspond to the 3D points:

```python
# Find which 3D points are visible in this frame
points_in_frame = points_xyf[:, 2] == fidx  # points_xyf: (x, y, frame_idx)

# Create 2D observations
for vidx in np.where(points_in_frame)[0]:
    point2d = points_xyf[vidx][:2]  # (u, v) pixel coordinates
    point3d_id = point3d_ids[vidx]   # Corresponding 3D point ID

    # Add 2D observation
    point2d_list.append(pycolmap.Point2D(point2d, point3d_id))

    # Update track (which images see this 3D point)
    reconstruction.point3D(point3d_id).track.add_element(
        image.image_id, len(point2d_list) - 1
    )
```

#### 3.4 Export

```python
reconstruction.write(export_dir)  # Writes COLMAP binary format
```

## Key Differences from Traditional COLMAP

| Aspect                 | Traditional COLMAP                         | DA3                              |
| ---------------------- | ------------------------------------------ | -------------------------------- |
| **Input**              | Images only                                | Images (poses optional)          |
| **Output**             | Sparse → Dense reconstruction              | Dense depth maps + poses         |
| **Point Cloud Source** | Triangulated from matched features         | Directly from depth maps         |
| **Optimization**       | Bundle adjustment optimizes poses & points | Poses & depth predicted by model |
| **Density**            | Sparse initially, dense via MVS            | Dense from the start             |
| **Feature Matching**   | Required                                   | Not needed                       |
| **Bundle Adjustment**  | Required                                   | Not needed                       |

## Advantages of DA3 Approach

1. **Dense from the start**: Every pixel with valid depth becomes a 3D point
2. **No feature matching**: Avoids issues with textureless regions or repetitive patterns
3. **Consistent geometry**: Model enforces geometric consistency across views
4. **Faster**: No iterative optimization required
5. **Works with fewer images**: Can work with just 2 images (traditional COLMAP needs more)

## Limitations

1. **No bundle adjustment**: Poses and depths are fixed from model prediction
2. **Model-dependent quality**: Quality depends on model training, not geometric optimization
3. **Scale ambiguity**: For monocular inputs, scale may need to be recovered (unless using metric models)

## Confidence Filtering

DA3 uses confidence maps to filter unreliable depth predictions:

```python
conf_thresh = np.percentile(prediction.conf, conf_thresh_percentile)  # Default: 40th percentile
valid = conf >= conf_thresh
```

This means only the top 60% most confident depth predictions are converted to 3D points, reducing noise in the point cloud.

### Deep Dive: How Confidence is Calculated

Confidence in DA3 is **learned by the neural network** during training, not computed from depth errors or geometric consistency at inference time. Here's the complete pipeline:

#### 1. Network Architecture

**File**: `src/depth_anything_3/model/dpt.py` and `src/depth_anything_3/model/dualdpt.py`

The depth head outputs **2 channels** when confidence is enabled (`output_dim=2`):

- **Channel 0**: Depth logits
- **Channel 1**: Confidence logits

**Configuration** (from `da3-large.yaml`):

```yaml
head:
  output_dim: 2 # 2 channels = 1 for depth + 1 for confidence
  conf_activation: 'expp1' # Activation function for confidence
```

#### 2. Confidence Activation Function

**File**: `src/depth_anything_3/model/dpt.py`, lines 286-309

The confidence logits go through the **`expp1` activation**:

```python
def _apply_activation_single(self, x: torch.Tensor, activation: str = "expp1"):
    if activation == "expp1":
        return torch.exp(x) + 1
```

**Formula**: `confidence = exp(logits) + 1`

**Properties**:

- **Range**: `[1, +∞)` - Confidence is always ≥ 1, unbounded above
- **Interpretation**: Higher values = higher confidence
- **Minimum**: The `+1` ensures minimum confidence of 1 (no zero confidence)

#### 3. Forward Pass Flow

**File**: `src/depth_anything_3/model/dpt.py`, lines 244-252

```python
# Main head outputs logits with 2 channels
main_logits = self.scratch.output_conv2(feat)  # (B, S, 2, H, W)

# Permute to (B, S, H, W, 2) for channel-wise processing
fmap = main_logits.permute(0, 2, 3, 1)

# Split channels: depth (channel 0) and confidence (channel 1)
pred = self._apply_activation_single(fmap[..., :-1], self.activation)  # depth = exp(logits)
conf = self._apply_activation_single(fmap[..., -1], self.conf_activation)  # conf = exp(logits) + 1

# Output
output["depth"] = pred.squeeze(1)  # (B, S, H, W)
output["depth_conf"] = conf.squeeze(1)  # (B, S, H, W)
```

#### 4. What Confidence Represents

Based on the DA3 paper (Section 3.3) and architecture:

**Confidence is learned to predict**:

- **Multi-view visibility**: Whether a pixel is visible and consistent across multiple views
- **Geometric reliability**: How reliable the depth prediction is based on:
  - Texture richness (textured regions → higher confidence)
  - Edge alignment (depth edges aligned with image edges → higher confidence)
  - Multi-view consistency (consistent across views → higher confidence)
  - Occlusion handling (occluded regions → lower confidence)

**Training supervision** (from DA3 paper Section 3.3):

The paper explicitly defines confidence training in the loss function:

```python
L = LD(D̂, D) + LM(R̂, M) + LP(D̂ ⊙ d + t, P) + βLC(ĉ, v) + αLgrad(D̂, D)
```

Where:

- `LC(ĉ, v)` is the **confidence loss** with weight `β = 1`
- `ĉ` is the **predicted confidence** (from network)
- `v` is the **ground truth visibility mask** (binary mask indicating pixel visibility across views)

**Depth loss with confidence weighting** (from paper):

```python
LD(D̂, D; Dc) = 1/Z_Ω Σ_{p∈Ω} m_p [|D̂_p - D_p|/D_c,p - λ_c log D_c,p]
```

Where:

- `D_c,p` is the **confidence of depth** `D_p` at pixel `p`
- The depth error is **inversely weighted by confidence**: `|D̂_p - D_p|/D_c,p`
- **Higher confidence** → **lower weight** on depth error (model trusts its prediction, less supervision needed)
- **Lower confidence** → **higher weight** on depth error (model is uncertain, needs more supervision)
- The `-λ_c log D_c,p` term is a **regularization** that prevents overconfidence (encourages appropriate confidence calibration)

**Uncertainty-aware training mechanism**:

This creates a **heteroscedastic regression** setup where:

- The model learns to predict both **depth** (mean) and **confidence** (inverse variance/uncertainty)
- High-confidence regions: Model is certain → small depth errors are acceptable → less gradient signal
- Low-confidence regions: Model is uncertain → depth errors are penalized more → stronger gradient signal
- This allows the model to **focus learning** on uncertain regions while being more lenient on confident predictions

**Example**:

- Pixel with `conf = 10`: Depth error of `0.1m` → weighted error = `0.1/10 = 0.01` (small contribution to loss)
- Pixel with `conf = 1`: Depth error of `0.1m` → weighted error = `0.1/1 = 0.1` (large contribution to loss)
- The model learns to assign high confidence to regions it can predict well, and low confidence to challenging regions

**Ground truth visibility masks** (from paper context):

- **Multi-view geometry**: Pixels visible in multiple views get `v = 1` (high confidence target)
- **Occluded regions**: Pixels occluded in other views get `v = 0` (low confidence target)
- **Sparse depth regions**: Areas with missing/noisy ground truth depth get lower confidence targets
- **Teacher-student alignment**: Teacher model's high-quality predictions help identify reliable regions

**Training process** (inferred from paper Section 4.2):

1. **Teacher model** generates high-quality pseudo-depth for real-world noisy data
2. **Visibility masks** are computed from multi-view geometry (COLMAP, SfM)
3. **Confidence loss** `LC(ĉ, v)` supervises the network to predict confidence matching visibility
4. **Depth loss** uses confidence to weight errors: unreliable regions (low confidence) contribute less to depth loss
5. **Joint optimization**: Confidence and depth are learned together, with confidence acting as an uncertainty estimate

**Key insight from paper**: Confidence serves dual purpose:

- **Training**: Acts as a **learned weighting** for depth loss (uncertainty-aware training)
- **Inference**: Acts as a **reliability score** for filtering unreliable predictions

**Confidence loss formulation** (inferred from paper and DINOv2-style training):

Based on the paper's mention of `LC(ĉ, v)` and similarity to DINOv2's loss patterns, the confidence loss likely follows:

```python
LC(ĉ, v) = ||ĉ - v||₁  # L1 loss between predicted confidence and visibility mask
# or
LC(ĉ, v) = BCE(σ(ĉ), v)  # Binary cross-entropy if v is binary
```

Where:

- `v ∈ {0, 1}` or `v ∈ [0, 1]` is the visibility mask (1 = visible/reliable, 0 = occluded/unreliable)
- The model learns to predict higher confidence (`ĉ`) for visible pixels and lower confidence for occluded pixels
- Since confidence uses `expp1` activation (range `[1, +∞)`), the loss likely normalizes or scales the visibility mask to match

**Connection to DINOv2 training paradigm**:

Similar to DINOv2's teacher-student training where:

- **Teacher** provides soft targets (high-quality pseudo-labels)
- **Student** learns from teacher with confidence weighting
- **Confidence** indicates how much to trust the teacher's supervision

In DA3:

- **Teacher model** provides high-quality depth pseudo-labels (Section 4.1-4.2)
- **Confidence** indicates how much to trust these pseudo-labels
- **Visibility masks** provide ground truth for confidence learning from multi-view geometry
- **Joint optimization**: Confidence and depth are learned together, creating uncertainty-aware training

#### 5. DualDPT Architecture (Main Models)

**File**: `src/depth_anything_3/model/dualdpt.py`

DA3 uses a **dual-head architecture** with two confidence outputs:

1. **Main head confidence** (`depth_conf`):

   - Primary depth prediction confidence
   - Used for filtering in COLMAP export

2. **Auxiliary head confidence** (`depth_aux_conf`):
   - Secondary confidence from auxiliary prediction branch
   - Used internally for multi-scale consistency

Both use the same `expp1` activation: `conf = exp(logits) + 1`

#### 6. Confidence Usage in Export

**File**: `src/depth_anything_3/utils/export/colmap.py`, lines 35-44

```python
# 1. Compute percentile threshold
conf_thresh = np.percentile(prediction.conf, conf_thresh_percentile)  # Default: 40th percentile

# 2. Filter points
points, colors = _depths_to_world_points_with_colors(
    prediction.depth,
    prediction.intrinsics,
    prediction.extrinsics,
    prediction.processed_images,
    prediction.conf,  # Confidence used for filtering
    conf_thresh,     # Threshold
)
```

**Filtering logic** (in `glb.py`):

```python
valid = np.isfinite(depth[i]) & (depth[i] > 0) & (conf[i] >= conf_thresh)
```

Only pixels with `confidence >= threshold` are converted to 3D points.

#### 7. Confidence Statistics

**Typical confidence values**:

- **High confidence**: `conf > 10-100` (well-textured, multi-view consistent regions)
- **Medium confidence**: `conf ≈ 2-10` (moderate texture, some consistency)
- **Low confidence**: `conf ≈ 1-2` (textureless, occluded, or inconsistent regions)

**Percentile filtering**:

- `conf_thresh_percentile=40.0` means: keep top 60% most confident pixels
- `conf_thresh_percentile=50.0` means: keep top 50% (median split)
- `conf_thresh_percentile=10.0` means: keep top 90% (very permissive)

#### 8. Key Insights

1. **Learned, not computed**: Confidence is a neural network prediction, not a post-hoc error metric
2. **Unbounded above**: No maximum confidence value (can be very large for highly reliable pixels)
3. **Minimum of 1**: The `+1` ensures no pixel has zero confidence (all pixels have some baseline reliability)
4. **Multi-view aware**: Confidence reflects multi-view geometric consistency (learned during training)
5. **Texture-dependent**: Higher confidence in textured regions, lower in textureless areas
6. **Occlusion-aware**: Lower confidence in occluded or boundary regions

#### 9. Comparison with Other Methods

| Method                     | Confidence Source        | Range     | Interpretation                    |
| -------------------------- | ------------------------ | --------- | --------------------------------- |
| **DA3**                    | Learned by network       | `[1, +∞)` | Higher = more reliable            |
| **COLMAP MVS**             | Photo-consistency score  | `[0, 1]`  | Higher = better photo-consistency |
| **Stereo matching**        | Matching cost/confidence | `[0, 1]`  | Higher = better match quality     |
| **Uncertainty estimation** | Predicted variance       | `[0, +∞)` | Lower = more certain              |

#### 10. Limitations

1. **No explicit error modeling**: Confidence doesn't directly predict depth error magnitude
2. **Training-dependent**: Quality depends on training data and supervision
3. **Relative, not absolute**: Confidence values are relative (higher = better), not absolute error bounds
4. **No calibration**: Confidence values aren't calibrated to actual error rates (unlike calibrated uncertainty)

#### Summary

DA3's confidence is a **learned reliability score** that:

- Is predicted by the neural network (not computed from errors)
- Uses `expp1` activation: `conf = exp(logits) + 1` (range: `[1, +∞)`)
- Reflects multi-view geometric consistency, texture richness, and occlusion
- Is used to filter unreliable depth predictions before point cloud generation
- Default filtering keeps top 60% most confident pixels (`conf_thresh_percentile=40.0`)

The confidence map is essentially the model's **self-assessment** of how reliable each depth prediction is, learned from training on multi-view datasets with ground truth visibility and geometry.

## Coordinate System Notes

- **Extrinsics format**: DA3 uses **w2c** (world-to-camera) format, which is converted to **c2w** (camera-to-world) for point cloud generation
- **COLMAP format**: COLMAP's `rig_from_world` expects w2c format, so the extrinsics are used directly
- **Camera model**: Pinhole camera model is assumed

## Summary

DA3's COLMAP export bypasses the traditional feature-based pipeline entirely. Instead:

1. **Model predicts** → Dense depth maps + camera poses
2. **Geometric projection** → Convert depth pixels to 3D points
3. **Structure creation** → Build COLMAP reconstruction with points, cameras, and observations

This results in a dense, geometrically consistent point cloud that can be directly used in downstream applications like 3D Gaussian Splatting, mesh reconstruction, or further COLMAP processing.