kadireks/car-damage-review

PanelPilot

Every dent, on the right panel.

A car damage inspection tool built for CMU 24-679 (Design and Prototyping with AI), Fall 2026.

Most damage detectors tell you that a car is damaged. PanelPilot tells you which body panel each piece of damage sits on — which is what a rental company actually needs on a handover form. Two models plus a geometric matcher, behind a review interface where a human corrects whatever the model got wrong.

This repo holds everything: the weights, the engine, and the notebook that runs the GUI.

Contents

file	what it is
Project1_CarDamage_Colab.ipynb	start here — runs the GUI in Colab
damage_best.pt	damage detector — YOLO11m, fine-tuned
panel_best.pt	body-panel segmenter — YOLO11s-seg, trained from scratch
pipeline.py	loads both models, runs them, matches each damage to a panel
cardiagram.py	draws the front/side/rear damage map
car_views.png	the hand-drawn car artwork cardiagram.py shades

The interface itself lives inside the notebook, not in a .py file here, so it can be edited and relaunched without re-uploading anything to this repo.

Quick start

1. Download Project1_CarDamage_Colab.ipynb from the Files tab above.
2. Open Colab → File → Upload notebook → pick it.
3. Runtime → Change runtime type → T4 GPU (optional: CPU works, ~10 s per photo instead of ~0.3 s).
4. Run steps 1 → 6. Nothing to edit — the notebook pulls this repo by name.

Step 6 prints a public *.gradio.live link, valid about 72 hours.

Using the interface

Left column — input. Enter a car or rental ID, pick pickup or return, add a photo, press Analyze this photo. The result banner says either No damage found with the confidence threshold it used, or how many findings there are and how many need a human look.

Two shot types:

• Overall shot — both models run. The panel segmenter finds the bodywork, the damage detector finds the damage, and each finding is tied to a panel.
• Detail shot — a close-up where no whole panel is visible. You name the panel; only the damage model runs. Use this when an overall shot returns no panels.

Centre — the annotated photo. Damage boxes numbered to match the findings table. The Panel overlay control under Thresholds and display switches between shaded panels, shaded with names, and off. Click any spot on the photo to log damage the model missed.

Right — the damage map. Front, side and rear views with affected panels shaded. By default it shows the current photo only; switch Map shows to All photos to combine several angles of the same car into one map.

Findings tab. Every finding as a row. Highlighted rows need a look — low confidence, no panel overlap, or matched by proximity rather than overlap. Select one to fix its damage type or panel, or reject it outright. The damage map highlights whichever finding is selected.

Export tab.

• Generate PDF report — verdict, damage map, findings table and every annotated photo, one document, for handing to a customer or filing with the rental record.
• Export data (zip) — CSV + JSON + damage-map PNG + annotated photos, for analysis.

Photos accumulate into one car record until you press Clear and start a new car.

Thresholds worth knowing

control	default	raise it / lower it
Damage confidence	0.25	lower to catch faint damage, raise to cut false alarms
Glass / lamp confidence	0.25	raise when glare reads as cracked glass
Panel confidence	0.25	lower when no panels are found on an awkward angle

The two models

Damage detector — YOLO11m, fine-tuned on CarDD plus a set of clean-car negatives added to cut false positives on undamaged bodywork. Six classes:

dent · scratch · crack · shattered_glass · broken_lamp · flat_tire

Panel segmenter — YOLO11s-seg, trained from scratch (not fine-tuned) on a human-in-the-loop car-parts set, with rotation and multi-scale augmentation. 21 panel classes, instance segmentation so each panel comes back as a mask rather than a box.

Two different training strategies on purpose: the course requires at least two of {trained-from-scratch, fine-tuned, off-the-shelf}, and two fine-tunes would only count once.

How damage gets assigned to a panel

For each damage box, pipeline.py measures how much of the box overlaps each panel mask.

1. Overlap ≥ 5% → assigned to the panel with the largest overlap. Any other panel above 20% is recorded in also_touches, so damage straddling a seam isn't silently simplified.
2. No overlap → assigned to the nearest panel, if it is within 5% of the image diagonal.
3. Otherwise → unassigned, and flagged for a human.

Every finding carries how it was matched, so a reviewer can see which calls to trust. Thresholds live in pipeline.Settings.

Using the engine without the GUI

from huggingface_hub import snapshot_download
import sys

ENGINE = snapshot_download("kadireks/car-damage-review")
sys.path.insert(0, ENGINE)

import pipeline
pipe = pipeline.Pipeline(f"{ENGINE}/damage_best.pt", f"{ENGINE}/panel_best.pt")

img, panels, df = pipe.analyze("car.jpg")   # df: one row per finding
vis = pipe.draw(img, panels, df)            # annotated image (BGR)

df columns:

damage · damage_conf · panel · panel_overlap · also_touches · match · x1 · y1 · x2 · y2

match is overlap, nearest (Npx), no panel within 5% of image diagonal, or reviewer once a human has corrected it.

Known limitations

• No end-to-end accuracy figure yet. The two models have their own training metrics, but the combined damage→panel pipeline has not been scored against a held-out set. The review interface logs every human correction, which is the intended source for that evaluation.
• "No damage found" is threshold-dependent. A clean result at damage_conf = 0.25 is not proof a car is undamaged. Lower the threshold to check.
• Glare and reflections can read as scratch or shattered_glass. A separate, higher threshold for glass and lamp classes exists for this reason.
• Angle matters. The panel segmenter is strongest on three-quarter and straight-on views; extreme angles and tight crops often produce no panels at all. That is what detail-shot mode is for.
• Speed: roughly 0.3 s per photo on a T4, 10–12 s on CPU.

If the result images come up blank

Some Colab runtimes will not serve generated image files back to the browser. Set USE_DATA_URI = True in step 3 of the notebook and re-run steps 3, 5 and 6. The pictures are then embedded directly in the page instead of being fetched. Click-to-add-missed-damage turns off in that mode; everything else, PDF included, still works.

Data and licence

The damage detector was fine-tuned on CarDD, which is licensed for research use and may not be redistributed. The trained weights are published here; the dataset images are not, and none are included in this repo. If you intend to use these weights for anything beyond research or coursework, check CarDD's terms first.

The panel segmenter's training images were collected and labelled by the project team.

Credits

Built by Krit and Shane for CMU 24-679, Fall 2026. Models trained with Ultralytics YOLO11.