UAV Ground School

Reading the ground from the air.

Week 2 of UAVs@Berkeley Software Ground School 2026, solved end to end: HSV color segmentation with object centers, SUAS target shapes, and a from scratch pipeline that stitches drone video into one map. In Python and C++17.

Source on GitHub Try the color demo
stack
Python 3.10+, OpenCV, NumPy, C++17
vision
HSV masks, CIELAB k-means, IoU shape templates
mapping
ORB/SIFT tracking, loop closure, bundle adjustment, multi band blending
accuracy
0.36 px pose RMSE on the rough flight, down from 56.72 px
A top down map of a blocky world with lakes, sand, forest and snow, assembled from 78 video keyframes by the stitching pipeline
Stitched from a simulated 5 pass survey with barrel distortion, motion blur, sensor noise and exposure drift. 78 keyframes, 254 loop closures

Pick one option. We did both.

The Week 2 skill booster covers computer vision for aerial imagery. Each option below lists the assignment as given and what the repo ships on top of it.

  1. Option 1

    Color Me Impressed

    Split an image into one image per color with OpenCV HSV masking. Bonus: print the center of each colored object.

    • Exact HSV partition: twelve bands, every pixel lands in exactly one color.
    • Speckle removal and 8-connected components, so each object gets its own center.
    • Palette discovery with k-means in CIELAB, k chosen by silhouette score.
    • SUAS shape classification by rotation searched IoU template matching: 100% on 13 shapes across sizes and rotations.
  2. Option 2

    I'll be Needin' Stitches

    Split a drone flight video into frames and stitch them into one image spanning the flight path, like the club did in competition.

    • Flight simulator: a procedural voxel world and lawnmower survey with exact ground truth poses.
    • Grid bucketed ORB/SIFT, symmetric ratio matching, keyframe tracking with relocalization.
    • Loop closure and Levenberg Marquardt similarity bundle adjustment with complex number Jacobians.
    • Brown and Lowe gain compensation, Laplacian pyramid multi band blending, and a benchmark against cv2.Stitcher.
  3. Extra challenge

    Do it in C++

    Solve the challenges with the OpenCV C++ API.

    • C++17 ports of both options, built with CMake against OpenCV 4 or 5.
    • CI compiles both binaries and runs parity tests against the Python implementation.

Color Me Impressed, running in this tab.

The named mode of colors.py, ported to JavaScript. The HSV conversion reproduces OpenCV's 8-bit fixed point math: across all 16,777,216 RGB colors it agrees with cv2.cvtColor exactly on hue and value and within one step on saturation. Twelve inclusive bands split the image, a 3x3 elliptical opening removes speckle, and 8-connected components give every object a center: the mean of its pixel coordinates.

Pick a test image or bring your own. Uploads are decoded and processed locally and never leave the browser.

Splitting colors
Showing the original with object centers. Drop an image here to try it.

Objects and centers

Largest first per color, like format_report.

    Twelve bands, no gaps, no overlaps

    OpenCV stores hue as 0 to 179 (half degrees) and saturation and value as 0 to 255. Hue wraps, so red needs two ranges. Brown is dark orange and dark yellow, carved out by value at or below 150. Anything with saturation of 50 or less is black, gray or white by brightness alone.

    Colors covering under 1% of the image are dropped unless they contain a real object, and blobs smaller than 0.08% of the image count as noise. That is how a thin white letter survives while JPEG fringes do not.

    Not ported: shape classification. The Python and C++ versions also label each object (octagon, star, cross) by rotation searched IoU template matching. Where the browser table shows a shape, it comes from the precomputed Python report for that test image.

    Hue 0 to 179 across, value 255 to 0 down, saturation 255.
    Saturation 0 to 255 across, at hue 60.

      I'll be Needin' Stitches.

      The club's competition footage is not public, so the repo makes its own: a seeded Minecraft style world and a lawnmower survey flight rendered to video, with the true camera pose of every frame. Stitching quality is then measured in pixels instead of eyeballed.

      Ground truth voxel world seen from directly above: lakes, beaches, forest, rock and snow
      Ground truth world. The drone never sees it whole.
      A single frame from the rough flight: a small, noisy, slightly blurred patch of grass and water
      One frame of the rough flight: barrel distortion (k1 = -0.06), motion blur, heavy noise, exposure drift of up to 18% and altitude wobble.

      Pipeline

      1. Sample

        Keep every Nth frame, resize, and undistort with the radial model.

        cv2.VideoCapture, frame % every

      2. Track

        Grid bucketed features after CLAHE, symmetric ratio test, RANSAC similarity against the current keyframe. Lost frames are relocalized.

        ORB or SIFT, 4 DOF RANSAC + LM

      3. Close loops

        Predict which earlier passes overlap each keyframe, then verify the candidates with feature matching.

        footprint overlap >= 25%

      4. Bundle adjust

        Solve every similarity at once. Residuals live in frame pixels, so frames cannot shrink to cheat.

        z → αz + β, LM + Huber IRLS

      5. Gain compensation

        One exposure gain per frame from mean intensities of every overlapping pair.

        Brown and Lowe, IJCV 2007

      6. Blend

        Low frequencies mix over wide seams, high frequencies over narrow ones.

        Laplacian pyramid, written from scratch

      Loop closures add edges between non consecutive keyframes. They are what bundle adjustment uses to pull accumulated drift back out.

      Drag to remove the drift.

      Chaining pairwise alignments compounds small errors: after five passes the sequential mosaic is off by 56.72 px RMSE (worst keyframe 159.65 px). Loop closure, bundle adjustment and undistortion bring it to 0.36 px.

      Sequentially chained mosaic with visible seams, doubled shorelines and warped edges Full pipeline mosaic with clean shorelines and consistent exposure Sequential chaining Full pipeline

      The flight path, recovered from pixels alone.

      Every white outline is a keyframe footprint and the orange track joins their centers. No GPS, no IMU: the lawnmower pattern falls out of the image alignment.

      The stitched map overlaid with rectangular keyframe footprints and an orange lawnmower flight track with five passes

      Against cv2.Stitcher.

      OpenCV's built in stitcher in SCANS mode was handed 67 frames of the calm flight and took 103.9 s to return the mosaic labeled cv2.Stitcher. This pipeline stitched the same flight in 20.1 s with 0.30 px pose RMSE and a ZNCC of 0.979 against the true world.

      cv2.Stitcher67 frames, 103.9 s

      cv2.Stitcher output for the calm flight, built from the sampled frames

      This pipeline20.1 s, 0.30 px RMSE

      Full pipeline mosaic of the calm flight: a clean rectangular map with lakes, forest and snow

      Benchmark

      Two simulated flights, stitched with progressively more of the pipeline switched on and scored against the simulator's true poses and world. Generated by python -m week02 benchmark docs/week02.

      Lower pose error is better; ZNCC of 1.0 means the mosaic matches the true world exactly.
      Variant Keyframes Loop closures Pose RMSE (px) Max error (px) ZNCC Time (s)
      Calm flight3 passes, light noise, no lens distortion
      sequential chaining5001.432.510.94112.8
      + loop closure and bundle adjustment501520.300.700.97618.5
      + gain compensation and multi band blending501520.300.700.97920.1
      cv2.Stitcher SCANS (67 frames)finished, no poses to score103.9
      Rough flight5 passes, barrel distortion k1=-0.06, motion blur, heavy noise, +/-18% exposure, +/-5% altitude
      sequential chaining75056.72159.650.48620.8
      + loop closure and bundle adjustment751997.2117.560.83927.3
      + gain compensation and multi band blending751997.2117.560.86725.5
      + lens undistortion (full pipeline)782480.370.910.94136.1
      full pipeline with SIFT782540.360.630.94774.7
      Pose RMSE and max error
      Error on a 5x5 grid of points in every keyframe, after a least squares similarity alignment to the true poses.
      ZNCC
      Zero normalized cross correlation between the mosaic and the true world, so blending and exposure count too.
      Time
      Wall clock seconds summed over the pipeline stages, single machine, same settings for every row.

      The extra challenge, in C++17.

      Both options are ported to OpenCV's C++ API, with the same bands, thresholds and pipeline stages as the Python modules.

      • color_me_impressed: HSV bands, object centers, IoU shape templates, JSON report.
      • needin_stitches: keyframe tracking, loop closure, complex number bundle adjustment, gain compensation, multi band blending, or cv::Stitcher for comparison.
      • One CMakeLists.txt, builds against OpenCV 4 (apt) or OpenCV 5 (Homebrew).
      C++17week02/cpp
      $ cmake -S week02/cpp -B build/cpp -DCMAKE_BUILD_TYPE=Release
$ cmake --build build/cpp -j
$ ./build/cpp/color_me_impressed docs/week02/colors/targets.jpg --out out/targets
$ ./build/cpp/needin_stitches out/flight/flight.mp4 --k1 -0.06 --out out/stitched.jpg
      Pythonpython -m week02
      $ pip install -e .
$ python -m week02 colors docs/week02/colors/targets.jpg --out out/targets
$ python -m week02 simulate out/flight --hard
$ python -m week02 stitch out/flight/flight.mp4 --k1 -0.06 --path-overlay out/path.jpg
$ python -m week02 benchmark docs/week02