Lesson 14.5: Multi-Zone Vision Processing for Autonomous Scoring Selection

---

The Autonomous Scoring Selection Problem

One of the highest-value autonomous tasks in recent FTC seasons requires the robot to identify which of three possible spike mark locations contains the team's scoring element before the match begins. The robot then commits to a path that scores in the correct location. A team that solves this reliably earns those points every match; a team that hard-codes a single location scores them only one-third of the time.

This lesson combines every technique from the unit: a VisionPortal to open the camera stream, a custom VisionProcessor to analyze the frame, Rect zone boundaries to isolate the three candidate locations, and a result that the OpMode reads to select the correct autonomous path.

---

Using the HSV Color Space

Raw RGB channel values are sensitive to brightness. A red object under bright stadium lights will have very different RGB values than the same red object in a dimly lit practice space. This variability makes RGB-based thresholds fragile across competition venues.

The HSV color space separates Hue (the color itself), Saturation (the color's purity), and Value (the overall brightness). The Saturation channel is particularly useful for element detection: a colored game element has high saturation, while the grey field tiles surrounding it have very low saturation. Comparing saturation across the three zones finds the element reliably across a much wider range of lighting conditions than a red-channel comparison would.

Converting the frame from RGB to HSV is a single Imgproc.cvtColor() call:

Imgproc.cvtColor(frame, hsvMat, Imgproc.COLOR_RGB2HSV);

After conversion, calling Core.mean(submat).val[1] extracts the average Saturation value from a zone. The zone with the highest average saturation is the one containing the colored game element.

---

Passing the Result to the OpMode

The VisionProcessor runs on a background camera thread. The OpMode runs on the main thread. The result of the vision analysis needs to cross that thread boundary safely. The standard pattern is to store the result in a public member variable of the processor class and read it from the OpMode using the processor object reference. Because the processor updates the variable atomically (assigning a single reference or enum value), this pattern is safe without explicit synchronization in the FTC context.

// In the processor class:
public volatile SpikeLocation detectedLocation = SpikeLocation.UNKNOWN;

// In the OpMode, after the camera has processed a few frames:
SpikeLocation result = spikeProcessor.detectedLocation;

Using volatile gives writes to this shared reference visibility across the processor and OpMode threads. It does not make a multi-step update atomic, so share one enum or immutable result rather than mutating several related fields independently.

---

Annotated Code

package org.firstinspires.ftc.teamcode.vision;

import org.firstinspires.ftc.vision.VisionProcessor;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.Rect;
import org.opencv.imgproc.Imgproc;
import android.graphics.Canvas;

public class SpikeMarkVisionProcessor implements VisionProcessor {

    public enum SpikeLocation { LEFT, CENTER, RIGHT, UNKNOWN }

    // The OpMode reads this field to get the latest result
    public volatile SpikeLocation detectedLocation = SpikeLocation.UNKNOWN;

    private static final Rect ZONE_LEFT   = new Rect(50,  340, 80, 80);
    private static final Rect ZONE_CENTER = new Rect(280, 340, 80, 80);
    private static final Rect ZONE_RIGHT  = new Rect(510, 340, 80, 80);

    private final Mat hsvMat = new Mat();

    @Override
    public Object processFrame(Mat frame, long captureTimeNanos) {
        // Convert to HSV so Saturation can be used as the comparison metric
        Imgproc.cvtColor(frame, hsvMat, Imgproc.COLOR_RGB2HSV);

        // Extract average saturation from each candidate zone
        double satLeft   = Core.mean(hsvMat.submat(ZONE_LEFT)).val[1];
        double satCenter = Core.mean(hsvMat.submat(ZONE_CENTER)).val[1];
        double satRight  = Core.mean(hsvMat.submat(ZONE_RIGHT)).val[1];

        // The zone with the highest saturation contains the colored element
        if (satLeft > satCenter && satLeft > satRight) {
            detectedLocation = SpikeLocation.LEFT;
        } else if (satCenter > satLeft && satCenter > satRight) {
            detectedLocation = SpikeLocation.CENTER;
        } else {
            // Default to right if left and center are both lower
            detectedLocation = SpikeLocation.RIGHT;
        }

        return null;
    }

    @Override
    public void init(int width, int height,
                     org.firstinspires.ftc.robotcore.internal.camera.calibration.CameraCalibration cal) {}

    @Override
    public void onDrawFrame(Canvas canvas, int onscreenWidth, int onscreenHeight,
                            float scaleBmpPxToCanvasPx, float scaleCanvasDensity,
                            Object userContext) {}
}

note

This is an autonomous program. No gamepad input is used.

package org.firstinspires.ftc.teamcode;

import com.qualcomm.robotcore.eventloop.opmode.LinearOpMode;
import com.qualcomm.robotcore.eventloop.opmode.Autonomous;
import com.qualcomm.robotcore.hardware.WebcamName;
import org.firstinspires.ftc.vision.VisionPortal;
import org.firstinspires.ftc.teamcode.vision.SpikeMarkVisionProcessor;
import org.firstinspires.ftc.teamcode.vision.SpikeMarkVisionProcessor.SpikeLocation;

@Autonomous(name="Spike_Selection_Auto")
public class SpikeSelectionAuto extends LinearOpMode {

    private VisionPortal            visionPortal;
    private SpikeMarkVisionProcessor spikeProcessor;

    @Override
    public void runOpMode() {
        WebcamName webcam = hardwareMap.get(WebcamName.class, "Webcam 1");

        spikeProcessor = new SpikeMarkVisionProcessor();
        visionPortal   = VisionPortal.easyCreateWithDefaults(webcam, spikeProcessor);

        // Allow multiple frames to be processed before reading the result
        while (!isStarted() && !isStopRequested()) {
            telemetry.addData("Detected Location", spikeProcessor.detectedLocation);
            telemetry.addData("Instruction", "Confirm aim, then press Start");
            telemetry.update();
        }

        // Capture the final result the instant Start is pressed
        SpikeLocation spikeSolution = spikeProcessor.detectedLocation;

        // Stop streaming. The decision is made, so the camera is no longer needed.
        visionPortal.stopStreaming();

        if (opModeIsActive()) {
            telemetry.addData("Running path for", spikeSolution);
            telemetry.update();

            // Route to the correct autonomous path based on the vision result
            switch (spikeSolution) {
                case LEFT:
                    // driveToLeftSpike();
                    break;
                case CENTER:
                    // driveToCenterSpike();
                    break;
                case RIGHT:
                    // driveToRightSpike();
                    break;
                default:
                    // driveToDefaultSpike();
                    break;
            }
        }

        // This autonomous will not use the camera again.
        visionPortal.close();
    }
}

---

Fill-in-the-Blank Practice

1. To convert a camera frame from the RGB color space to HSV, use Imgproc.__________() with the constant Imgproc.COLOR_RGB2HSV.
2. In the HSV color space, index __________ of the array returned by Core.mean() corresponds to the Saturation channel.
3. The __________ keyword gives writes to a shared result field cross-thread visibility.

Show answers

1. cvtColor
2. 1 (index 1, after Hue at index 0)
3. volatile

---

Simulator Practice

Complete the simulator's processFrame(Mat frame) method for a two-zone processor. It should read average saturation from LEFT_ZONE and RIGHT_ZONE with the provided getAverageSaturation(frame, zone) helper, assign detectedLocation to SpikeLocation.LEFT if the left zone's saturation is higher, and assign SpikeLocation.RIGHT otherwise.

1.Complete the codeWork in fullscreen when you need more room for the editor and visualizer. Where file tabs are available, hover over the tab strip and choose + to create a file or reuse saved code.

2.Run the lifecyclePress Init, then Start. Use the available gamepad or scene controls.

3.Debug the behaviorCheck telemetry, motion, requirements, and hints before opening the answer.

Telemark Unit 14.5 Simulator

Loads the lesson-specific Telemark computer vision challenge with incomplete starter code for students to complete.

The simulator editor contains the starter code; use the answer below only after trying the challenge.

Show answer

void processFrame(Mat frame) {
    double leftSat  = getAverageSaturation(frame, LEFT_ZONE);
    double rightSat = getAverageSaturation(frame, RIGHT_ZONE);

    if (leftSat > rightSat) {
        detectedLocation = SpikeLocation.LEFT;
    } else {
        detectedLocation = SpikeLocation.RIGHT;
    }
}

Ready to move on?

Only mark complete if you genuinely understand the material. Your progress will be saved in this browser.

Stuck on this lesson?

Ask about anything on this page. It can see which lesson you have open and which part you are reading.