Computer Science
Gilmour AcademyLancerTech
  • Our Curriculum and Department
  • Intro to Programming
    • 1: Parts of a Computer
    • 2: Parts of Python
    • 3: DRY Turtle
    • 4: Turtle Design App
    • Wordle with Turtles
    • 5: Interactive Turtles
    • OLD 5: Replit, GitHub, and repositories (Oh my!)
    • 6: Raspberry Pi / GoPiGo
    • 7: Kivy
  • Intro to Web Design
    • 1: Internet?
    • 2: Websites?
    • 3: Bootstrap Template
    • 4: Graphics and Branding
    • 5: Collaboration
    • 6: Advanced Editing
    • Publish Static HTML
  • AP Computer Science
    • 1: Logic & Instances
    • 2: How Java Works
    • 3: Data Types & Flow
    • 4: Strings
    • 5: Objects & References
    • 6: Inheritance & Algorithms
    • 7: Data Structures
    • 8: Sorting
    • 9: Review
    • Data Science
  • Web App Dev
    • 1: Core Concepts
    • 2: MVT Pattern
    • 3: Hello Flask
    • 4: Install Flaskinni
    • 5: Tour Flaskinni
    • 6: Visualize Your App
    • 7: Theme & Blueprint
    • 8: Standup Your DB
    • 9: Advanced Topics
    • 10: Deployment
  • 2D Game Design
    • Class Overview
    • Gamemaker Studio 2 and Github Setup
    • Game 1: Bouncing Ball
    • Turning in your games
    • Game 2: Maze
    • Game 3: Ping Pong
    • Game 4: Breakout
    • Game 5: Tank Battle
    • Game 6 Highlights
    • DO NOT DO:
    • Game 7: Final Project
    • Publish to Opera
    • FAQ
  • 3D Game Design
    • 1: Class Overview
    • 2: Installation
    • 3: Exploring the Unity UI
    • Game 1: Rolling Ball
    • Game 2: Tanks
    • Game 3: Third Person Platformer
    • Game 4: Final project
    • FAQs
    • OLD: Distance Learning Setup
    • OLD: GIT
  • 3D Modeling & Fabrication
    • Installation
    • Fusion 360 Interface and Sketch Modeling
    • Primitive Modeling
    • Patterns
    • Appearances and Rendering
    • Building Community Gallery Page 2023
    • Parametric Modeling
    • 3D Printing Concerns
    • Assemblies and Mechanical Design
    • Laser Cutting
    • Sculpt Tools
    • Milling Concerns
  • Robotics 7
    • Software Installation
    • Python basics (trinket.io)
    • Python Turtle
    • Programming for the Ev3
    • Setting up for clarity
  • Robotics 8
    • Replit
    • Python review
    • Kivy Basics
    • Calculator
  • Competitive Robotics
    • Hardware Team
      • CAD Examples
      • Elevators
    • Software Team
      • Command Pattern
      • Example Command
      • Subsystem
      • Running Your Code
      • Under the Hood
      • RoadRunner
      • Vision Processing
  • Archives
    • Adiletta Archives
      • Old Web
        • Ex: WordPress CMS
      • ItP
        • OLD: Parts of Python (old -- Mr. A)
        • OLD: 5: Raspberry Pi
        • OLD: 6: Deploying Code
        • OLD 7: Nav Algorithm
    • Vanek Archives
      • OLD Robotics 8
        • OLD: End of Class Project
      • OLD Competitive Robotics
        • Untitled
        • Webots Videos
      • OLD Robotics 7
        • Trinket Introduction
        • Lists: x/y position
        • Functions: Math program
        • Lists: Grocery List
        • Study Guide Program
        • Tic Tac Toe Game
        • Dice Roller Program
        • Visualization
        • Dice Roller + Visualization
        • OpenSCAD: Installation
        • OpenSCAD: Command Sheet and Intro
        • OpenSCAD: Difference
        • OpenSCAD: Variables
        • OpenSCAD: Union
        • OpenSCAD: For Loops
        • OpenSCAD: Final Project
      • OLD Art I - Blender Sculpting
        • Class Overview
        • Installation
        • Lesson 1 - Tools
        • Lesson 2 - Detail
        • Lesson 3 - Base Mesh: Metaballs
        • Lesson 4: Converting metaballs and adding detail
        • Lesson 5: Masking, Hiding, and Working with Multiple Objects
        • Lesson 6: Joining Objects & Basing
        • Lesson 7: Sculpture Painting
        • Student Gallery: Animal Sculpts
        • Lesson 8: 3D Compositon
        • Lesson 9: The Project - Putting it all together
        • Lesson 10: Developing the image further
        • Lesson 11: Layout the base metaball mesh.
        • Lesson 12: Final Detail
        • Lesson 13: Basing and Painting
        • Final Project Gallery
      • OLD Fab
        • OLD Building Community Project Gallery
        • Copy of Building Community Project Gallery
        • old Building Community Project Gallery
      • OLD: Turtle Design App
      • OLD Arduino Robotics 8
        • Arduino Basic Commands Cheat Sheet
        • Logging into Tinkercad
        • Arduino, Circuits, LEDs and Resistors
        • Functions and Variables
        • Serial Monitor
        • Buttons and Interrupts
        • Traffic Light Project
        • Potentiometers + Servos
        • Piezo Buzzer and Tone();
        • Sequencer Project
        • Arrays and for loops
        • Extra Loop Practice
        • Refining the Sequencer
        • Servos
        • Ultrasonic Sensors
        • Final Project
Powered by GitBook
On this page
  • What is OpenCV?
  • Basics on our Structure
  • Vocabulary
  • Pseudocode

Was this helpful?

Export as PDF
  1. Competitive Robotics
  2. Software Team

Vision Processing

Let's get our robot to recognize certain objects and their relative position.

PreviousRoadRunnerNextAdiletta Archives

Last updated 7 months ago

Was this helpful?

What is OpenCV?

OpenCV began as an Intel research project in 1999, aiming to accelerate real-time computer vision tasks. Its 2002 open-source release fostered a large developer community, leading to its growth into a powerful and versatile library for various computer vision applications across fields like robotics, self-driving cars, and medical imaging.

OpenCV's success as a powerful yet general-purpose library sparked the development of more specialized solutions. EasyOpenCV (EOCV) emerged within the FTC (FIRST Tech Challenge) robotics community. It leverages OpenCV's core functionalities but simplifies them and adds robot-specific features for easier integration with FTC robots. This allows FTC teams to focus on their robot's vision-based tasks without needing in-depth computer vision expertise.

Basics on our Structure

Vocabulary

  • Tensor: A multi-dimensional array used to represent data in machine learning models. Tensors are the fundamental data structures in deep learning frameworks like TensorFlow and PyTorch.

  • Inference: The process of running a trained machine learning model on new input data to obtain predictions or outputs.

  • Allocate Tensors: The process of allocating memory for tensors in the model's input and output layers. This is typically done before running the model's inference on input data.

  • Output Tensor: The tensor(s) produced by the machine learning model after running inference on the input data. The output tensor(s) contain the model's predictions or outputs, such as object bounding boxes, class probabilities, or segmentation masks.

  • Non-Maximum Suppression (NMS): A post-processing technique used in object detection to eliminate redundant overlapping bounding boxes for the same object. NMS helps to ensure that only the most confident and accurate bounding box is retained for each object.

Pseudocode

This is rough AI-generated code that has served our class well as a springboard conversation.

// Import necessary libraries
import org.opencv.core.Mat;
import org.opencv.core.Rect;
import org.opencv.core.Point;
import org.opencv.imgproc.Imgproc;
import org.opencv.core.Core;
import org.opencv.core.Scalar;
import org.tensorflow.lite.Interpreter;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

// Define a class for the ObjectDetectionPipeline
public class ObjectDetectionPipeline extends EasyOpenCvPipeline {

    // Declare variables for model and labels
    private Interpreter tfliteInterpreter;
    private String[] labels;

    // Constructor to load the TensorFlow Lite model and labels
    public ObjectDetectionPipeline(String modelPath, String labelsPath) throws IOException {
        // Load TensorFlow Lite model from file
        tfliteInterpreter = new Interpreter(FileUtil.loadEssentiaModel(modelPath));

        // Load labels from file (assuming each line represents a label)
        labels = FileUtil.readFile(labelsPath).split("\\n");
    }

    @Override
    public Mat processFrame(Mat input) {

        // Pre-process the image for the TensorFlow Lite model (e.g., resize, normalize)
        Mat processedImage = preProcessImage(input);

        // Run inference on the pre-processed image
        runInference(processedImage);

        // Get detection results (bounding boxes and class IDs)
        List<Rect> boundingBoxes = getBoundingBoxes();
        List<Integer> classIds = getClassIds();

        // Draw bounding boxes and labels on the original image
        for (int i = 0; i < boundingBoxes.size(); i++) {
            Rect box = boundingBoxes.get(i);
            int classId = classIds.get(i);
            String label = labels[classId];

            // Draw bounding box and label on the image using OpenCV functions
            Imgproc.rectangle(input, box.tl(), box.br(), new Scalar(255, 0, 0), 2);
            Imgproc.putText(input, label, box.tl(), Core.FONT_HERSHEY_SIMPLEX, 1.0, new Scalar(255, 0, 0), 2);
        }
        return input;
    }

    // Helper methods to pre-process the image, run inference, and get results
    private Mat preProcessImage(Mat image) {
        // Implement your specific pre-processing steps here (e.g., resize)
        return image;
    }

    private void runInference(Mat image) {
        // Allocate tensors based on the model's input and output requirements
        tfliteInterpreter.run(new Object[]{image.toArray()}, null);
    }

    /**
     * This method is responsible for extracting the bounding box coordinates from the model's output tensor.
     * The implementation details will depend on the specific format of the output tensor produced by your
     * TensorFlow Lite model.
     *
     * Below is an example implementation that assumes the output tensor is a 4D float tensor with the following shape:
     * [batch_size, num_detections, 4 (coordinates), 2 (class_id and confidence)]
     *
     * Each detection is represented as a vector of length 6, where the first 4 values represent the normalized
     * bounding box coordinates [y_min, x_min, y_max, x_max], and the last two values represent the class ID and
     * confidence score, respectively.
     *
     * Note: This is just an example implementation, and you will need to modify it according to your model's
     * output format.
     */
    private List<Rect> getBoundingBoxes() {
        List<Rect> boundingBoxes = new ArrayList<>();

        // Get the output tensor from the interpreter
        float[][][][] outputTensor = tfliteInterpreter.getOutputTensor(0);

        // Iterate over the detections
        for (int i = 0; i < outputTensor[0].length; i++) {
            // Extract the bounding box coordinates and confidence score
            float[] detection = outputTensor[0][i][0];
            float yMin = detection[0];
            float xMin = detection[1];
            float yMax = detection[2];
            float xMax = detection[3];
            float confidence = detection[5];

            // Apply a confidence threshold (e.g., 0.5)
            if (confidence > 0.5) {
                // Denormalize the bounding box coordinates
                int imageHeight = /* Get the input image height */;
                int imageWidth = /* Get the input image width */;
                int left = (int) (xMin * imageWidth);
                int top = (int) (yMin * imageHeight);
                int right = (int) (xMax * imageWidth);
                int bottom = (int) (yMax * imageHeight);

                // Create a Rect object and add it to the list
                Rect rect = new Rect(left, top, right - left, bottom - top);
                boundingBoxes.add(rect);
            }
        }

        return boundingBoxes;
    }

    private List<Integer> getClassIds() {
        // Extract class ID data from the model's output tensor
        // ... (implementation details depend on the model's output format)
        return new ArrayList<>();
    }
}