GT ECE 4180 Final Project Spring 2025

Click for Demo Video

Overview

A proof of concept for a presenter tracking camera with intuitive gesture controlled 2-axis pan and tilt. By prioritizing gesture control over head tracking, presenters can easily divert the camera's attention to specific areas, allowing for more immersive remote presentations compared to footage produced by static viewpoint, multi-region selection, or head tracking systems without gesture control.

Hardware

• Raspberry Pi Zero 2 W
  • Video footage capture/streaming, 2-axis servo control, respond to remote commands
• mbed LPC1768
  • Laser control, sending commands to tracking camera over Bluetooth
• Inference Accelerator
  • Presenter and gesture detection
  • For the purposes of this project, a MacBook with Metal Performance Shaders backend was used to emulate the functionality of a dedicated inference accelerator like the Coral Edge TPU
• Raspberry Pi Camera v2
  • Video footage capture
• Hitech HS-422 Servo (x2)
  • 2-axis camera orientation control
• DSD TECH HC-05
  • Bluetooth communication between mbed and RPi
• LM2596 DC-DC Buck Converter
  • Step down 9V power supply for RPi and servos
• KY-008 Laser Module
  • Laser pointer functionality
• Miscellaneous
  • Camera status LED
  • HDMI Video Capture Card
  • Wireless microphone
  • Mini-HDMI to HDMI adapter
  • Mini-USB to USB-A adapter
  • Ultra thin HDMI cable
  • Micro-USB to USB-A cable
  • 32GB SD Card
  • 5V Power Bank
  • 9V Power Supply

Software

• Tracking Camera
  • Libraries:
    • pigpio
    • Picamera2
    • pySerial
    • NumPy
  • System:
    • g_serial
    • LightDM (Disabled)
• Inference Accelerator
  • Libraries:
    • PyTorch
    • Ultralytics (YOLOv11)
    • NanoTrack
    • OpenCV
    • NumPy
    • pySerial
• Remote
  • N/A
• Miscellaneous
  • OBS Studio (HDMI Capture Card Virtual Camera)

Schematic

Setup Details

Raspberry Pi

• First boot
  • In /boot/firmware, edit config.txt and cmdline.txt to match respective files in src/Camera
    • Note: Replace <UUID> in the provided cmdline.txt with the original UUID from the RPi Imager
  • Afterward, sudo reboot
• Install Libraries

  sudo apt update
  sudo apt install python3-picamera2
  sudo apt install pigpio
  sudo systemctl enable pigpiod
  sudo systemctl start pigpiod
  sudo pip install pyserial --break-system-packages

• Pair Bluetooth

  sudo systemctl start bluetooth
  sudo systemctl enable bluetooth
  bluetoothctl
  scan on

  • Look for the MAC address of DSD TECH HC-05

  pair <MAC ADDRESS>
  trust <MAC ADDRESS>
  connect <MAC ADDRESS>
  exit

• Disable Desktop Manager

  sudo systemctl stop lightdm
  sudo systemctl disable lightdm

• Create Script Files
  • In the root directory:

  mkdir TrackingCamera
  cd TrackingCamera

  • sudo nano <FILENAME>.py for every script file in src/Camera
    • Note: Replace BLUETOOTH_MAC_ADDRESS in config.py with the MAC address from the bluetooth pairing step
• Create Systemd Service
  • Create a service file: sudo nano /etc/systemd/system/tracking_camera.service
  • Edit to match tracking_camera.service

  sudo systemctl daemon-reload
  sudo systemctl enable tracking_camera
  sudo systemctl start tracking_camera

Inference Accelerator

• Install Libraries
  • pip install -r src/Accelerator/requirements.txt
• Run
  • In src/Accelerator/main.py, change PORT to RPi portname
  • python src/Accelerator/main.py

Remote

• Upload Binary
  • Drag and drop Remote.bin onto mbed
• Run
  • Power cycle to start execution

Usage

• Tracking
  • Toggled on or off using the top (black) button on the remote
  • Status LED is powered on when tracking is enabled
• Laser Pointer
  • Activated by the center (red) button on the remote
• Target Changing
  • Controlled by the bottom (white) button on the remote where every click iterates to the next target ID and eventually loops back to the first target ID

Issues / Unimplemented Improvements

• Impractical Laser Control
  • The original goal of this project was to guide the camera's focus using a laser rather than gestures as it would allow for longer range control. However, the combination of significant camera noise and a low power laser (for safety reasons) made the task of reliably detecting a laser point impractical. Potentially with a higher quality camera or dedicated low exposure camera, reliability could be improved to usable standards.
• Inconvenient External Inference
  • Offloading inference to a completely external device is both inefficient and unreliable. A dedicated edge accelerator would allow for better performance and more efficient use of resources.
• Footage Quality
  • As a proof of concept, the Raspberry Pi Camera v2's noisy frames combined with the parallel compute bottleneck of the Raspberry Pi Zero 2 W severely degrade the final recording quality. A higher quality camera and better choice of processor (ideally extensive support for parallel operations) could easily elevate the output quality to acceptable levels.
• Inefficient Use of Hardware
  • Given the simplicity of the overall system, the microcontrollers are by no means being used in an efficient manner. Choosing microcontrollers more tailored to the requirements of the system would make it much more reasonable to use.