← Portfolio · P-03
Vision + Agents · Real-time
SkyPilot — a voice-commanded autonomous drone.
A conversational control stack that converts spoken intent into either direct flight commands or continuous target tracking — voice input, LLM routing, and real-time computer vision in one control loop.
Natural language to autonomous action.
Voice transcription, LLM intent parsing, and a vision loop decide whether the drone executes a discrete movement command or enters a continuous tracking mode.
Whisper transcribes spoken commands so the drone can be controlled through conversational language instead of a rigid command console. An OpenAI-powered agent then routes that intent one of two ways: discrete flight actions — turns, altitude changes, repositioning — or longer-running follow behavior, where YOLOv8 and OpenCV maintain a target lock and recover through occlusion. The architecture is easiest to understand as four cooperating scripts that hand control from transcription to intent parsing, then into flight execution or follow mode.
voice_transcriber.py
Handles speech-to-text using Whisper.
openapi.py
Uses an OpenAI-backed agent and custom tools to parse user intent.
drone_controller.py
Executes discrete movement instructions like turns, altitude changes, and repositioning.
skytrack.py
Runs the live YOLOv8 and OpenCV tracking loop for follow mode.
Real-time coordination mattered as much as model choice.
Getting the loop to feel responsive was less about which model to call and more about threading, timing, and signal quality.
- Multi-threading was needed for heartbeat, video feed, command routing, and continuous control.
- Smooth tracking required tuning acceleration and update timing to avoid overshoot.
- Vision design balanced OpenCV speed against YOLOv8 robustness for re-acquiring targets.
- Voice isolation was important to reduce background noise and false triggers.
