After nearly calling it quits three times throughout TreeHacks, all-sophomore team Recon was awarded “Most Technically Challenging Project” for its search and delivery hack on consumer drones at the end of the weekend. The award was one of the three hackathon final prizes.
Named after the term “reconnaissance,” Recon is a functioning proof of concept of three consumer drone hacks: object recognition, voice-activated image recognition and voice-activated drone control.
Courtesy of Stanford’s Unmanned Aerial Vehicle Enthusiasts Club (SUAVE), team members Chaitanya Asawa ’18, Andrew Chen ’18, Noa Glaser ’18 and Marcus Gomez ’18 arrived at the hackathon with a drone —- and with an idea in mind. They planned to work on drone-aided search for something as small as lost keys or backpacks, with the broader implications of assisting search-and-rescue missions.
“We just wanted to build something cool [with drones],” Glaser said. “We’re going to have to be smart about how we integrate them into our lives, but drones are an exciting new technology that can be used for social good.”
Their plan would be thwarted from the get-go. Even before the hackathon, they almost didn’t show up because they discovered that though the SUAVE’s Phantom drone had the camera capabilities for their search function, the drone was unprogrammable.
“[The drone] was built for consumers who don’t code them, don’t hack them,” Asawa said.
“We’re limited in how we’re tinkering with the drone,” he added.
Asawa, Chen, Glaser and Gomez arrived nonetheless and later checked out a programmable Parrot drone from TreeHacks, which not only lacked cameras but also did not have workable batteries.
After spending all Friday attempting to program the Phantom drone, they managed to buy, with TreeHacks’ reimbursement, another Parrot drone early Saturday morning. But because of the limitations on each, Asawa, Chen, Glaser and Gomez had to split the three tasks between the two drones. Phantom aided in the search process by capturing videos for object recognition, while Asawa and Gomez later hacked the Parrot drone to receive voice commands from Amazon Alexa.
The Phantom’s limitations also meant that the team could not directly process live video to recognize objects. Instead, they streamed a video recording from the drone surveying a beach ball-scattered Lake Lagunita Saturday afternoon to an Android phone, which stored screenshots of the video. Chen then used a technique called image segmentation to more accurately analyze the screenshots and recognize objects in the video.
“I’m kind of glad we didn’t get one big drone with everything we wanted,” Chen said. “We had to get creative.”
As Chen worked on the image processing strand inside Huang Saturday night, Asawa and Gomez positioned themselves outside in the courtyard, working on interaction between the Parrot drone and an Amazon Echo. Before midnight, they were able to transform Amazon Alexa’s voice-to-text transcription into drone commands to communicate with the Parrot drone to launch — and, with the command “impress me,” lift into an aerial flip. Later into the night, they also had working functionality of voice-controlled drone delivery, simulated with a bottle cap attached to the Parrot drone and a penny.
Finally, the team was able to implement voice-command computer vision, essentially training the classifier to search for an image mentioned — for instance, “imagine a beach ball” — harnessing natural language processing and the IBM Watson and Bing machine learning platforms.
Along with the “Most Technically Challenging” TreeHacks finalist prize, Recon was awarded sponsor prizes from Amazon Alexa, Amazon Web Services and IBM Bluemix. They also received a $1,000 grant from 1517, a venture capital fund, to continue working on the project.
“A lot of parts are going to be redone, but hackathons are about having an idea, knowing you can do it and being able to iterate over it quickly,” Glaser said.
“It’s something we definitely want to work on, because, well, it works,” Gomez added.
And among their prizes: Amazon Echo and Parrot drones.
Contact Irene Hsu at ihsu5595 ‘at’ stanford.edu.