Using Motion Capture Cameras, Queen’s University’s Human Media Lab Help Create Flying Lego Bricks that Could Mimic Real World Movements

For more than half a century, Lego building blocks have sparked the imaginations of both children and adults across the globe, helping to transform imaginative concepts into concrete (or, more accurately, plastic) reality. Now, thanks to a collaboration between Queen’s University in Ontario, Canada and Lego Creative Play Lab, Lego’s bricks are finding their way into a new dimension.

By applying passive markers to colored Lego bricks, researchers at Queen’s Human Media Lab (HML) are able to create a flying robotic swarm capable of mimicking the movement and placement of specific Lego bricks. To demonstrate this, HML created a flexible, gyroscopic controller that looked like a butterfly. Users could then place red, green or blue bricks on the butterfly controller, and small quadcopter drones with matching colors flew into the air in a formation exactly matching the layout on the controller. Users could then turn, flip or bend the controller and the drones would mimic the movements in real-time.

Swarm robotics is still a relatively young field in the world of robotics, in part due to the technology needed to achieve an accurate swarm. It’s one thing to command several small robots all at once, but it’s another to have them move in unison – that requires precision tracking.

To record drones in flight, the team at HML currently uses Vicon T-series motion capture cameras, capable of tracking dozens of moving objects at once at 120 frames per second. These cameras can isolate and record the movement of each drone without issue, then send that data to Vicon’s Tracker 3 software, where it can be controlled using a peripheral device -- even a Lego controller.

HML showed off its cool new Lego-infused project in February 2018 at the Lego World Expo in Copenhagen, Denmark, where attendees young and old took to the skies with their designs. While this might be one of the most sophisticated uses of Lego bricks to date, the brick-maker is no stranger to robotics.

In 1998, Lego introduced the “Lego Mindstorms” line, introducing “intelligent bricks,” capable of receiving basic programming that allowed them to perform simple, task-oriented movements. The product line included a brick computer that controls the system. It worked with a set of modular sensors and motors, along with Lego “Technic” bricks designed to build mechanical systems. Lego’s robotics endeavor created a relatively small but passionate fanbase, leading to global competitions and ushering a new scientific-focused side to the massive Lego brand.

As youth robotics programs have proliferated, competitions like FIRST Lego League -- where young builders create Lego-based solutions to real-world problems -- have kept pace. Given Lego’s history with robotic systems, it made the partnership with HML a natural fit.

Queen’s University’s HML is known for its groundbreaking work developing flexible displays, having created interactive displays (read: touchscreens) on non-flat surfaces like soda cans and even bendable tablets. The collaboration with Lego is an extension of HML’s BitDrone system, which initially debuted in 2015.

HML’s BitDrones project began as a 3D modeling tool. Programmers used it to model several unique scenarios, including one where you could navigate a file folder using touchpads on drones. Each drone contained a digital readout to represent files or subfolders, and the Vicon system recorded the movements. Lego bricks weren’t yet in the equation, but even then HML Director Dr. Roel Vertegaal acknowledged the system’s resemblance to the building blocks.

“They sort of resemble flying Lego bricks, but the application goes beyond simple toys,” said Vertegaal.

Currently, HML’s BitDrones are relatively large, which works fine for demonstrations using Lego bricks. The goal is to make them smaller, so that the swarm can act like a form of interconnecting programmable matter. In a similar vein, HML aims to make the system work with far bigger drone swarms. While users at the Lego World Expo controlled no more than ten or fifteen drones at once, the team at Queen’s is hoping to push that number into the hundreds, or even the thousands.

Although amazing crowds is a nice bonus, the program’s potential reaches in multiple directions, opening up new possibilities for students and professionals alike. For instance, “DisplayDrones” -- one of three BitDrones revealed at the 2015 ACM Symposium for User Interface Software and Technology -- are fitted with high-resolution cameras of their own, allowing users to control the drones with simple head movements while teleconferencing. With this tech, prospective buyers could remotely inspect a home, business partners could take a self-guided tour of a model property and much more.

Vertegaal also believes the technology could help revolutionize the ways in which we teach physics to children.

“We believe [the BitDrones project] has the potential to take experiential learning to an entirely new level,” said Vertegaal. “With this technology, we are able to simulate the physics of the natural world like gravity, planetary orbits, and more, giving children a chance to see what they have long learned from textbooks and two-dimensional depictions, in a real physical environment.”