In the summers of 2016 and 2017, I worked in a art and technology lab at CMU on an experimental interactive performance system called Dranimate.
How does it work?
Drawings are placed under a webcam/lighting rig which captures an image of the drawing. A computer-vision system processes the image and creates a two-dimensional mesh that can be squished and stretched using an algorithm called As Rigid as Possible. These virtual puppets are then rigged and controlled with hand gestures by using a Leap Motion Controller.
For more technical information on Dranimate, take a look at Dranimate: Rapid Real-time Gestural Rigging and Control of Animation (UIST 2015).
My role at Dranimate was leading development and design of the software system. I was also responsible for instructing several interns on how to contribute to the codebase and teaching various C++, computer vision, and computer graphics concepts.
I also often acted as the Dranimate Puppeteer at various test sites and performances - see the vieo above!