THEATER OF FUTURE PUPPETRY
Type
Interactive System
Role
XR Software Engineer
Date
May 2022 - Jan 2025
Client
National Science and Technology Council (NSTC)
Credits
Production: Chun-Cheng Hsu Lab
IoTtalk: Yi-Bing Lin Lab
Character Model: NTUA Experimental Game Lab
An immersive puppetry system that integrates VR gesture recognition, multi-user mobile interaction, and traditional Taiwanese glove puppetry. It enables audiences to engage with puppetry in innovative ways while contributing to cultural heritage preservation.
2025 — DiversePuppetry, Posters, SIGGRAPH, Vancouver Convention Centre, Canada
2025 — Theater of Future Puppetry, Student Innovation, SXSW, USA
2024 — Theater of Future Puppetry, XR Program, SIGGRAPH Asia, Tokyo International Forum, Japan
2024 — Future Puppet Theater, FUTEX, Taipei World Trade Center, Taipei
2024 — Heart of the Machine, Ars Electronica, Austria
In VR, users control five classic puppet roles via Quest hand tracking. The gesture set, inspired by Budaixi and co-designed with masters, is implemented with the Meta SDK and mapped to real-time puppet actions.
To bring puppetry into public spaces, we created a mobile-based system with immersive projection that enables multi-user participation. Visitors scan QR codes to select roles, animate puppets, and interact collaboratively in real time.
Each platform reveals unique ways to experience puppetry. Building on these strengths, we developed DiversePuppetry, an asymmetric system that integrates VR-HMD, MR-HMD, and VR-CAVE to animate digital puppets across devices.
This approach enables collaborative performances and reimagines the future of puppet theater.
Process & Experiments
VR Puppetry
Developed an extensible real-time animation system that integrates Meta Interaction SDK Hand Pose Detection and custom gesture recognition to drive expressive character animation.
Mobile Puppetry
Built a multi-user web interface (Node.js/WebSocket) to stream mobile IMU data into Unity, enabling custom motion-driven character animation, and deployed the production system on IoTtalk.
