Minecraft IRL turns Minecraft into a full-body game. Cardboard props — a sword, shield, and bow — are each fitted with an ESP32 and MPU6050 IMU, so picking one up auto-equips it in the in-game hotbar and swinging, blocking, or drawing and releasing triggers the matching in-game action. A webcam running MediaPipe Pose Landmarker tracks the player's body to drive sprinting (via arm-pumping), jumping, and head-tracked camera look, so instead of WASD and a mouse, you actually swing, block, aim, sprint, and turn your head to play.
Minecraft's combat and movement are inherently physical — swinging, blocking, sprinting, jumping — yet players sit still at a keyboard the whole time. We wanted to close that gap and get real embodiment into the game world.
- Cardboard sword, shield, and bow props, each with auto-equip on pickup detection
- Gesture recognition for sword swings, shield raises, and bow draw-and-release, via IMU thresholding and rising-edge detection
- ESP-NOW wireless mesh so props broadcast state to the game with no wires
- Webcam body tracking (MediaPipe Pose Landmarker) drives sprinting, jumping, and head-tracked camera control
- Custom Fabric mod (Minecraft 1.21.1, Java 21) consuming a unified CSV protocol across hardware, CV, and game layers
- Distinguishing deliberate gestures from incidental motion — solved with threshold tuning and rising-edge detection
- Telling a real prop pickup apart from an accidental table bump
- Getting head tracking smooth and non-nauseating through filtering and calibration
- Juggling serial and TCP input threads safely alongside Minecraft's client tick thread
- Keeping end-to-end latency low enough to stay immersive
- Building cardboard props durable enough to survive repeated swinging on demo day
Took 3rd place overall at Hack the 6ix 2026, out of the full competition field.
Josh Wu (University of Waterloo), Andy Duong, Junnur Mustaqeem, James Li (University of Waterloo)