Teaching an AI to Solve a Physical Marble Maze

A real-world reinforcement learning system combining 3D printing, embedded control, computer vision, and continuous-action policy learning.

RL-controlled marble maze running in real time

Overview

This project started with a simple question: can reinforcement learning solve a physical marble maze, not just a simulator? I built a two-axis motorized maze and trained an agent to drive a steel ball from start to goal using camera feedback.

After several hardware and software iterations, the system learned robust behavior and can solve the maze in as few as eight actions in best runs, while recovering from occasional bad moves and sensing noise.

System Setup

Illustration of the RL marble maze system architecture
  • Two 28BYJ-48 stepper motors with ULN2003 drivers control tilt on X and Y axes.
  • An Arduino Uno runs motion control, homing logic, and host serial communication.
  • A top-mounted USB camera tracks the maze and marble states in real time.
  • Limit switches provide repeatable startup calibration and zero reference.

Perception and Environment

A YOLO-based vision pipeline detects maze corners and ball position. Corner detections are used to compute a perspective transform, producing a normalized top-down coordinate system for control and reward evaluation.

The environment follows a Gym-like interaction cycle: reset, step(action), observation, and reward. Each action maps to motor targets, waits for motion to settle, then reads the next ball state from vision.

Reward Design

Potential map used for reward shaping in the maze

Reward shaping uses a precomputed map over the maze geometry and the current implementation uses 1 - potential. As a result, rewards are close to -1 near the maze start and approach 0 near the goal region, with an additional +100 completion bonus when the marble reaches the target.

Learning Framework

The controller is based on DDPG for continuous control. The actor outputs two continuous tilt commands, and the critic evaluates state-action quality. Training supports both online interaction with hardware and offline replay-based updates from logged trajectories.

I also added Optuna-based hyperparameter search to tune learning rates, noise settings, and discounting, which significantly reduced manual trial-and-error.

Training Behavior

Actor and critic loss curves during offline training

Typical offline runs show unstable early learning followed by convergence to smoother critic behavior and a more stable policy. In practice, policy quality is evaluated on real maze success rate and step efficiency, not loss values alone.

Key Challenges and Lessons

  • Real-world latency and settling time dominate control quality more than algorithm complexity.
  • Reliable serial communication and robust state logging were essential for reproducible training.
  • Vision calibration quality directly impacted policy stability and sample efficiency.
  • Data infrastructure (MariaDB-backed replay data) became critical as experiment volume grew.

Next Steps

  • Compare DDPG against SAC and TD3 under the same hardware constraints.
  • Improve actuation smoothness with faster motors and refined low-level control.
  • Add richer live monitoring for training diagnostics and policy evaluation.

Resources