Double Inverted Pendulum: SAC Swing-Up + Policy Compression

The double inverted pendulum swing-up task is a classic underactuated control benchmark. Here the goal isn't just to solve it in simulation — it's to get a policy small enough to run at control-loop rate on resource-constrained hardware.

Approach

A Soft Actor-Critic agent is trained in MuJoCo on the full nonlinear dynamics. The reward balances swing-up speed against control effort:

$$J(\pi) = \sum_{t=0}^{T} \mathbb{E}_{(s_t,a_t)\sim\rho_\pi}\Big[r(s_t,a_t) + \alpha\,\mathcal{H}\big(\pi(\cdot\mid s_t)\big)\Big]$$

Once the teacher policy converges, it's distilled into a small feedforward network via knowledge distillation, then quantized to INT8 for deployment.

Deployment target

The compressed policy runs on an STM32H7, reading pendulum angle from an encoder and issuing motor commands at the control loop rate. This is where the sim-to-real gap becomes the real problem: everything from sensor noise to actuation latency has to be accounted for that the simulator never modeled.