Notes on Soft Actor-Critic for Robotic Control

Working through Soft Actor-Critic this week, mostly to make sure I understand exactly what the entropy term is doing to the swing-up policy before I start distilling it.

The maximum-entropy objective is:

$$J(\pi) = \sum_{t=0}^{T} \; \mathbb{E}_{(s_t,a_t)\sim\rho_\pi} \left[ r(s_t,a_t) + \alpha\,\mathcal{H}\bigl(\pi(\cdot\mid s_t)\bigr) \right]$$

The entropy bonus keeps the policy from collapsing too early into a single deterministic strategy, which matters a lot for swing-up: there are multiple qualitatively different ways to pump energy into the pendulum, and early collapse tends to lock in a suboptimal one.