We present a new analysis of the decay processes $H \to \ell^-\ell^+\gamma$ ($\ell = e, \mu$) which are predominantly mediated by electroweak one-loop diagrams. We derive compact and readily usable analytic expressions for the amplitude, fully reduced to scalar one-loop functions. In addition, we propose a gauge-invariant decomposition of the resonant and non-resonant contributions, which can be used for a reliable experimental extraction of the $H\to Z\gamma$ decay rate. We provide differential distributions, including the dilepton invariant mass spectrum, Dalitz plots, and forward-backward asymmetries. We also provide brief remarks on the related process $H\to \nu\bar{\nu}\gamma$.