One of the most promising quantities for the search of signatures of physics beyond the Standard Model is the anomalous magnetic moment $g-2$ of the muon, where a comparison of the experimental result with the Standard Model estimate yields a deviation of about $3.5~\sigma$. On the theory side, the largest uncertainty arises from the hadronic sector, namely the hadronic vacuum polarisation and the hadronic light-by-light scattering. I review recent progress in calculating the hadronic contributions to the muon $g-2$ from the lattice and discuss the prospects and challenges to match the precision of the upcoming experiments.