Measurements of charged-particle production in heavy-ion collisions and their comparison to $pp$ data provide insight into the properties of the quark-gluon plasma. In 2015, the ATLAS detector at the LHC recorded 0.49nb$^{-1}$ of Pb+Pb collisions and 25pb$^{-1}$ of $pp$ collisions at a center-of-mass energy of $\sqrt{s_{\mathrm{NN}}}=\sqrt{s}=5.02$TeV. In addition, around 3$\mu$b$^{-1}$ of Xe+Xe collisions at $\sqrt{s_{\mathrm{NN}}}=5.44$TeV were recorded in 2017. These samples provide an opportunity to study the dependence of the parton energy loss over the system size. The large acceptance of the ATLAS detector allows measurements of charged-particle spectra in a wide range of both pseudorapidity and transverse momentum, and differential in collision centrality. Charged-particle spectra measured in Pb+Pb and Xe+Xe collisions are compared to the analogous spectra measured in $pp$ collisions, and the resulting nuclear modification factors are scrutinized. In particular, the nuclear modification factors are found to scale approximately with the number of participating nucleons, which may be a key to predicting the behavior of even smaller collision systems.