Constraints on the Higgs boson self-coupling are set by combining the single Higgs boson analyses targeting the $\gamma\gamma$, $ZZ^{*}$, $WW^{*}$, $\tau^{+}\tau^{-}$ and $b\bar{b}$ decay channels and the double Higgs boson analyses in the $b\bar{b}b\bar{b}$, $b\bar{b}\tau^{+}\tau^{-}$ and $b\bar{b}\gamma\gamma$ decay channels, using data collected at $\sqrt{s}=13$ TeV with the ATLAS detector at the LHC. The data used in these analyses correspond to an integrated luminosity of up to 79.8 fb$^{-1}$ for single Higgs boson analyses and up to 36.1 fb$^{-1}$ for the double Higgs boson analyses. With the assumption that new physics affects only the Higgs boson self-coupling ($\lambda_{HHH}$), values outside the interval $-2.3<\lambda_{HHH}/\lambda_{HHH}^{SM}<10.3$ are excluded at 95% confidence level. Results with less stringent assumptions are also provided, introducing additional coupling modifiers for the Higgs boson interactions with the other Standard Model particles.