We investigate the in-medium modifications of heavy quarkonia in the vector channel and the heavy quark diffusion coefficient by comparing Euclidean correlators from the lattice to a perturbative spectral function. On the lattice side, we work with continuum extrapolated data from four different large and fine lattices with Clover-improved Wilson fermions in the quenched approximation at five temperatures (0.75, 1.1, 1.3, 1.5 and 2.25$T_c$). On the perturbative side, we use a combination of pNRQCD and vacuum asymptotics to describe the spectral function. After accounting for systematic errors, we obtain a spectral function that is suited to describe the bound state region. This spectral function describes charmonium well without a resonance peak at any of our analyzed temperatures above $T_c$, while we observe a thermally broadened resonance peak for bottomonium that is only melted at our largest temperature, $2.25T_c$. For the transport contribution we assume a Breit-Wigner shaped peak and find that the drag coefficient of charm quarks is larger than that of bottom quarks.