We present the first study of coherent exclusive quarkonium ($J/\psi$, $\Upsilon$) photoproduction in ultraperipheral nucleus-nucleus collisions (UPCs) at the LHC in the framework of collinear factorization and next-to-leading order (NLO) perturbative QCD (pQCD). We make NLO predictions for the $J/\psi$ and $\Upsilon$ rapidity distributions for lead (Pb) and oxygen (O) beams, and quantify their dependence on the factorization/renormalization scale, nuclear parton distribution functions (PDFs) and their uncertainties, and on differences between nuclear PDFs and generalized parton distribution functions (GPDs). We show that within the PDF-originating uncertainties our approach provides a good description of the available $J/\psi$ photoproduction data in Pb+Pb UPCs at the LHC but that the scale uncertainty is significant. We demonstrate that at NLO pQCD the quark contributions are important in the $J/\psi$ case but that gluons clearly dominate the $\Upsilon$ cross sections. We also study how the scale dependence could be tamed by considering O+O/Pb+Pb ratios of the exclusive $J/\psi$ UPC cross sections, and how HERA and p+p/Pb LHC data can help in obtaining better-controlled NLO predictions in the $\Upsilon$ case.