Auger data show evidence for a correlation between ultrahigh-energy cosmic rays (UHECRs) and nearby starburst galaxies. This intriguing correlation is consistent with data collected by the Telescope Array, which have revealed a much more pronounced directional "hot spot" in arrival directions not far from the starburst galaxy M82. In this work, we assume starbursts are sources of UHECRs and investigate the prospects to use the observed distribution of UHECR arrival directions to constrain Galactic magnetic field models. We show that if the Telescope Array hot spot indeed originates from M82, UHECR data would place a strong constraint on the coherent and turbulent components of the Galactic magnetic field.