Cosmic-ray positrons are a powerful probe of annihilating dark matter. While the unexpected rise in the positron fraction has been attributed to contributions from either dark matter or pulsars, in this work we instead investigate contributions from sharply peaked leptophilic dark matter models that do not fit the properties of the positron excess. Making use of the extremely precise measurements of the local positron flux by AMS-02, we create a detailed cosmic-ray propagation model, attributing the rising positron flux to primary positrons produced by Galactic pulsars, and constraining a secondary positron component with proton and Helium AMS-02 data. We do not find evidence for leptophilic dark matter and instead derive strong constraints on the annihilation cross section that lie below the thermal annihilation cross-section for dark matter masses below 60 GeV and 380 GeV for annihilation into $\tau^+\tau^-$ and e$^+$e$^-$ in our default model, respectively.