Dilepton spectra are a classic probe to study ultra-relativistic heavy ion collisions. At RHIC energies, the dimuon continuum is dominated by correlated pairs from semi-leptonic decays of charm and bottom hadrons and the Drell-Yan process. Heavy flavor angular correlations can be measured through dimuons, which can help constrain the relative contributions from different heavy flavor production mechanisms in $p$+$p$ collisions, and may provide further insight on cold nuclear matter effects in $p$+Au collisions. Measurements of the Drell-Yan cross-section in $p$+Au collisions can provide constraints to nPDFs, and further our understanding in initial state effects. In this talk, we report recent measurements of $\mu\mu$ pairs from charm, bottom, and Drell-Yan in $p$+$p$ and $p$+Au collisions at $\sqrt{s_{NN}}=200$ GeV.