In previous work, it has been shown that the recently proposed LLR method is very efficient at overcoming strong metastabilities that arise near first-order phase transition points. Here we present a systematic study of the performance of the algorithm near (pseudo-)critical points for $q$-state Potts models with $q$ as large as 20, in two and three dimensions. In particular, we shall focus our study on the ergodicity of the replica exchange step and the underlying physical mechanism. When compared with both analytical and numerical results present in the literature, our determinations of thermodynamic observables (including the order-disorder interface tension at criticality) show an impressive degree of relative accuracy (up to $2.5 \times 10^{-6}$), which confirms the reliability and the efficiency of the proposed approach.