The complex Langevin (CL) method shows significant potential in addressing the numerical sign problem. Nonetheless, it often produces incorrect results when used without any stabilization techniques. Leveraging insights from previous research that links Lefschetz thimbles and CL, we explore a strategy to regularize the CL method to address this issue of incorrect convergence. Specifically, we implement weight regularizations inspired by the associated Lefschetz thimble structure and correct the bias to retrieve the correct results of the original theory. We demonstrate the effectiveness of this approach by solving the SU(N) Polyakov chain model and various scalar models, including the cosine model and the one-link model, across a broad range of couplings where the CL method previously failed. We also discuss the potential application of these insights to gauge theories in practical scenarios.