This work summarizes the lattice QCD calculations of the axial $\gamma W$-box diagrams relevant for the semi-leptonic decays of pions and kaons. Using the method combining lattice data at small $Q^2$ and perturbative calculation at large $Q^2$, the $\gamma W$-box contributions to the pion decays at the physical point and the kaon decays at the flavor SU(3) limit are calculated with the total uncertainty controlled at the level of $\sim 1\%$. These results could be used to determine the low energy constants for chiral perturbation theory with its uncertainty reduced significantly. This work could be generalized to the cases of the free and bound neutron decays, which plays an important role in the determination of the CKM matrix element $|V_{ud}|$, which is essential to the unitarity test of the CKM matrix.