We explore the QCD phase diagram at finite density with four-flavor staggered fermions using the complex Langevin method, which is a promising approach to overcome the sign problem. In our previous work on an $8^3\times16$ lattice at $\beta=5.7$ with the quark mass $m= 0.01$, we have found that the baryon number density has a clear plateau as a　function of the chemical potential. In this study, we use a $16^3\times32$ lattice to reduce finite volume effects and find that the plateau structure survives. Moreover, the number of quarks in the plateau region turns out to be 24, which is exactly the same as the one obtained previously on the $8^3\times16$ lattice. We provide a simple interpretation of this number, which suggests that the Fermi sphere is starting to form.