The Calorimetric Electron Telescope (CALET) is a space mission installed on the Exposed Facility of the Japanese Experiment Module (JEM-EF) of the International Space Station (ISS) in August 2015 and collecting data since October 2015. In addition to high precision measurements of the electron spectrum up to TeV scale, CALET will also investigate the mechanism of cosmic-ray (CR) acceleration and propagation in the Galaxy, by performing direct measurements of the energy spectra and elemental composition of CR nuclei from H to Fe, and the abundance of trans-iron elements up to about Z=40. The instrument consists of two layers of segmented plastic scintillators to identify the particle charge, a thin (3 radiation lengths) tungsten-scintillating fiber calorimeter providing accurate particle tracking, and a thick (27 radiation lengths) calorimeter made of lead-tungstate crystal logs. In this paper we will describe an original reconstruction method of the primary particle direction based on a combinatorial Kalman filter algorithm. This method exploits the fine granularity and imaging capability of the IMC and provides robust track finding and fitting, allowing to identify the incident CR track in a large amount of shower particle tracks backscattered from the calorimeter. The track fitting algorithm has been extensively validated and tuned with simulated data. Its performance (angular resolution, impact point resolution, tracking efficiency) for electrons and nuclei will be discussed and comparisons between flight data and simulations will be shown.