The Inert Doublet Model (IDM) is one of the simplest extensions of the Standard Model (SM), providing a dark matter candidate. It is a two Higgs doublet model with a discrete $Z_2$ symmetry, that prevents the scalars of the second doublet (inert scalars) from coupling to the SM fermions and makes the lightest of them stable. We study a large group of IDM scenarios, which are consistent with current constraints on direct detection, including the most recent bounds from the XENON1T experiment and relic density of dark matter, as well as with all collider and low-energy limits. We propose a set of benchmark points with different kinematic features, that promise detectable signals at future $e^+e^-$ colliders. Two inert scalar pair-production processes are considered, $e^+e^- \to H^+H^-$ and $e^+e^- \to AH$, followed by decays of $H^\pm$ and $A$ into final states which include the lightest and stable neutral scalar dark matter candidate $H$. Significance of the expected observations is studied for different benchmark models and different running scenarios, for centre-of-mass energies up to 3 TeV. Numerical results are presented for the signal signatures with two muons or an electron and a muon in the final state. For high mass scenarios, when the significance is too low for the leptonic signatures, the semi-leptonic signature can be used as the discovery channel.