Dark matter (DM) models with a conserved particle-antiparticle number, , and the asymmetry in the cosmological abundance , are known to be challenged by the existence of old neutron stars (NSs), as the sufficient accumulation of DM will lead to the collapse of NSs into black holes. We demonstrate that the applicability of these constraints is much wider and covers models with symmetric populations of DM, , as the process of DM capture regulated by a nucleon-DM scattering can be inherently asymmetric, . The asymmetry is induced by the interference of different types of interactions, provided that their combination is odd under charge conjugation in the DM sector, , and even under combined parity . We provide a complete analysis of DM-nucleon bilinear interactions and find that this asymmetry is very generic. Using canonical NS parameters and local DM halo inputs, we exclude spin-averaged scattering cross sections down to at DM mass for the maximally asymmetric capture rate, and show that the constraints persist down to very small values of the cross-section asymmetry, .