In this paper, we show that similar open-source codes for general relativistic magnetohydrodynamic (GRMHD) produce different results for key features of binary neutron star mergers. First, we present a new open-source version of the publicly available IllinoisGRMHD code that provides support for realistic, finite temperature equations of state. After stringent tests of our upgraded code, we perform a code comparison between GRHydro, IllinoisGRMHD, Spritz, and WhiskyTHC, which implement the same physics, but slightly different computational methods. The benefit of the comparison is that all codes are embedded in the EinsteinToolkit suite, hence their only difference is algorithmic. We find similar convergence properties, fluid dynamics, and gravitational waves, but different merger times, remnant lifetimes, and gravitational wave phases. Such differences must be resolved before the post-merger dynamics modeled with such simulations can be reliably used to infer the properties of nuclear matter especially in the era of precision gravitational wave astronomy.