An effective time-dependent Hamiltonian can be implemented by making a quantum system fly through an inhomogeneous potential, realizing, for example, a quantum gate on its internal degrees of freedom. However, flying systems have a spatial spread that will generically entangle the internal and spatial degrees of freedom, leading to decoherence in the internal state dynamics, even in the absence of any external reservoir. We provide formulas valid at all times for the dynamics, fidelity, and change of entropy for small spatial spreads, quantified by $\Delta x$. This decoherence is non-Markovian and its effect can be significant for ballistic qubits (scaling as $\Delta x^2$) but not for qubits carried by a moving potential well (scaling as $\Delta x^6$). We also discuss a method to completely counteract this decoherence for a ballistic qubit later measured.