The tracking of multiple particles in lengthy image sequences is challenged by the stochastic nature of displacements, particles detection errors, and the combinatorial explosion of all possible trajectories. As such, extensive work has focused on the modeling of noisy trajectories to try and predict the most likely trajectory-to-measurements associations. Recently, transformers have been shown to significantly accelerate the evaluation of probabilistic models for the system dynamics and detection clutter generated from false positives. However, little work has focused on clutter-free scenarios with multiple particles moving erratically, where the challenge resides not in the model complexity, but in the combinatorial burden of considering all possible trajectory-to-measurements associations. This is a common occurrence in fluorescence microscopy at low framerate. This paper offers a proof-of-concept study of the benefit of the transformer architecture in such scenarios through the simulation of two-particle-systems undergoing Brownian motion. Specifically, we designed a transformer for this estimation task and compared it with the Multiple Hypothesis Tracker (MHT), the optimal estimator when all trajectory-to-measurement associations can be computed. We first show increased robustness of the transformer against erratic displacements over long sequences, with significantly lower computational complexity than MHT. Then, we show that while the transformer requires very little training to significantly outperform MHT on long sequences, it cannot match the theoretically optimal performances of MHT on short sequences even with extensive training. Hence, our work motivates the broader application of transformers in high-SNR sequences and opens the way to the development of frugal methods thanks to the combination of both statistical and neural network frameworks for particle tracking.