The challenge of maximizing the diversity of a collection of points arises in a variety of settings, including the setting of search methods for hard optimization problems. One version of this problem, called the Maximum Diversity Problem (MDP), produces a quadratic binary optimization problem subject to a cardinality constraint, and has been the subject of numerous studies. This study is focused on the Maximum Minimum Diversity Problem (MMDP) but we also introduce a new formulation using MDP as a secondary objective. We propose a fast local search based on separate add and drop operations and on simple tabu mechanisms. Compared to previous local search approaches, the complexity of searching for the best move at each iteration is reduced from quadratic to linear; only certain streamlining calculations might (rarely) require quadratic time per iteration. Furthermore, the strong tabu rules of the drop strategy ensure a powerful diversification capacity. Despite its simplicity, the approach proves superior to most of the more advanced methods from the literature, yielding optimally-proved solutions for many problems in a matter of seconds and even attaining a new lower bound.