The Late Acceptance Hill Climbing heuristic is a Hill Climbing algorithm that uses a record of the history of objective values of previously encountered solutions in order to decide whether to accept a new solution. Literature has shown that Late Acceptance Hill Climbing is generally better at not getting stuck in local optima because of the history. In this paper, we propose and investigate a simple, yet effective, modification to Late Acceptance Hill Climbing, where we change how values in the history are replaced. In our tests, referring to the Traveling Salesman Problem, we analyze the behavior of the proposed approach for different history sizes. We also show that the simple change in the algorithm allows the heuristic to find better solutions than the original one on most of the instances tested.