Specular surfaces, by focusing the light that is being reflected or refracted, cause bright spots in the scene, called caustics. These caustics are challenging to compute for global illumination algorithms. Manifold-based methods (Manifold Exploration, Manifold Next-Event Estimation, Specular Next Event Estimation) compute these caustics as the zeros of an objective function, using the Newton-Raphson method. They are efficient, but require computing the derivatives of the objective function, which in turn requires local surface derivatives around the reflection point, which can be challenging to implement. In this paper, we leverage the Nelder-Mead method to compute caustics using Manifold Next-Event Estimation without having to compute local derivatives. Our method only requires local evaluations of the objective function, making it an easy addition to any path-tracing algorithm.