To want something now rather than later is a common attitude that reflects the brain's tendency to value the passage of time. Because the time taken to accomplish an action inevitably delays task achievement and reward acquisition, this idea was ported to neural movement control within the “cost of time” theory. This theory provides a normative framework to account for the underpinnings of movement time formation within the brain and the origin of a self-selected pace in human and animal motion. Then, how does the brain exactly value time in the control of action? To tackle this issue, we used an inverse optimal control approach and developed a general methodology allowing to squarely sample infinitesimal values of the time cost from experimental motion data. The cost of time underlying saccades was found to have a concave growth, thereby confirming previous results on hyperbolic reward discounting, yet without making any prior assumption about this hypothetical nature. For self-paced reaching, however, movement time was primarily valued according to a striking sigmoidal shape; its rate of change consistently presented a steep rise before a maximum was reached and a slower decay was observed. Theoretical properties of uniqueness and robustness of the inferred time cost were established for the class of problems under investigation, thus reinforcing the significance of the present findings. These results may offer a unique opportunity to uncover how the brain values the passage of time in healthy and pathological motor control and shed new light on the processes underlying action invigoration.