We study the problem of data-driven resource allocation in Multi-Tenant Edge Computing: a Network Operator (NO) owns resources at the Edge and dynamically allocates them to third party application Service Providers (SPs). The objective of the NO is to reduce its operational cost. Since SPs' traffic is encrypted, NO's allocation strategy is based solely on the amount of traffic measured. In this exploratory work, we solve this problem via Reinforcement Learning (RL). RL has mainly been intended to be trained in simulation, before applying it in real scenarios. We instead employ RL online, training it directly while optimizing resource allocation. An important factor, which we call perturbation cost, emerges in this case: in order to learn how to optimize a system, we need to perturb it and measure its reaction. While this perturbation cost has no physical meaning when training RL in simulation, it cannot be ignored when it is paid by the real system. We explore in this work the trade-off between perturbing a lot the system to learn faster to optimize the allocation, or learning slower to reduce the perturbation cost. In our case study, the resource we allocate is storage. We show results from simulation and make the entire code available as open-source.