We consider the adaptive resource allocation problem in downlink Orthogonal Frequency Division Multiple Access (OFDMA) system with strict packet delay constraints in the range of . In this range of delay constraints, resource optimization has to be simultaneously performed over multiple time slots. Thus optimal allocation decisions require future Channel State Information (CSI) and packet arrival rate information. The causal nature of CSI combined with the increase in the number of optimization variables makes it a very challenging problem. We propose a two-step solution by separating scheduling from subcarrier and power allocation. Our proposed causal scheduler ensures delay guarantees by deriving a minimum data rate out of the user queues while minimizing transmit power in every time slot. The output rates are fed to the resource allocation block and the problem is formulated as a convex optimization problem. The subcarrier and power allocation decisions are made in order to satisfy the demanded rates within the peak power constraint. We address the feasibility of the physical layer resource allocation problem and develop efficient algorithms. When the problem is infeasible we devise a strategy which incurs minimum deviation from the proposed rates for maximum number of users. We show by simulations that our proposed scheme can efficiently utilize time variations as well as multiuser diversity in the system.