Lyapunov methods are one of the main tools to investigate local and global stability properties of dynamical systems. Even though Lyapunov methods have been studied and applied over many decades to unconstrained systems, extensions to systems with more complicated state constraints have been limited. This paper proposes an extension of classical control Lyapunov functions (CLFs) for differential inclusions by incorporating in particular bounded (nonconvex) state constraints in the form of obstacles in the CLF formulation. We show that the extended CLF formulation, which is called a complete CLF (CCLF) in the following, implies obstacle avoidance and weak stability (or asymptotic controllability) of the equilibrium of the dynamical system. Additionally, the necessity of nonsmooth CCLFs is highlighted. In the last part we construct CCLFs for linear systems, highlighting the difficulties of constructing such functions.