Voice and speech production greatly relies on the ability of the vocal tract to articulate a wide variety of sounds. This ability is related to the accurate control of the geometry (and its variations in space and time) in order to generate vowels (including diphthongs) and consonants. Some well-known vibro-acoustic models of the vocal tract rely on a discretized geometry, such as concatenated cylinders, the radius of which varies in time to account for the articulation (see e.g. Maeda, Speech Comm. 1:199-229, 1982). We here propose a lumped parameter model of waves in the vocal tract considering the motion of the boundaries. A particular attention is paid to passivity and the well-posedness of the power balance in the context of time-varying geometrical parameters. To this end, the proposed model is recast in the theoretical framework of port-Hamiltonian systems that ensure the power balance. The modularity of this framework is also well-suited to interconnect this model to that of deformable walls (in a power-balanced way). We show the capacities of the model in two time-domain numerical experiments: first for a static configuration (time-invariant geometry), then a dynamic one (time-varying geometries) of a two-cylinder vocal tract.