We present a class of new physics scenarios wherein the Standard Model (SM) is extended with a new $SU(2)_D$ dark gauge sector. These frameworks typically include a $\mathbb{Z}_2$ parity under which states transforming non-trivially under $SU(2)_D$ have different parity assignments. After spontaneous breaking of the $SU(2)_D$ symmetry via a new scalar doublet, the ensuing massive vector bosons with non-zero D-isospin are Dark Matter (DM) candidates. Non-renormalisable kinetic mixing in the gauge sector or a Higgs portal are not required to connect the dark sector to the SM. We introduce instead a Vector-Like (VL) fermion doublet of $SU(2)_D$, the members of which are singlets of the SM Electro-Weak (EW) gauge group, which mediates the interactions between the dark sector and the SM itself. This class of Fermion Portal Vector DM (FPVDM) allows multiple realisations, depending on the properties of the the VL partner and the scalar potential. As a practical example, here, we discuss in detail a realisation involving a VL top partner assuming no mixing between the two scalars of the theory. We then provide bounds from collider and astroparticle observables. The origin of the $\mathbb{Z}_2$ symmetry in this class of models can be justified in the form of a dark EW sector or through an underlying composite structure. The new class of FPVDM models we suggest here has a large number of phenomenological applications depending on the mediator properties and can be used for collider and non-collider studies alike, some of which we outline here for future studies.