Modular functors are traditionally defined as systems of projective representations of mapping class groups of surfaces that are compatible with gluing. They can formally be described as modular algebras over central extensions of the modular surface operad, with the values of the modular algebra lying in a suitable symmetric monoidal $(2,1)$-category $\mathcal{S}$ of linear categories. In this paper, we characterize modular functors in $\mathcal{S}$ as self-dual balanced braided algebras $\mathcal{A}$ in $\mathcal{S}$ -- a categorification of the notion of a commutative Frobenius algebra -- for which a condition formulated in terms of factorization homology with coefficients in $\mathcal{A}$ is satisfied. Our construction is topological, and can be thought of as a far reaching generalization of the construction of modular functors from skein theory. We prove that cofactorizability of $\mathcal{A}$ is sufficient for this condition to be fulfilled. This recovers in particular Lyubashenko's construction of a modular functor from a (not necessarily semisimple) modular category, and show that it is determined by its genus zero part. Additionally, we exhibit modular functors that do not come from modular categories and outline applications to the theory of vertex operator algebras.