Exotic phases of matter emerge from the interplay between strong electron interactions and non-trivial topology. Owing to their lack of dispersion at the single-particle level, systems harboring flat bands are excellent testbeds for strongly interacting physics, with twisted bilayer graphene serving as a prime example. On the other hand, existing theoretical models for obtaining flat bands in crystalline materials, such as the line-graph formalism, are often too restrictive for real-life material realizations. Here we present a generic technique for constructing perfectly flat bands from bipartite crystalline lattices. Our prescription encapsulates and generalizes the various flat band models in the literature, being applicable to systems with any orbital content, with or without spin-orbit coupling. Using Topological Quantum Chemistry, we build a complete topological classification in terms of symmetry eigenvalues of all the gapped and gapless flat bands, for all 1651 Magnetic Space Groups. In addition, we derive criteria for the existence of symmetry-protected band touching points between the flat and dispersive bands, and we identify the gapped flat bands as prime candidates for fragile topological phases. Finally, we show that the set of all (gapped and gapless) perfectly flat bands is finitely generated and construct the corresponding bases for all 1651 Shubnikov Space Groups.