We study the weight part of Serre's conjecture for generic $n$-dimensional mod $p$ Galois representations. We first generalize Herzig's conjecture to the case where the field is ramified at $p$ and prove the weight elimination direction of our conjecture. We then introduce a new class of weights associated to $n$-dimensional local mod $p$ representations which we call \emph{extremal weights}. Using a ``Levi reduction" property of certain potentially crystalline Galois deformation spaces, we prove the modularity of these weights. As a consequence, we deduce the weight part of Serre's conjecture for unit groups of some division algebras in generic situations.