A model of mushy zone instability is developed for characterization and prediction of channel segregation in castings. The model highlights the connection of the mushy zone instability with the mush permeability. A new criterion for amplification of the mush instability is derived, which depends on the interdendritic velocity, the isotherm velocity, the temperature gradient, and explicitly on the mush permeability. The capability of the instability criterion in the characterization of channel segregates is illustrated by comparing the estimated possible channel locations with that of numerically simulated channel segregations in a benchmark test case of solidification of Sn-Pb alloy. An Opensource CFD software OpenFOAM is used to simulate the solidification of Sn-Pb alloy in a side-cooled rectangular cavity. The new instability criterion very well characterizes the channel segregates shown in the numerical simulations. The near-liquidus part of the mushy region is more prone to the initiation of instability, which can initiate the formation of channel segregates in castings. Permeability and its derivative with respect to the liquid fraction in the outer part of mushy zone (liquid fraction more than 0.9) plays a key role in the amplification of this instability that can be responsible for the development and the formation of channel segregates. The locations of channel segregates estimated by the new instability model are also compared with those with a remelting criterion, and it has been noticed that the remelting criterion severely under characterizes these defects (e.g., their number and length). We noticed that local remelting might not be necessary for the initiation of a channel; some destabilization of the mushy zone due to local instability could be sufficient.