Multilayer films comprising polystyrene (PS)/polymethyl methalcrylate (PMMA) and PS/polycaprolatone (PCL) alternating nanolayers with varied layer thickness were fabricated by multilayer coextrusion. The nanolayers breakup phenomena of PMMA and PCL were characterized using atomic force microscopy (AFM), oxygen permeability, light transmission, wide-angle X-ray scattering (WAXS), and differential scanning calorimetry (DSC). The continuous layers started to break up into nanosheets and nanodroplets during the coextrusion process when the nominal layer thickness decreased to between 30 nm and 40 nm. Further decrease of the nominal layer thickness of PMMA and PCL resulted in less nanosheets and more nanodroplets. Oxygen permeability was effective for characterizing the onset thickness of layer breakup. The oxygen permeability for the PS/PCL system was modeled and demonstrated good correlation with estimated composition of continuous layers, nanosheets, and nanodroplets.