Electric vehicle (EV) grid integration presents significant challenges and opportunities for electricity system operation and planning. Proper assessment of the costs and benefits involved in EV integration hinges on correctly modeling and evaluating EV-user driving and charging patterns. Recent studies have evidenced that EV users do not plug in their vehicle every day (here called non-systematic plug-in behavior), which can alter the impacts of EV charging and the flexibility that EV fleets can provide to the system. This work set out to evaluate the effect of considering non-systematic plug-in behavior in EV grid integration studies. To do so, an open-access agent-based EV simulation model that includes a probabilistic plug-in decision module was developed and calibrated to match the charging behavior observed in the Electric Nation project, a large-scale smart charging trial. Analysis shows that users tend to plug-in their EV between 2 and 3 times per week, with a lower plug-in frequency for large-battery EVs and large heterogeneity in user charging preferences. Results computed using our model show that non-systematic plug-in behavior effects reduce the impact of EV charging, especially for price-responsive charging, as fewer EVs charge simultaneously. On the other hand, non-systematic plug-in can reduce available flexibility, particularly when considering current trends towards larger battery sizes. Counter-intuitively, large-battery fleets can have reduced flexibility compared to small-battery fleets, both in power and stored energy, due to lower plug-in frequency and higher energy requirements per charging session. Improving plug-in ratios of EV users appears as key enabler for flexibility. In comparison, augmenting charging power can increase the flexibility provided by EV fleets but at the expense of larger impacts on distribution grids.