Walls in discrete element method simulations of granular flows are sometimes modeled as a closely packed monolayer of fixed particles, resulting in a rough wall rather than a geometrically smooth wall. An implicit assumption is that the resulting rough wall differs from a smooth wall only locally at the particle scale. Here we test this assumption by considering the impact of the wall roughness at the periphery of the flowing layer on the flow of monodisperse particles in a rotating spherical tumbler. We find that varying the wall roughness significantly alters average particle trajectories even far from the wall. Rough walls induce greater poleward axial drift of particles near the flowing layer surface but decrease the curvature of the trajectories. Increasing the volume fill level in the tumbler has little effect on the axial drift for rough walls but increases the drift while reducing curvature of the particle trajectories for smooth walls. The mechanism for these effects is related to the degree of local slip at the bounding wall, which alters the flowing layer thickness near the walls, affecting the particle trajectories even far from the walls near the equator of the tumbler. Thus, the proper choice of wall conditions is important in the accurate simulation of granular flows, even far from the bounding wall.