The viscoplastic behavior of coarse and bimodal fine/ultrafine grained (F/UFG) Al5083 alloy was investigated between 20 °C and 200 °C through tensile tests at various strain rates, and stress relaxation tests to deduce the strain rate sensitivity (SRS). The plastic strain fields were measured by correlation of SEM images. In the F/UFG material at high temperature, very high strains were measured in shear bands which sometimes crossed the whole gage width and exhibited intensive grain boundary sliding (GBS). Both the SRS and ductility rose with the temperature, and as the strain rate decreased, mainly due to a rising contribution of GBS, which accommodated a much larger fraction of the global strain in the F/UFG material. The boundary between the temperature–strain rate domains where grain refinement led either to strengthening or to softening was determined. Finite element simulations of tension and relaxation tests with viscoplastic grains and sliding grain boundaries captured the macro-scale behavior of the F/UFG material. It also provided some insight into the mechanisms of correlated and cooperative GBS and grain rotation along percolation paths (both inter and intragranular), probably, responsible for macro shear banding.