We prove that the free energy of the log-gamma polymer between lattice points $(1,1)$ and $(M,N)$ converges to the GUE Tracy-Widom distribution in the $M^{1/3}$ scaling, provided that $N/M$ remains bounded away from zero and infinity. We prove this result for the model with inverse gamma weights of any shape parameter $\theta>0$ and furthermore establish a moderate deviation estimate for the upper tail of the free energy in this case. Finally, we consider a non i.i.d. setting where the weights on finitely many rows and columns have different parameters, and we show that when these parameters are critically scaled the limiting free energy fluctuations are governed by a generalization of the Baik-Ben Arous-P\'ech\'e distribution from spiked random matrices with two sets of parameters.