Recently, Hammond and Sheffield introduced a model of correlated random walks that scale to fractional Brownian motions with long-range dependence. In this paper, we consider a natural generalization of this model to dimension $d\geq 2$. We define a $\mathbb Z^d$-indexed random field with dependence relations governed by an underlying random graph with vertices $\mathbb Z^d$, and we study the scaling limits of the partial sums of the random field over rectangular sets. An interesting phenomenon appears: depending on how fast the rectangular sets increase along different directions, different random fields arise in the limit. In particular, there is a critical regime where the limit random field is operator-scaling and inherits the full dependence structure of the discrete model, whereas in other regimes the limit random fields have at least one direction that has either invariant or independent increments, no longer reflecting the dependence structure in the discrete model. The limit random fields form a general class of operator-scaling Gaussian random fields. Their increments and path properties are investigated.