This paper concerns portfolio selection with multiple assets under rough covariance matrix. We investigate the continuous-time Markowitz mean-variance problem for a multivariate class of affine and quadratic Volterra models. In this incomplete non-Markovian and non-semimartingale market framework with unbounded random coefficients, the optimal portfolio strategy is expressed by means of a Riccati backward stochastic differential equation (BSDE). In the case of affine Volterra models, we derive explicit solutions to this BSDE in terms of multi-dimensional Riccati-Volterra equations. This framework includes multivariate rough Heston models and extends the results of \cite{han2019mean}. In the quadratic case, we obtain new analytic formulae for the the Riccati BSDE and we establish their link with infinite dimensional Riccati equations. This covers rough Stein-Stein and Wishart type covariance models. Numerical results on a two dimensional rough Stein-Stein model illustrate the impact of rough volatilities and stochastic correlations on the optimal Markowitz strategy. In particular for positively correlated assets, we find that the optimal strategy in our model is a `buy rough sell smooth' one.