Traditional recommender systems employ a history of item preferences by a set of users for recommending items of interest to a given user. Matrix factorization based models have achieved the state-of-the-art success in the personal recommendation tasks by aiming at predicting ratings through learning latent factors of users and items via the rating matrix decomposition. Matrix factorization based models treat ratings as integers which are supposed to be generated from Gaussian distribution assumption all the time, but long-tail distribution widely exists in real rating data. Moreover, these approaches always ignore the semantic meanings of ratings which can expose the underlying attitudes of users with respect to many aspects of items. In this paper we propose a novel non-Gaussian embedding based model for recommender systems (NGEM) that represents each rating as a translation between users and items in the rating-dependent subspace. Then various types of initializations are incorporated into the proposed model to enhance the learned embeddings of users, items and ratings for better optimization. The effectiveness of our proposed model is demonstrated on rating-prediction task by extensive experiments on the real-world data sets. The results also show that different types of initializations help our model achieve an improvement on accuracy in comparison with some state-of-the-art matrix factorization based models.