In this article we design a novel quasi-regression Monte Carlo algorithm in order to approximate the solution of discrete time backward stochastic differential equations (BSDEs), and we analyze the convergence of the proposed method. The algorithm also approximates the solution to the related semi-linear parabolic partial differential equation (PDE) obtained through the well known Feynman-Kac representation. For the sake of enriching the algorithm with high order convergence a weighted approximation of the solution is computed and appropriate conditions on the parameters of the method are inferred. With the challenge of tackling problems in high dimensions we propose suitable projections of the solution and efficient parallelizations of the algorithm taking advantage of powerful many core processors such as graphics processing units (GPUs).