This paper presents a new Bit Flipping (BF) decoder, called the Probabilistic Parallel Bit Flipping (PPBF) for Low-Density Parity-Check (LDPC) codes on the Binary Symmetric Channel. In the PPBF, the flipping operation is performed in a probabilistic manner which is shown to improve significantly the error correction performance. The advantage of the PPBF also comes from the fact that no global computation is required during the decoding process and that all the computations can be executed in the local computing units and in-parallel. The PPBF provides a considerable improvement of the operating frequency and complexity, compared to other known BF decoders, while obtaining a significant gain in error correction. An improved version of the PPBF, called non-syndrome PPBF (NS-PPBF) is also introduced, in which the global syndrome check is moved out of the critical path and a new terminating mechanism is proposed. In order to show the superiority of the new decoders in terms of hardware efficiency and decoding throughput, the corresponding hardware architectures are presented in the second part of the paper. The ASIC synthesis results confirm that the operating frequency of the proposed decoders is significantly improved, compared to that of the BF decoders in the literature while requiring lower complexity to be efficiently implemented.