Self-mixing (SM) or optical feedback interferometry has been extensively used for high precision displacement and vibration sensing. However, presence of speckle can significantly degrade the SM interferometric signal and cause changes in signal amplitude as well as in the operating optical feedback regime, resulting in reduction in measurement precision. Previously, different advanced digital signal processing techniques have been proposed to undo the effects caused by speckle. However, their complex and computationally heavy nature inhibits their use for real-time, high bandwidth sensing applications. In this regard, an all analog signal processing algorithm has been presented in this paper which allows real- time processing of speckle affected SM signal while using standard analog circuits. Various simulations indicated that it is able to correctly process speckle affected SM signals having amplitude variation of at least one order and optical feedback parameter C reduction till 0.5. This proposed algorithm has been tested on experimentally acquired speckle affected SM signals and found capable of dealing with variations in optical feedback regime and amplitude modulation of SM signals, in accordance with simulation results. The developed hardware prototype circuit measures maximum displacement amplitude of 0.4 mm at maximum target velocity of 8 mm/s for an SM sensor with laser wavelength of 785 nm as long as C > 0.5. The proposed all analog processing could be a significant step towards a robust, low-cost, integrated, real-time SM displacement sensor.