We propose a 3-step method for structure and motion computation from two or more images taken by a one or multiple moving rolling shutter cameras. This work is motivated by the realization that existing reconstruction methods using rolling shutter images do not give satisfactory results or even fail in many configurations due to singularities and degenerate configurations. The first contribution consists in decoupling the rotate ego motion from the remaining parameters by adding a constraint on image curves basing on the a priori knowledge that they correspond to world 3D straight lines with unknown directions. Straight lines frequently appear in man-made environments such as urban or indoor scenes. After introducing the parameterization of a curve projected from a 3D straight line observed by a moving camera using three rolling shutter projection models, we show how to linearly extract angular velocity of each camera by using detected curves. Then we develop a linear method to recover the translational velocities and the motion between the cameras using point-matches, after compensating effects of angular velocity on each image. The second contribution consists in a novel point based bundle adjustment for rolling shutter cameras (C-RSBA) which does not consider a static row index during structure and motion optimization contrarily to existing methods. This enables to refine the parameters obtained thanks to the straightness constraint by avoiding degenerate configurations, thus outperforming existing RSBA methods. The approach was evaluated on both synthetic and real data.