Based on the need to reduce the carbon footprint of binders used for plant-based concretes, the development of lime–pozzolan binders is on the rise. The carbon emissions due to the binder production have to be reduced in such a way to ensure a high level of carbon storage while improving the thermal resistance of building envelopes. This paper intends to assess the technical feasibility of making plant-based concretes by mixing agricultural by-products (rapeseed straw, hemp and flax shives) with a binder composed of micronized marine sediment (non-calcined) and air lime. The hardening kinetics of the lime–sediment binder with sodium sulfate added was examined as a first step. The effect of various drying conditions on the strength development of lime–sediment mortars was then investigated to help understand the strength performances of plant-based concretes following 7 days of heat curing and 21 days at 65%RH. The results indicate that the newly formed solid products in the binder collected in plant-based concretes were mainly calcium carbonate and CO3-AFm. In spite of the low hydration degree of the binder, it was possible to reach an ultimate strength of 0.5─0.6 MPa at 7 % strain for a stabilized bulk density of 560 kg m−3. The rapeseed straw concrete was characterized by a strain hardening behavior due to the presence of tangled fiber-like particles and showed the lowest thermal conductivity (0.077 W m−1 K−1 in the dry state).