Let p be prime number, K be a p-adically closed field, X ⊆ K^m a semi-algebraic set defined over K and L(X) the lattice of semi-algebraic subsets of X which are closed in X. We prove that the complete theory of L(X) eliminates the quantifiers in a certain language LASC, the LASC-structure on L(X) being an extension by definition of the lattice structure. Moreover it is decidable, contrary to what happens over a real closed field. We classify these LASC-structures up to elementary equivalence, and get in particular that the complete theory of L(K^m) only depends on m, not on K nor even on p. As an application we obtain a classification of semi-algebraic sets over countable p-adically closed fields up to so-called " pre-algebraic " homeomorphisms.