In this paper, we develop an on-the-fly and incremental technique for fault diagnosis of discrete event systems modeled by labeled Petri nets, in order to tackle the combinatorial explosion problem. K-diagnosability, diagnosability, Kmin (the minimum K ensuring diagnosability) and on-line diagnosis are solved on the basis of the on-the-fly and incremental building of two structures, called respectively fault marking graph and fault marking set graph, in parallel. We build on existing results, namely those establishing necessary and sufficient conditions for diagnosability, but we bring mechanisms to make the checking of such conditions potentially more efficient. We show that, in general, analyzing or even building the whole reachability graph is unnecessary to analyze diagnosability and build an on-line diagnoser. Our technique was implemented in a prototype tool called OF-PENDA, and a railway level crossing benchmark is used to make a comparative discussion pertaining to efficiency in terms of time and memory relative to some existing approaches.