Small-scale irrigation systems (SIS) and water user associations (WUA) are key instruments to enhance food security, rural income and livelihoods in agricultural development. Current monitoring and evaluation (M&E) methods for SIS/WUA mainly use limited quantitative data that are difficult to obtain and often come up short in assessing true sustainability. Our research proposes qualitative methods to evaluate SIS/WUA. First a set of qualitative sustainability indicators and associated variables for SISs/ WUAs are proposed. Six composite indicators were produced: (1) water governance, (2) technical competence, (3) functional competence, (4) participation, (5) viability and (6) impact/benefit. Two qualitative evaluation strategies were then applied: (1) 90 in-depth WUA member interviews and participatory observation techniques (extended field visits, case studies, and trained observer ratings); and (2) 75 rapid surveys of members within the same WUAs. Strategies were applied to 10 WUAs in eastern Madagascar affiliated with an integrated poverty reduction programme sponsored by IFAD and targeting subsistence rice farmers. Variable values were weighted and summed for interviews and surveys to arrive at an overall sustainability value (sval) index for irrigators and WUAs. Participatory observation techniques prove the resulting sval index as a legitimate monitoring tool to assess SIS/WUA. Secondly, this work measures the concurrence between collection methods (interviews and surveys) to determine if they are interchangeable. Typologies, data triangulation and statistical analysis were used for this task. Typologies were created and analysed using a (1) key variable qualitative method, (2) best and least performing WUA and (3) an agglomerative hierarchical clustering (AHC) method. Typologies confirm s-val as a sound sustainability index. Differences were found between global s-val means for surveys and interviews though both samples fit a normal distribution pattern. Discrepancies are attributed to incoherency of certain variables between members of the evaluation team. Statistical t-test results of interview and survey data samples indicate statistical similarity between the two collection methods when incongruities are accounted for. Interpretation and judgment of the typologies and statistical analysis results were triangulated with participatory observations, trained observer ratings and case studies. This work defends the legitimacy of an integrated qualitative evaluation based on sound social theory, data collection and analysis tools.