Quality of what? From whose perspective? Thoughts on measuring sanitation quality

Yesterday I was on a panel at the UNC Water and Health conference side-event entitled “What is “Quality” Sanitation? Investigating Service Standards and User Experience in Rural and Urban Settings”. You can watch back the event here (if registered for the conference, which is free). Below follows some of my views on the measurement of quality, covering three aspects: (i) quality of what? (ii) quality from whose perspective? (iii) quality for what purpose?

1. Measuring the quality of what?

As background to “quality of what”, here’s three blogposts I wrote last year:

• This post explains the concept of sanitation-related quality of life (SanQoL)
• This post discusses differences between “quality of life” and “quality of service”
• This post discusses subjectivity / objectivity, and generalises to the service chain beyond containment

In summary, measures of quality of life (QoL) focus on subjective user-perceived outcomes (e.g. how safe the user feels). By contrast, measures of quality of service (QoS) focus on infrastructure attributes (e.g. presence of an inside lock) or level of service attributes (e.g. number of households sharing). Both QoS and QoL measures provide useful information and can be complementary, but it is important to be clear on whose perspective they provide, to inform their intended purpose (Table 1).

An example of a QoL measure is SanQoL, and I hope to publish a paper on its development, validity and reliability soon. An example of a QoS measure is the “sanitation quality index” (SQI) presented in the side-event (link at top) by EAWAG and others. Both SanQoL and SQI focus on the containment stage, but QoL and QoS have applications for emptying, treatment etc.

Table 1: Quality of what?

2. Measuring quality from whose perspective?

There are three aspects to this: (i) user or decision-maker focus, (ii) what goes in the measure, and (iii) who is evaluating/scaling its attributes.

First, the two main focuses for evaluating quality are typically the user or the decision-maker. For users, this might be direct (by asking users, e.g. SanQoL) or by proxy through evaluating infrastructure characteristics and inferring user-perceived quality indirectly (e.g. SQI). For decision-makers, sanitation regulation, for example, may give explicit or implicit guidance on minimum quality standards. In some cities, there are debates as to whether container-based sanitation should be permitted. Furthermore, downstream implications of quality for broader impacts are also important to decision-makers. The most obvious example here is health of the broader public via externalities (e.g. when septic tanks discharge to drains, illegal dumping of sludge, leaking sewers). This is a broader point raised in the first two presentations in the side-event. In summary, if you ask users and decision-makers about what is important for sanitation quality, you are likely to get slightly different answers.

Second, on what goes into a measure of quality, QoL measures generally follow the principle that the inclusion of attributes/items should be based on the priorities of the target population. That means consulting users rather than “expert opinion”. The qualitative study for the SanQoL measure is currently undergoing peer review. The same should go for QoS measures, and qualitative work underlying the SQI was recently published here (leading them to include things like material of floor, roof, solid waste, insects etc.)

Third, the question of “whose perspective” during fieldwork is also important (Table 1). For assessing infrastructure attributes (slab, walls, lock on door) there are many options, e.g. an enumerator can observe, or ask the user, or both. In theory, such indicators should be objective. However, there will always be an element of subjectivity when an enumerator is evaluating presence/absence of faeces, or the material of the walls is (e.g. how large do holes in corrugated metal sheets have to be until privacy is unlikely?). When it comes to less observable technical aspects (e.g. lining, septic tank or holding tank?) this can be very hard to verify regardless of who is responding. However, level of service attributes which are part of QoS (e.g. number of users, sharing status), cannot be observed and users must be asked.

When it comes to QoL (e.g. user perception of safety, privacy), only the user is in a position to respond, and each individual is different. This highlights an important distinction between QoS and QoL measures. Results for QoS measures should be identical or near-identical when asking two users of the same toilet (i.e. very high inter-rater reliability). For QoL measures, however, there is no reason to expect this. There should probably be correlation, but two users might experience the same toilet very differently (see example in this post) and so have different QoL outcomes. In the QoL case then, it is not a question of inter-rater reliability, though inter-rater methods could be used to assess the level of correlation.

3. Measuring quality for what purpose?

The purpose of informing regulation or defining standards was flagged above. The SQI has its roots in WSUP wanting to push back on the SDG sanitation ladder categorising shared sanitation as “limited”. Other possible purposes of measuring quality are routine M&E (e.g. by municipalities or NGOs), impact evaluation aspiring to causal inference, or economic evaluation. SanQoL could be used in impact evaluation – it would be useful to compare multiple intervention options in the extent to which they improve QoL. However, the ultimate intended purpose of SanQoL is in economic evaluation, that is, cost-benefit analysis (CBA) or cost-effectiveness analysis (CEA). I have developed a measure of the value of sanitation, weighted using SanQoL, which could be used in CBA or CEA. More on that in due course… The broad point here is that the purpose of measuring quality should inform the design of any measure. Economic evaluation frameworks are quite restrictive in terms of what methods should be used to infer value, for example. Routine M&E may be restricted by what data are feasible to collect quickly, and implementers may be more interested in basic aspects of QoS from a quality control perspective.

4. Conclusion

When measuring the quality of sanitation, it is important to be clear on three things: the quality of what, quality from whose perspective, and the purpose which the measurement aims to inform. To illustrate these points, I have distinguished between quality of service (QoS) and quality of life (QoL), and I continue to believe both are complementary. Some purposes require QoS data and others QoL data. If the ultimate aim is to understand user experience, it is probably important to collect both, in order to explore QoS variables as influences on QoL, as the Clean Team study presented in the side-event aimed to do.