Recall bias and cost data

I’ve been working on costing a few programmes recently where the intervention happened between 3-10 years ago. Both used household surveys asking people what they spent (in cash and in kind) towards the original infrastructure output (CapEx), towards regular operational and maintenance (OpEx) and irregular capital maintenance (CapManEx). It’s got me thinking about the various recall bias issues involved.

Look at the graph below, which is completely hypothetical. Let’s assume it’s for a sanitation intervention amongst 1,000 households which happened in 2008. The y-axis is some measure of ‘data quality’ when you ask 1,000 households about expenditure. If you asked in 2008 how much they spent (cash and kind) to construct a toilet (CapEx – blue line), they’ll probably still have a good idea because it was very recent. However, as time goes by, they’ll forget exactly what they spent, so the blue line drops. Data quality will drop fast at first, but then plateau after a while because people are likely to remember the order of magnitude of  what they paid.

For OpEx (orange line), which is by definition recurrent expenditure which occurs with a regularity of one year or less, it’s a different problem. If you ask people in 2009 about their OpEx, they won’t have that good an idea because they only have 1 year’s experience of using the toilet. Maybe they’ve cleaned it regularly but not spent much more money so far. Over time, they build up more experience of how much they tend to spend on OpEx and data quality becomes good.

CapManEx (grey line) is the hardest. It is recurrent expenditure occurring less than once a year (e.g. pit or tank emptying costs). So any time you ask people about it they’re less likely to have experienced it recently than with OpEx. Stuff normally works well when it’s new, so people are unlikely to experience CapManEx for a fair few years. With a toilet, for example, you’re only likely to need to empty the pit or septic tank 3-8 years after installation depending on numerous factors. So you only start getting likelihood of good data on CapManEx many years after the intervention, but even then it’s never going to be brilliant because many people may not have incurred it recently, and you have the same recall bias issues.

So when should you do your cost data collection? It depends on your objectives. If you’re most interested in CapEx, then do it ASAP. But if the lifecycle element is more important to your study, then it’s probably best to wait at least 3 years, maybe even 6-8 years if it’s the CapManEx you’re most interested in. You can always impute the CapEx from other sources or ask a different sample of people who constructed more recently. Of course the best case would be to get regular data with the same panel of households at intervals, but who is going to fund that…