In this step you will:
- Determine the extent of the changes on identified values
- Create measures (descriptive scales, proxy, and natural) for the changes
After identifying the benefits, this step will help you characterize and measure the degree of change you can attribute to your program so you can communicate the level of benefits your program provides. You do this by establishing performance measures for your program.
| For this step: | |
| Resources required: | Conducted through focus groups and workshops, or as as a desk analysis by the coordinator. |
| Expected outcomes: | Performance measures and indicators, cost saving analysis and benefit return on investment. |
| Tools: | Characterizing the Change (Tool 4a) Measuring Attribution (Tool 4b) Timescale of Benefits Matrix (Tool 4c) |
Time spent gathering data
Be aware of how much time you spend gathering data. Data gathering can be a very time-consuming task, especially if you are chasing after an accuracy which may not be there. The purpose of measuring is to understand and manage the changes, as well as to be able to communicate value. Be prepared to use the estimates based on best available information and sound methods.
Understanding performance measures
A performance measure or indicator is a way to quantify or qualify the change in benefits (or adverse impacts). They help to articulate the impact on the environment, the community, and financial returns. This step determines the extent of the changes (positive benefits and adverse impacts) on identified values, helping to clarify:
- How to describe changes resulting from the program
- The best measures for those changes
- How to assess the impact of the benefits
- Cost-saving and cost-efficiency analyses
- Understanding financial and non-financial benefits, including ROI
Performance measures help to answer questions like; how much has water quality improved, or how much peace of mind has the community gained from consistently knowing water conditions?
Principles of good performance measures for CBWM benefits
Relevance: The measures need to directly relate to the objectives (e.g., peace of mind, ecological health, and park fee revenue).
Sensitivity: Indicators should capture even subtle changes that might affect these broader objectives.
Accuracy: Ensure consistency in data collection, especially when dealing with subjective metrics like surveys.
Easy to Understand: Measures should be easily understood and interpreted by the community (especially when dealing with complex indices or scales).
Process
1. Identify good performance measures for your program (Tool 4a)
Anything that you’ve identified as a key benefit needs to be measured. Performance measures can be qualitative or quantitative, and can be a specific direct indicator, a proxy, an index, or a scale. Types of performance measures include:
Natural measures: These are direct measurements of the parameter of interest. They are the most straightforward and understandable indicators and can often be interpreted without additional context (e.g. # of jobs, cost in $, hectares, # of species). However, they may be costly, time-consuming, or challenging to obtain. They may also focus on just one aspect of a broader issue, which might not capture the full picture of a situation.
Examples:
- Peace of mind: Direct survey of community survey or questionnaire where community members are asked to rate their level of peace of mind regarding the quality of their water supply on a scale.
- Day fees: Actual number of day fee sales in dollars based on renewed interest in parks, adjusted for the market and before/after CBWM intervention.
Proxy measures: These are indirect measures that can substitute for direct ones and are easier to obtain. However, proxies can sometimes be misleading or not widely accepted.
Examples:
- Ecological health: Using water quality measurements like pH or contaminant levels (e.g., nitrates) as a proxy for overall ecosystem health.
- Peace of mind: Measuring the number of complaints or inquiries about water quality. A drop in complaints may indicate increased peace of mind, assuming other factors remain constant.
Index or constructed scale measures: These combine multiple individual measures into a composite score or index. They help in summarizing and interpreting complex data by reducing it to a single value or a set of values. These are often represented in 1-5 scale or a High – Med – Low scale – such as in satisfaction from surveys. These are often used because they can be created to measure anything using meaningful language, making interpretation and comparability not only possible but easier.
Validity and sensitivity of scales
The validity and sensitivity of the scale may be affected by how it is developed and scored (i.e. expert judgement, community survey).
Choosing the best measure
Which is the ‘best’ measurement? Natural measures are often preferred for their simplicity and directness. When resources are limited or direct measurement is difficult, proxy measures may be the best option. If you need a holistic view that captures multiple dimensions, constructed scale measures are ideal. Keep in mind the purpose of the Case for Investment. The overall goal is to get a better understanding of the relative changes from baseline that can be attributed to the program. So, a relative measure such as the “community peace of mind” has improved by 20% (or gone from a Low to a High) compared to the previous year gives a clearer context than the absolute measure of a number from a survey.
2. Measure the impact of the benefits from your program (Tool 4b)
When trying to figure out how much your community water monitoring program has helped, it is worth asking a few questions to assess whether the outcomes you have identified result from your activities. These questions will help you to reduce the risk of over-claiming, making your Case for Investment more credible:
- Would these end benefits have happened without your program? (Counterfactuals and Deadweight)
- Did your program’s impact simply shift the problem elsewhere rather than solving it? (Displacement)
- Did your program encourage others to get involved? (Ripple)
- Did or do you expect the effects of the program will reduce over time? (Drop-Off or Decay)
- Do you expect some effects to take time to appear? (Delay)
- How certain are you about the impacts, benefits and predictions? (Uncertainty)
When assessing your program’s impact, it’s important to consider a few key factors:
- Deadweight: What would have happened without your program? Comparing similar water bodies without intervention can help estimate this, though it may require using trends or best guesses.
- Displacement or ripple effects: Did your program replace other positive outcomes, or did it inspire others to take action?
- Timing: Some outcomes take time to appear, while others have an immediate but shorter-lived impact.
- Uncertainty: It’s natural to face challenges like separating your program’s impact from other factors or dealing with limited data. Acknowledging these uncertainties and including them in your evaluation (e.g., through attribution factors) strengthens your case.
When communicating your program’s value, focus on clear evidence that ties your results to your efforts and show how additional resources can enhance your impact. Engaging the community and stakeholders can also provide valuable insights when advanced methods aren’t possible.
3. Consider program benefits over time (Tool 4c)
Will the influence of the program on outcomes diminish, maintain or increase as time passes? Thinking about CBWM program benefits in terms of time, from immediate awareness and behavioral changes to long-term data accumulation and policy shifts, encourages the articulation of a full range of benefits, and underscores the importance of time in realizing these benefits.
In the short-term, community-based water monitoring provides immediate awareness and education. Participants and local residents become more knowledgeable about the quality and health of their water sources. Another short-term benefit is fostering stewardship and community engagement. When people are directly involved in monitoring their water sources, they feel more connected to their environment and have a vested interest in its wellbeing. This can lead to immediate behavioural changes, such as reduced littering or mindful water use. An example is programs like the Stream Selfie, where community members photograph and report on local stream conditions, providing real-time snapshots of water health. Such programs can quickly identify issues like pollution incidents, prompting rapid responses from both community members and authorities.
Long-term benefits, however, unfold over extended periods, such as guiding restoration efforts overtime. One key aspect is the accumulation of data, which allows for trend analysis and long-term environmental planning. For example, the data collected over years by community volunteers in the Chesapeake Bay Program has been instrumental in understanding long-term changes in water quality and guiding restoration efforts.
Note on data collection
Data collected in your CBWM efforts should be tied to a current or future decision. Otherwise, data may be collected for the sake of collecting data, which would end up being an expensive waste of resources. Similarly, the data collected to validate your Case for Investment should be relevant to an outcome/benefit you are seeking to express, ensuring solid performance measures.
Another long-term benefit is the gradual change in public policy and behavioral changes, such as community practices. Continuous community monitoring can provide the data necessary to influence policy decisions, leading to more sustainable water management practices. This was evident in the Lake Winnipeg Community-Based Monitoring Network, where data collected by volunteers over several years influenced agricultural practices and policies to reduce nutrient runoff.
Generalizing your values categories
Many of these categories of benefits are relatively universal, so it may be possible to develop a generic set of value categories over time based on the approach outlined in this guidebook. A follow up study could also support this effort by characterizing the social and cultural benefits of these programs.