There are numerous questions that might be addressed in a process evaluation. These questions can be selected by carefully considering what is important to know about the program. Examples of questions to ask yourself when designing an evaluation to understand and/or closely examine the processes in your programs, are:
1. On what basis do employees and/or the customers decide that products or services are needed?
2. What is required of employees in order to deliver the product or services?
3. How are employees trained about how to deliver the product or services?
4. How do customers or clients come into the program?
5. What is required of customers or client?
6. How do employees select which products or services will be provided to the customer or client?
7. What is the general process that customers or clients go through with the product or program?
8. What do customers or clients consider to be strengths of the program?
9. What do staff consider to be strengths of the product or program?
10. What typical complaints are heard from employees and/or customers?
11. What do employees and/or customers recommend to improve the product or program?
12. On what basis do employees and/or the customer decide that the product or services are no longer needed?
Outcomes-Based Evaluation
Program evaluation with an outcomes focus is increasingly important for nonprofits and asked for by funders.An outcomes-based evaluation facilitates your asking if your organization is really doing the right program activities to bring about the outcomes you believe (or better yet, you’ve verified) to be needed by your clients (rather than just engaging in busy activities which seem reasonable to do at the time). Outcomes are benefits to clients from participation in the program. Outcomes are usually in terms of enhanced learning (knowledge, perceptions/attitudes or skills) or conditions, e.g., increased literacy, self-reliance, etc. Outcomes are often confused with program outputs or units of services, e.g., the number of clients who went through a program.
The United Way of America (http://www.unitedway.org/outcomes/) provides an excellent overview of outcomes-based evaluation, including introduction to outcomes measurement, a program outcome model, why to measure outcomes, use of program outcome findings by agencies, eight steps to success for measuring outcomes, examples of outcomes and outcome indicators for various programs and the resources needed for measuring outcomes. The following information is a top-level summary of information from this site.
To accomplish an outcomes-based evaluation, you should first pilot, or test, this evaluation approach on one or two programs at most (before doing all programs).
The general steps to accomplish an outcomes-based evaluation include to:
1. Identify the major outcomes that you want to examine or verify for the program under evaluation. You might reflect on your mission (the overall purpose of your organization) and ask yourself what impacts you will have on your clients as you work towards your mission. For example, if your overall mission is to provide shelter and resources to abused women, then ask yourself what benefits this will have on those women if you effectively provide them shelter and other services or resources. As a last resort, you might ask yourself, “What major activities are we doing now?” and then for each activity, ask “Why are we doing that?” The answer to this “Why?” question is usually an outcome. This “last resort” approach, though, may just end up justifying ineffective activities you are doing now, rather than examining what you should be doing in the first place.
2. Choose the outcomes that you want to examine, prioritize the outcomes and, if your time and resources are limited, pick the top two to four most important outcomes to examine for now.
3. For each outcome, specify what observable measures, or indicators, will suggest that you’re achieving that key outcome with your clients. This is often the most important and enlightening step in outcomes-based evaluation. However, it is often the most challenging and even confusing step, too, because you’re suddenly going from a rather intangible concept, e.g., increased self-reliance, to specific activities, e.g., supporting clients to get themselves to and from work, staying off drugs and alcohol, etc. It helps to have a “devil’s advocate” during this phase of identifying indicators, i.e., someone who can question why you can assume that an outcome was reached because certain associated indicators were present.
4. Specify a “target” goal of clients, i.e., what number or percent of clients you commit to achieving specific outcomes with, e.g., “increased self-reliance (an outcome) for 70% of adult, African American women living in the inner city of Minneapolis as evidenced by the following measures (indicators) …”
5. Identify what information is needed to show these indicators, e.g., you’ll need to know how many clients in the target group went through the program, how many of them reliably undertook their own transportation to work and stayed off drugs,
6. Decide how can that information be efficiently and realistically gathered (see Selecting Which Methods to Use below). Consider program documentation, observation of program personnel and clients in the program, questionnaires and interviews about clients perceived benefits from the program, case studies of program failures and successes, etc. You may not need all of the above.
