On this page

HCET Home > On-line Training > What Works in Adolescent Sexual Health Programs > 4. Components & Integration of a Science-Based Approach > Evaluation

Evaluation

• Determine whether or not the program is actually doing what it was intended to do
• Discover new knowledge about the program and its activities
• Recognize problem areas and strengthen the program in the future
• Meet requirements for funding and expansion
• Determine changes in knowledge, attitude, and behavior among participants in the program

Evaluation Types

1. Process
2. Outcome
3. Impact

1. Process Evaluation: Evaluation throughout the program design, planning, implementation, is known as a process evaluation. The purpose of the process evaluation is to determine whether or not the program is being implemented the way it was intended to be. The process evaluation should be conducted during the implementation and final events of the program.
   The process evaluation should address such things as ^[26]:
    
   • Is the program serving the population it was intended to serve?
   • How many people participated in the program?
   • Is the program using the amount of money it had planned to use?
   • Were the events which occurred during the program purposeful?
   • How many activities and meetings were actually conducted?
   • How many training sessions were there total?
   • How many staff members were needed to deliver the services?
2. Outcome Evaluation: Evaluation at the end of the program or later is known as outcome evaluation. The purpose of an outcome evaluation is to determine whether or not there was a change in any or all of the following ^[26]:
    
   • Risk and Protective Factors
   • Participant’s attitudes, knowledge, or behavior
   • Environment (policy, social norms, etc.)
   • Trends in morbidity/mortality
3. Impact Evaluation: Evaluation following a period of time after the program is complete is known as impact evaluation. The purpose of an impact evaluation is to determine whether or not actual changes in knowledge, attitudes, and/or behavior among program participants can be attributed to their participation in the program. Impact evaluations are difficult to conduct and may not always be required. This type of evaluation requires a lot of time. It may be difficult to track participants for a long period of time and to identify changes among those participants. On the other hand, if you do have the time, impact evaluations can be beneficial. Impact evaluation can improve program efficacy by addressing such things as ^[28]:

 
• Did the program achieve its desired result?
• Can the changes in outcomes be attributed to the program, or could they have been influenced by other factors not related to the program?
• Were there any unintended outcomes, positive or negative?
• Is the program worth the resources?

1. Write Objectives
1. Process Objectives
2. Outcome Objectives
3. Impact Objectives
2. Create Evaluation Tools
1. Qualitative Evaluation Methods
2. Quantitative Evaluation Methods

1. Write Objectives: Creating a logic model is one of the first steps in evaluating a program. With the logic model, you have your program goals and activities outlined for you. For the program evaluation plan, you can use the logic model to write process, outcome, and impact objectives (Figure 8). These objectives should be specific (who, what, how much, and by when), measurable, and realistic.
Template for writing objectives ^[27]: By (date of completion) (describe specifically who) will (explain change/goal) as shown by (explain method of recognizing that the change has occurred).
1. Process Objectives: Write Process Objectives using the intervention activities from your BDI Logic Model.

   Examples:

• By the end of December 2007, 60% of participants in the ACTION Program will have attended 12 or more of the 16 education sessions from “Becoming a Responsible Teen,” as indicated by attendance records.
• By the completion of the program, 80% of participants in the ACTION Program will have been given written information on where condoms and contraception are available near Centerville High School.
2. Outcome Objectives: Write Outcome objectives using the determinants and behaviors from the BDI Logic Model.

 
• At the completion of the program, 50% of the ACTION program participants will have increased knowledge about the consequences of unsafe sexual intercourse as indicated in pre and post tests.
• By August 2007, 75% of Center High School students participating in the program will have demonstrated an increase in knowledge about contraception availability by identifying three locations where students can access contraception or condoms as indicated in pre and post tests.
3. Impact Objectives: Write Impact Objectives using the Health Goal from the BDI Logic Model as a guide in determining the focus of your objectives.

   Examples:

• By December 2008, 10% of program’s sexually active participants will have increased the correct use of condoms while engaging in sexual intercourse as illustrated by self-reported data from follow-up surveys delivered through e-mail.
• By August 2007, 25% of program’s sexually active participants will have increased contraceptive use as reported by participants in year-end interviews.
2. Create Evaluation Tools: Now that the Logic Model is complete and the process, outcome, and impact objectives are written, you should have a pretty good idea what you need to measure and how by describing what questions to ask and what tools to use. This will assist you in creating evaluation tools that will measure your stated objectives. There are many different ways to evaluate a program – through pre/post tests, surveys, questionnaires, interviews, conversations, and observation. Some evaluations may use qualitative methods and others may use quantitative methods.
1. Qualitative evaluation methods emphasize collecting in-depth information about the participants’ experience ^[27]. Qualitative evaluation may be conducted through interviews, focus groups, journals, observations, open ended questions on surveys, etc. This can be more time consuming, but it provides more detailed information about individual participant perspective.
2. Quantitative evaluation methods are data and statistics driven. They emphasize precise results that can be measured through questions answered by yes or no on surveys and questionnaires, attendance, grades, and tests ^[27]. Often times quantitative evaluations are easier to conduct with large populations. Many funders want to see results from quantitative evaluations because they are easiest to measure statistical effectiveness and impact on program participants.
From the objectives, you can see that some evaluations require quantitative methods and others will be fulfilled by qualitative methods.
For example:
One process objective stated: “By the end of December 2007, 60% of participants in the ACTION Program will have attended 12 or more of the 16 education sessions from ‘Becoming a Responsible Teen,’ as indicated by attendance records.” This is a process objective that can be monitored and fulfilled by attendance, a quantitative method of evaluation. In order to achieve your process objective, attendance should be recorded regularly, and if attendance drops below 60%, action should be taken to make improvements.
One of the impact objectives stated: “By August 2007, 25% of program sexually active participants will have increased use of contraceptives as self-reported by participants in year-end interviews.” The comments made in personal interviews at the end of the year, a qualitative evaluation method, will determine if the impact objective is fulfilled. Participants should either self-report an increase in their use of condoms or contraception throughout the year or at the beginning of the year, program administrators should have gathered baseline data on participants’ condom or contraception use in order to compare information to see if there was an increase.
With either qualitative or quantitative evaluation, programs are able to gain insight into program achievements, challenges, and needs. Evaluation of any kind is helpful in guiding future program planning and services. It is important to remember, however, evaluation is only useful if the programs use the evaluation tools and consider results of the evaluations in future program planning.

Section: Main 1 2 3 4 5 6 7 8 9

©Health Care Education & Training, Inc.
URL: http://65.163.14.21/training/ASH/4eEvaluation.htm
 Last update: 03/05/08