PME 802 - Module 1A
Let me just start by saying that I am all about the penguins.
The added reading of the ‘Making Sense of Evaluation’ handbook published by Superu won the 2017 Plain English Award and I love everything about it. For those who haven’t been in a PME class with me yet, my backstory for academic writing is rooted very firmly in my grandfather’s teaching. He drilled into me at a very early age that if you understand something you should be able to say it in one sentence or even show it in pictures.
Simple = understood.
To this day I find it difficult to wax poetic about just about anything so the penguin-filled, plain English manual was a reading for my warmed-up, boiled-down heart.
But I digress.
In Module 1A, we were tasked with presenting our take-aways from two videos on the whys of program evaluation. Here’s what resonated for me,
Reasons for evaluation:
Improving effectiveness
Objectively determine impact
Describe and create new programs
Provide accountability
Ensure sustainability
Approaches to Program Evaluation
Three types for the three stages of a program (implementation process – end-stage impact – long term outcome).
1. Process Evaluation
Main Question: Is implementation (materials, facilities, and methods) working?
Best for: program creators and facilitators
As Chen notes, “…the earlier that program evaluation techniques are incorporated in the planning of a program, the easier it becomes for the directors and implementers to improve the new program…” It stands to reason that process evaluations should be conducted as early as possible in order to inform alterations and reworking of all aspects of the program in question by allowing for meaningful feedback from both facilitators and participants. At this stage, it might be presumed that outcome goals are secondary to the desire for improvement, meaning that purposeful and useful changes can be more easily made. These changes, as Chen describes, ‘tend to be remedial’ and be quickly applied to improve the program’s experience.
2. Impact Evaluation
Main Question: Did the desired change(s) actually occur?
Best for: participant stakeholder
Impact evaluation can be undertaken immediately and is less costly (due to being less holistic and expansive) than its sister, the outcome evaluation. Quite simply, this level of evaluation asks that goals be determined and then measures whether that has occurred. As Chen points out, however, this can prove difficult as these outcomes usually require other data sets. The example given is a useful one: “data relating to outcome measures do not exist. For example, outcome data on participants in smoking cessation programs is not widely available. For such an eventuality, evaluators should be prepared to devise new instruments for measuring outcomes and other data.” If you are trying to measure whether your program made a difference compared to what would likely have otherwise occurred, you need to know what would have occurred!
Sources:
Chen, H. Monitoring the progress of a program. (2005). Practical program evaluation (pp. 182-193). Thousand Oaks, CA: SAGE Publications Ltd. doi: 10.4135/9781412985444
