Queensland Stay On Your Feet® - Toolkit Phase 4 Process Evaluation

Are we reaching the target audience and achieving planned strategies? (Process evaluation)

Process evaluation measures the actions of delivering the project/program and whether or not the planned strategies are being undertaken as they were intended ^{[89, 90]}.

Process evaluation is an ongoing examination of:

• adoption - who is delivering the project/program and what are they delivering? ^{[17, 21, 22, 89, 90]}
• reach - is the target group getting the information and accessing the project/program? ^{[17, 21, 22, 89, 90]}
• implementation - what is being delivered, how it is being delivered and is the project/program being delivered as intended? ^{[17, 21, 22, 89, 90]}.

These steps are from the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation and Maintenance) which can be used when planning, evaluating and reporting the results of the project/program ^[90].

Adoption

Adoption measures the number and proportion of settings, stakeholders and agencies involved with the project/program ^{[89, 90]}. This helps determine the level of involvement and types of resources that have been adopted by other organisations and communities ^{[89, 90]}. This is similar to reach; however, the focus is at an organisational level. By improving adoption, the project/program will reach more individuals ^[89].

Information that could be collected includes:

• the number of settings that have adopted the project/program ^[18] eg. community dwelling, acute care or residential aged care
• the number of organisations that have participated in project/program ^[89]
• the proportion of stakeholders/organisations that have adopted the services, strategies and resources ^[89]
• the number of organisations that have included falls prevention information in their orientations and training
• the number of organisations that have adopted any training/education packages that have been developed ^[89]
• the number of organisations that decline to be involved and why ^[90].

Reach

Reach is concerned with measuring the number and proportion of individual participants involved in your project/program ^{[89, 90, 93]}. It may also determine if your group is similar to the general population. This is known as representativeness ^{[89, 90, 93]}.

Reach measures the success of your recruitment, marketing and retention strategies by focusing on whether the target group was aware, involved with and retained by your project/program ^[89]. For more information on improving the reach of your project/program, visit: Reach and marketing

Information that could be collected includes:

• the number of resources distributed ^[90]
• the number of participants who said they received resources ^[90]
• the number of people who participated in the project/program ^[90]
• the reach to those most in need ^[90]
• the percentage of the population reached ^[93]
• the use of the project/program resources by individuals, compared with their intended use ^[90]
• the characteristics of participants and non-participants ^[89]
• representativeness of the participants in your project/program, compared with demographic profiles of participants of the broader community ^[90]
• satisfaction of participants and stakeholders ^[90]
• accessibility and affordability of the project/program and its resources ^[91]
• identification of what helped people to be involved (enablers) ^[90]
• identification of what discouraged people from being involved (barriers) ^[90]

top of page

Implementation

Reviewing the implementation process identifies whether there has been an effective or ineffective intervention and/or implementation process ^[90]. This can be analysed with individuals, organisations and the community ^{[89, 93]}. It is important to collect and analyse enough information to decide whether the project/program (or parts of it) could be repeated or taken up by others ^{[23, 89, 93]}.

Information that could be collected includes:

• the number of meetings held and who attended
• the number and quality of project/program materials ^[23]
• the use of project/program resources by organisations and communities, compared with their intended use ^[90]
• the delivery of any training/education packages developed, compared to expected delivery ^[90]
• the number and type of services delivered to the community
• the cost of implementation ^[91]
• the type of media coverage gained eg. press coverage measured in square centimetres, number of aired radio and television advertisements and media news interviews
• the delivery of the project/program within specified timeframes
• the quality of staff delivery of the project/program ^[23]
• issues raised throughout the delivery of the project/program
• delivery of all project/program elements and activities as planned ^{[17, 21, 89]}.

The evaluation planning worksheet and the list of evaluation tools will be useful for keeping track of all the evaluation questions. This may need to be modified to meet your project/program evaluation goals.

evaluation planning worksheet
evaluation tools

It is important to continually evaluate and analyse as the project/program is being implemented. If the target group are not being reached, it is unrealistic to expect that the project /program will be effective or have a positive impact on the target group.

You may need to take active steps to improve the effectiveness of the project/program’s strategies by contacting those organisations and/or individuals not participating and/or follow up with those who were participating but are not any longer.

To accurately record process information, you will need a systematic monitoring, record keeping and documentation process for the project/program ^[16]. For suggested methods of collecting process information, visit: Process evaluation tools.

top of page