Volunteer community health workers (CHWs) are a major strategy for increasing access to and coverage of basic health interventions. Our village health worker training course reviews the process of training and continuing education of CHWs as an important component of involving communities in their own health service delivery. Participants will be guided through the steps of planning training and continuing education activities for village volunteers. The course draws on real-life examples from community-directed onchocerciasis control, village health worker programs, community case management efforts, peer educators programs and patent medicine vendor training programs, to name a few.
Lecture 9B: Pre- and Post-Tests
Loading...
来自 Johns Hopkins University 的课程
Training and Learning Programs for Volunteer Community Health Workers
17 个评分
从本节课中
Program Evaluation and Follow-Up
In this module, we will focus on evaluation, performance, and supervision. There will be a quiz covering lectures 9-10 after completing the lectures in this module.
与讲师见面
William Brieger, DrPHProfessor, Health Systems Program of the Department of International Health
Bloomberg School of Public Health
At this point, we want to look first
at the pre-post test process of evaluating training outcomes.
A pre-test can be planned as part of the
activities during the very first session of the training program.
Maybe an hour or could be set aside the first morning.
Or if you have a registration going on the day before, maybe there could be a quiet
room, where people can take the pre-test as part of the registration activities.
we want to make sure that this provides, you know, that we
have an opportunity, a timing that is not going to eat into the main schedule.
But we want to make sure that we have an opportunity to get this information.
Now, a pre-test would be slightly different than a baseline survey.
A baseline survey of
potential trainees may include many people who
are not actually attending the current training session.
So, what the value of a pre-test can do is that
you can have information on specific individuals, both before and after.
And measure, in fact, does that individual's
score on the test change, does it improve?
There are various ways of doing this.
Again, the question should be, based on
our diagnostic findings, on our baseline findings.
And on the specific content that's going to
be presented as evidence in our training objectives.
These tests generally focus more on knowledge.
we can certainly link in skills if we're talking about the skill
of preparing oral rehydration
[INAUDIBLE]
solution, we can certainly ask people to list the
steps involved in correct order and the amounts involved.
But it's usually these tests do not show knowledge.
Now we have an example where there are possibilities of doing skill tests
one of our MPH students in Ibadan wanted to train teachers on giving
visual acuity test to their pupils.
And he actually provided the materials and asked the teachers
to demonstrate it or as you can imagine, prior to
the training, most of them just looked at the materials
on the table and didn't know what to do with them.
But at least there was a skill baseline
pre-test that could be used to compare with post-test.
And again, the post-test to be useful has to be the exact
same questions, exact same procedures that occurred at baseline and
pre-test that would be repeated again at the last session.
So just as you've scheduled time during the first
day or the registration for people to take the
pre-test, you also have to schedule time at the
end of the session for them to take the post-test.
One thing just to note for confidentiality,
you may want to assign participants a secret number that only they know.
And they put that secret number on their pre-test and post-test so that you
can compare them but not necessarily divulge whose results.
On the other hand, if, if the participants want their results back, they want
to compare them, then clearly you would want to put their names on it,
so that you could return the the tests to them.
So this has to be decided as part of the training committee plans.
The tests have various types of questions on them.
These can include open-ended questions, multiple choice
questions, and attitude, opinion statements, or items.
I need to balance the types of questions that we use
in our tests.
We want to cover the scope of the knowledge, but we want to make sure
these tests are simple enough that the participants
can complete them in 30 minutes or less.
As we mentioned before, you need to schedule time to do
these tests, and you don't want to eat into the process.
You don't want to make the trainees feel too nervous or worried about a
testing procedure, so you want to have
it relatively simple, straightforward, and consuming little time.
Open-ended questions are useful in terms of
people describing processes or giving their opinions.
it takes a little more effort to score these
kinds of questions, you have to develop a coding guide.
But it lets the trainees put down information in their own words.
Examples of open-ended questions could be, list all the ingredients that can be
mixed together for a homemade solution for oral rehydration
or describe the signs of, danger signs of dehydration.
Or mention the steps in order, for making homemade ORS.
Open-ended interviews have been very useful when you
have trainees who have a low literacy skills.
Where they can't, you can't give them a four or
five-page test and expect them to sit down and read it.
And fill it out, even if it's in a local language.
So, in situations like that, what we've done is arrange a
group of trainers to serve as, just like interviewers, and provide rooms
or space where the trainees can one by one by one go
and be interviewed with the open-ended
questions, and then their results recorded.
Multiple choice questions. You all
have experienced those before. They give you options.
And you obviously mark on the option which is correct.
Now, there are different ways of doing multiple choice.
You can have a situation where only one option is correct.
Or you could have a situation where many options are correct.
And, again, the scoring depends on how you do this.
If you agree in advance that only one option
will be correct and you have a ten-item multiple
choice questionnaire, then what we would have is
a situation where the maximum score would be 10.
There would only be one right answer for each question.
But if you have ten items on ten questions, and each
question has a choice of five items, as we see here.
what are all the, what are the ingredients that one can mix together.
We have five different ingredients.
Sugar, soft drink, salt, vegetables, water.
So they have to decide if each of these is right or wrong.
Almost like a true-false.
And if you had ten questions like this, with five options,
you would really have a score of 50 possible correct answers.
So clearly, you think about it, though look
at this yourself and think what would you score?
What's your right? What's your wrong?
What's your wrong?
Okay, the homemade solution that we've been
teaching people includes sugar, but not soft drinks.
Soft drinks would be bad for a child with diarrhea.
Salt is another ingredient.
Vegetables per se, solid food.
Continued feeding might be possible for oral rehydration,
but it wouldn't be part of a solution.
You can't put a zucchini or, or a potato in, in a solution easily.
And then the ingredient water, of course, that makes a solution.
So, sugar, salt, and water should be ticked.
And soft drink and vegetables would be left blank.
And that's how you would score it.
Another type of question that you can have
on your pre- and post-tests are attitude items.
If you're trying to encourage
people to value or have a positive attitude about
oral rehydration, then you could use some of these questions.
We've found, for example, that health workers were
uncomfortable at first when oral rehydration was instituted.
They wanted to prescribe drugs.
They didn't feel that were doing their job properly,
unless they were prescribing drugs for children with diarrhea.
And it took a lot of effort not just to
teach them the reasons for this, the importance of replacing electrolytes, the
fact that many diarrheals are caused by viruses that don't respond to antibiotics.
the issue of antidiarrheal drugs clogging up the
intestines and holding bacteria in or viruses in.
So, it was more than just teaching them those facts.
It was also, you know, letting them think about the fact that this was
a good, a valuable, a useful treatment and
that they would be doing the best for mothers.
So here's a simple attitude test with, with four items mentioned.
And you can obviously check or tick which one
you agree with or disagree with or are uncertain.
You can frame some of these in positive ways.
mothers can be trusted to mix ORS
at home.
And then a few items later you can reverse
it just to see if the people are being consistent
and say we can't trust mothers to mix ORS
or take care of their children with diarrhea at home.
So you can, can have that.
And then, of course, in your scoring process, if a question is phrased
in a positive way, agree might be three points, uncertain, two, and disagree, one.
But if it's framed in a negative way, agree may be one
point, uncertain two points, and disagree with B3.
So, you in terms of getting a final score, you have to be aware.
With any kinds of testing procedures, just as we talked about.
education materials, tests also need to be
pre-tested to see if people understand the questions
if the questions are easy to answer if
people are responding in ways that make sense.
If
[INAUDIBLE]
you have a question where 99% of the people agree
with it or disagree with it, it's not a very good
test question or it also shows that people you know, are
already having the attitudes that you're going to train them on.
So you want to make sure that you get a sample of
people to try out the test first, to see if it works.
Some examples of how we've used tests, include the patent medicine vendor
training in a rural community where I worked in Nigeria.
what we did, of course, was a we did a baseline survey
to find out what they were thinking and doing in their shops.
but we also had a pre-test, the, as we mentioned before, a training
committee was formed among the patent medicine
vendors, so that they could plan activities.
They, in addition
to the information that we gathered during our baseline survey,
that committee members requested certain lessons on certain illness and skills.
Like reading a prescription.
We had a situation where either the clerk in the shop or the shop owner
attended, usually it was not both because they didn't want to leave the shop vacant.
And then we did a pre/post test. We had a, in addition,
we had a control community where it was
about 25 miles away in another local government.
And so, we interviewed patent medicine vendors there at about the same
time we did the pre-test for the ones in our intervention area.
And then, did a post-test after the training in both places.
In terms of the pre-test, you can see from graph, that the trainees and
the control patent medicine vendors had very similar scores.
And then, afterwards, the post-test shows a significant increase in the
knowledge of the trainees, the people who went through the program.
And the value of this, as you've
probably learned from your various epidemiological courses is
that if we just have the trainees alone, a pre-test post test, we couldn't be sure
if it was really our training or maybe they enjoyed the experience.
Maybe they were upset by the initial questionnaire and were inhibited.
Maybe during the course of the training they went out
and were exposed to other activities that increased their knowledge.
So by having a control group, we can say that people under the similar
type of people, you know, whether they
experienced something in the environment that increase
their knowledge or not, or whether the changes that we're
observing at post-test can be really attributed to the training program.
One thing, another thing that was important, as they said
before, if you can identify the people who are in
the training program, and either give them some sort of
identi, ID number, so you can compare pre and post.
one thing that we found is that
the people who took the pre-test and
post-test were not exact, were not identical.
And this happens.
People don't show up for all the sessions. Some people join late.
So we found that 33 took the pre-test, 37 took the post-test.
but 28 of them took both tests.
So, in addition to looking at the gross results of the 33 pre-tests
versus the 37 post-tests as you saw in the graph in the previous slide.
We could also do what we call a paired t-test on the 28 simply subtracting the
[INAUDIBLE]
pre-test from the post-test and getting a mean for that and seeing if this
difference between the two scores was different than zero.
And we found that the average pre-test for this group was 46.
The post-test was 71 and by subtracting each individual's pre-test from post-test,
found that these
[INAUDIBLE]
these differences were significantly different than zero or zero
meaning no change, as we saw in the control group.
So, ideally if you, for yourselves, as, as trainers and evaluators,
you would want to see, did the training actually make a difference?
To the individuals, not just the whole group.
Because the whole group, you don't know
some may have lost knowledge for some reason.
Some may have
gained more than others what was going on.
So, this being able to pair up the results and look at it
gives you more confidence that the
improvement did occur with each, individual trainee.
Another patent medicine vendor training program was conducted
in Kenya by a USAID project called the Quality Assurance Project.
Their goal was to equip the patent medicine vendors
with customized job aids to communicate
new malaria guidelines to put drug outlets.
So there are a number of private drug shops around in the community.
the country had changed from chloroquine
to sulfadoxine-pyrimethamine because of drug resistance.
And they wanted to make sure that patent medicine
vendors who dispense quite a lot of these countries'
drugs understood this change.
So again, the idea was that if you
train the patent medicine vendors, they would be in
a good position to educate their customers on the
new drugs, malaria recognition, the importance of prompt treatment.
But also the trainees who would, in turn,
be able to train other patent medicine vendors.
So this was sort of an outreach type of program.
They were tested on specific areas of knowledge about malaria.
And as we can see in the chart on this
table there was an intervention group, there was a control group.
And interestingly enough can see that there are some
elements of knowledge that are, must be common knowledge in the community.
The very first thing, that fever is a symptom
of malaria and see that both groups knew that.
Majority, we ask majority knew that.
the issue of the new drug SP is also known commercially as Fansidar.
You can see the items about Fansidar, which was the focus of training and how to
use this new drug, were all higher for
the intervention group compared to the control group that
did not have this training.
So, again, not only do we have a gross score that
we can compare, but we can also look at specific items.
And we can look at that and say, okay, well, overall at the end of training
a number of these items, 95% for you
shouldn't sell drugs below the exist, the correct dose.
Again, recognizing the, the symptoms noting
that the Fansidar is a very effective drugs.
At the same time you can see that still,
about a quarter of them didn't really believe or
understand that Fansidar could be sold in their shops,
that you didn't have to get it at a pharmacy.
And we can see that 17% still didn't know that Fansidar,
a single dose, was enough for treatment. That it's packaged
that way so you just take it once.
So there were still some gaps but again, you
know, the, this overall improvement compared to the control
group, but it does tell you that you may
need to follow up with some of the trainees.
So by having knowledge specific results, you know item specific results, you can
see where the training was successful and where there are still some knowledge gaps.
Another thing that this Kenyan
program showed was that there are different categories of trainees.
They all may have been in one room together, but it was found that the
response was different by educational level.
And even though in all cases, as you can
see the darker green bar shows that the control
of the control group, I mean the
intervention group, did better than the control group.
You can also see differential among the intervention and the control respondents
in terms of their level of knowledge by their level of education.
Std 1-8 is like primary school.
Form 4 is like, sort of like junior high. people that have let's
see, then you can see that then senior high.
Also whats interesting is that the people with
high school education were more knowledgeable to begin with.
And so the intervention group, although they gained over
the control group, it wasn't that great a gain.
Whereas, those who had less education, only primary education,
they really gained a lot more by this program.
So we have to recognize that people of,
of different levels of education perform differently and
we have to make sure that our training
materials and exercises are geared toward all these differences.
That with the program is comprehensible by people who have only finished primary
school as easy as it is for people who have finished high school.
As I mentioned
earlier, one of our MPH students for his dissertation did a
program on training primary school teachers to test for visual acuity.
He was an optometrist himself when he came to the program.
And he was interested in preventative work community-based work.
He developed both a knowledge test as well as performance test.
The 17-point knowledge test that has
things like what structure of the eye is responsible for image formation?
What are three common causes of blindness?
Name two ways of detecting visual problems in student.
And just as we have done before, we did a pre-test, post-test with both intervention
schools teachers and intervention schools and teachers in control schools.
And as you can see here at baseline, the scores were on average four
out of 17. At
post test the the trainees
achieved almost over 13 points on average out of 17.
But you can see also that the the control group
did increase their knowledge by, on average, about two points.
And this is one of the things that happens in,
in the testing process by exposing the intervention teachers to the
pre-test itself. They thought about these questions.
They became curious.
They may have asked people around them, well,
I had this test today, what is this about?
And so, the testing process itself stimulated some knowledge acquisition.
But clearly it was not of the level that the people who experienced
a real training program.
In terms of training affecting attitudes, we have an
example from the CDT community-directed treatment in onchocerciasis control.
What was involved was training local government,
district health workers, to carry out the program.
We've talked about this before.
But it involves organizing community meetings explaining the program to
the community encouraging the community to
take responsibility for distributing it's own ivermectin.
Which involves volunteers who would be
trained that involves community collecting as
drugs, community collecting the records, statistics,
reporting the community managing side effects.
These types of things. And so, the program hands over
much of the responsibility for the day-to-day
delivery of the program to the community.
Which is why it's called community directive.
This idea of handing over some control to the community
was seen as somewhat threatening to some of the healtworkers.
They were concerned that the community may not do it correctly, that they may not, if
anything went wrong, they would be blamed. And so during prior to the intervention
of this program we had a series
of attitude statements with the health workers.
The health workers were trained to
organize the program in their local government.
And then
[INAUDIBLE]
we followed up.
Some of the statements with they could agree-disagree included
communities are quite capable of managing the distribution of ivermectin.
And in contrast onchocerciasis control should best be run by district.
So these are some things that we asked them in addition.
Community involvement in ivermectin distribution saves time for the health
workers who could be doing other things.
So, again, seeing a different value, seeing it in a positive light.
Health workers in this community cannot handle
ivermectin because they're distribution because they're too busy.
They'r overworked.
And then, again, health workers do
not believe that community directed distribution
of ivermectin is the best way to make it available to people.
In contrast to programs that have had it based
in health centers per se, or the health workers going
out and handing it out themselves.
So their response to these statements showed
whether they favored the program or not.
And as we can see in the chart, we have a pre-intervention survey,
where we have all of the health workers in the study districts combined.
And this was the distribution of their attitude scores.
it was reconfigured
to be on a scale of minus 25 to
25 whereas the positive side indicated a positive attitude.
So, in other words, the mid, midline was set at zero.
We see that, after the intervention, we have two types of intervention.
One, where we train the health workers simply to go out
into the villages and do their normal mobilization and helping the village
setup it's own program and decisions.
And then, we had an enhanced version where we,
in addition to the village meetings, we brought villagers
together at central places to have stakeholder meetings where
they could interact more freely with the health workers.
In both of these situations, the health workers improve their attitudes toward
this community-directed distribution. But those
who had the additional intervention of a stakeholder's meeting where there
was more interaction with the community developed a more positive attitude.
But, just going through the process, experiencing the, the
training, then going out in the field and applying it.
Interacting with the community.
They recognize that the community didn't have the resources and, and
the willingness to help carry out the program, and their attitudes improved.
So, here is an example of measuring that.
Another variable that we often look at, in
terms of training programs, is not just enhancing
people's skills but people's self-confidence or self-efficacy that
they can carry out a skill or a program.
And this is something we can clearly
measure before people leave the training site.
The actual application
of the skill we may not know until they get
back to the classroom, to the community, wherever they're working.
The visual acuity testing looked at this question of self-efficacy.
And as you can see, we asked the teachers
did they, how confident did they feel in four things.
Recognizing a child with visual problems, using the
visual acuity chart, interpreting the chart results, and
knowing when to make a referral.
And so, of course the options are
they're very capable, capable, unsure, not capable.
And they could check what they, what they felt.
And we did this again before and after.
One thing that was very interesting is that
prior to the intervention, the groups were very similar.
The intervention group, basically, doubled their perception
of self-confidence.
Whereas, interesting enough, the control group decreased.
So the, confrontation of the test, thinking about
these things, thinking about what's involved, actually made those
who did not receive the training feel less
secure in their ability to, to identify visual problems.
So, they recognize their limitations more after taking the test.
So we did see changes in, because
of exposure, we had changes in people's confidence.
And, again, exposure to the test itself,
especially the skill test made the teachers
uncomfortable and, and feel less about their
ability to, to do this without training.
In summary, we can see that pre-/post-test document are knowledge gains.
If we match the results, we have an identification number of the trainees, we
can match pre- and post-test results and
be really sure that individuals did gain knowledge.
And then using a control group definitely helps us know whether the
changes we've seen were due to the effect of the training program and not some
other intervening factor.
For example, the training that was done for
the teachers was not done all in one session.
It was done over a couple weeks, so could have been something going
on in the environment where teachers could have learned more about visual acuity.
having a control group makes sure that, if we see changes,
it's likely to have occurred because of the, the training program.
The issue of control groups
especially if your implementing a program that is ultimately designed for
all of the health workers If you do it in phases,
your control group could be people coming for the next round
of training, so that your not excluding people from the program, ultimately.
Also, in terms of testing, we're looking at not just the overall
summary scores to see if there was improvement, but looking at individual
test questions to identify areas of strength
or weakness in the training program itself.
And finally, we need to consider the
results in terms of the trainee's background.
Does their educational level, for example, affect
their scoring, affect their ability to absorb
the knowledge that's being provided in the training, and how we can accomplish that?
We've talked about tests.
In our next section, we will be looking at other methods of gaining information.
