At this point, we want to look first

at the pre-post test process of evaluating training outcomes.

A pre-test can be planned as part of the

activities during the very first session of the training program.

Maybe an hour or could be set aside the first morning.

Or if you have a registration going on the day before, maybe there could be a quiet

room, where people can take the pre-test as part of the registration activities.

we want to make sure that this provides, you know, that we

have an opportunity, a timing that is not going to eat into the main schedule.

But we want to make sure that we have an opportunity to get this information.

Now, a pre-test would be slightly different than a baseline survey.

A baseline survey of

potential trainees may include many people who

are not actually attending the current training session.

So, what the value of a pre-test can do is that

you can have information on specific individuals, both before and after.

And measure, in fact, does that individual's

score on the test change, does it improve?

There are various ways of doing this.

Again, the question should be, based on

our diagnostic findings, on our baseline findings.

And on the specific content that's going to

be presented as evidence in our training objectives.

These tests generally focus more on knowledge.

we can certainly link in skills if we're talking about the skill

of preparing oral rehydration

[INAUDIBLE]

solution, we can certainly ask people to list the

steps involved in correct order and the amounts involved.

But it's usually these tests do not show knowledge.

Now we have an example where there are possibilities of doing skill tests

one of our MPH students in Ibadan wanted to train teachers on giving

visual acuity test to their pupils.

And he actually provided the materials and asked the teachers

to demonstrate it or as you can imagine, prior to

the training, most of them just looked at the materials

on the table and didn't know what to do with them.

But at least there was a skill baseline

pre-test that could be used to compare with post-test.

And again, the post-test to be useful has to be the exact

same questions, exact same procedures that occurred at baseline and

pre-test that would be repeated again at the last session.

So just as you've scheduled time during the first

day or the registration for people to take the

pre-test, you also have to schedule time at the

end of the session for them to take the post-test.

One thing just to note for confidentiality,

you may want to assign participants a secret number that only they know.

And they put that secret number on their pre-test and post-test so that you

can compare them but not necessarily divulge whose results.

On the other hand, if, if the participants want their results back, they want

to compare them, then clearly you would want to put their names on it,

so that you could return the the tests to them.

So this has to be decided as part of the training committee plans.

The tests have various types of questions on them.

These can include open-ended questions, multiple choice

questions, and attitude, opinion statements, or items.

I need to balance the types of questions that we use

in our tests.

We want to cover the scope of the knowledge, but we want to make sure

these tests are simple enough that the participants

can complete them in 30 minutes or less.

As we mentioned before, you need to schedule time to do

these tests, and you don't want to eat into the process.

You don't want to make the trainees feel too nervous or worried about a

testing procedure, so you want to have

it relatively simple, straightforward, and consuming little time.

Open-ended questions are useful in terms of

people describing processes or giving their opinions.

it takes a little more effort to score these

kinds of questions, you have to develop a coding guide.

But it lets the trainees put down information in their own words.

Examples of open-ended questions could be, list all the ingredients that can be

mixed together for a homemade solution for oral rehydration

or describe the signs of, danger signs of dehydration.

Or mention the steps in order, for making homemade ORS.

Open-ended interviews have been very useful when you

have trainees who have a low literacy skills.

Where they can't, you can't give them a four or

five-page test and expect them to sit down and read it.

And fill it out, even if it's in a local language.

So, in situations like that, what we've done is arrange a

group of trainers to serve as, just like interviewers, and provide rooms

or space where the trainees can one by one by one go

and be interviewed with the open-ended

questions, and then their results recorded.

Multiple choice questions. You all

have experienced those before. They give you options.

And you obviously mark on the option which is correct.

Now, there are different ways of doing multiple choice.

You can have a situation where only one option is correct.

Or you could have a situation where many options are correct.

And, again, the scoring depends on how you do this.

If you agree in advance that only one option

will be correct and you have a ten-item multiple

choice questionnaire, then what we would have is

a situation where the maximum score would be 10.

There would only be one right answer for each question.

But if you have ten items on ten questions, and each

question has a choice of five items, as we see here.

what are all the, what are the ingredients that one can mix together.

We have five different ingredients.

Sugar, soft drink, salt, vegetables, water.

So they have to decide if each of these is right or wrong.

Almost like a true-false.

And if you had ten questions like this, with five options,

you would really have a score of 50 possible correct answers.

So clearly, you think about it, though look

at this yourself and think what would you score?

What's your right? What's your wrong?

What's your wrong?

Okay, the homemade solution that we've been

teaching people includes sugar, but not soft drinks.

Soft drinks would be bad for a child with diarrhea.

Salt is another ingredient.

Vegetables per se, solid food.

Continued feeding might be possible for oral rehydration,

but it wouldn't be part of a solution.

You can't put a zucchini or, or a potato in, in a solution easily.

And then the ingredient water, of course, that makes a solution.

So, sugar, salt, and water should be ticked.

And soft drink and vegetables would be left blank.

And that's how you would score it.

Another type of question that you can have

on your pre- and post-tests are attitude items.

If you're trying to encourage

people to value or have a positive attitude about

oral rehydration, then you could use some of these questions.

We've found, for example, that health workers were

uncomfortable at first when oral rehydration was instituted.

They wanted to prescribe drugs.

They didn't feel that were doing their job properly,

unless they were prescribing drugs for children with diarrhea.

And it took a lot of effort not just to

teach them the reasons for this, the importance of replacing electrolytes, the

fact that many diarrheals are caused by viruses that don't respond to antibiotics.

the issue of antidiarrheal drugs clogging up the

intestines and holding bacteria in or viruses in.

So, it was more than just teaching them those facts.

It was also, you know, letting them think about the fact that this was

a good, a valuable, a useful treatment and

that they would be doing the best for mothers.

So here's a simple attitude test with, with four items mentioned.

And you can obviously check or tick which one

you agree with or disagree with or are uncertain.

You can frame some of these in positive ways.

mothers can be trusted to mix ORS

at home.

And then a few items later you can reverse

it just to see if the people are being consistent

and say we can't trust mothers to mix ORS

or take care of their children with diarrhea at home.

So you can, can have that.

And then, of course, in your scoring process, if a question is phrased

in a positive way, agree might be three points, uncertain, two, and disagree, one.

But if it's framed in a negative way, agree may be one

point, uncertain two points, and disagree with B3.

So, you in terms of getting a final score, you have to be aware.

With any kinds of testing procedures, just as we talked about.

education materials, tests also need to be

pre-tested to see if people understand the questions

if the questions are easy to answer if

people are responding in ways that make sense.

[INAUDIBLE]

you have a question where 99% of the people agree

with it or disagree with it, it's not a very good

test question or it also shows that people you know, are

already having the attitudes that you're going to train them on.

So you want to make sure that you get a sample of

people to try out the test first, to see if it works.

Some examples of how we've used tests, include the patent medicine vendor

training in a rural community where I worked in Nigeria.

what we did, of course, was a we did a baseline survey

to find out what they were thinking and doing in their shops.

but we also had a pre-test, the, as we mentioned before, a training

committee was formed among the patent medicine

vendors, so that they could plan activities.

They, in addition

to the information that we gathered during our baseline survey,

that committee members requested certain lessons on certain illness and skills.

Like reading a prescription.

We had a situation where either the clerk in the shop or the shop owner

attended, usually it was not both because they didn't want to leave the shop vacant.

And then we did a pre/post test. We had a, in addition,

we had a control community where it was

about 25 miles away in another local government.

And so, we interviewed patent medicine vendors there at about the same

time we did the pre-test for the ones in our intervention area.

And then, did a post-test after the training in both places.

In terms of the pre-test, you can see from graph, that the trainees and

the control patent medicine vendors had very similar scores.

And then, afterwards, the post-test shows a significant increase in the

knowledge of the trainees, the people who went through the program.

And the value of this, as you've

probably learned from your various epidemiological courses is

that if we just have the trainees alone, a pre-test post test, we couldn't be sure

if it was really our training or maybe they enjoyed the experience.

Maybe they were upset by the initial questionnaire and were inhibited.

Maybe during the course of the training they went out

and were exposed to other activities that increased their knowledge.

So by having a control group, we can say that people under the similar

type of people, you know, whether they

experienced something in the environment that increase

their knowledge or not, or whether the changes that we're

observing at post-test can be really attributed to the training program.

One thing, another thing that was important, as they said

before, if you can identify the people who are in

the training program, and either give them some sort of

identi, ID number, so you can compare pre and post.

one thing that we found is that

the people who took the pre-test and

post-test were not exact, were not identical.

And this happens.

People don't show up for all the sessions. Some people join late.

So we found that 33 took the pre-test, 37 took the post-test.

but 28 of them took both tests.

So, in addition to looking at the gross results of the 33 pre-tests

versus the 37 post-tests as you saw in the graph in the previous slide.

We could also do what we call a paired t-test on the 28 simply subtracting the

[INAUDIBLE]

pre-test from the post-test and getting a mean for that and seeing if this

difference between the two scores was different than zero.

And we found that the average pre-test for this group was 46.

The post-test was 71 and by subtracting each individual's pre-test from post-test,

found that these

[INAUDIBLE]

these differences were significantly different than zero or zero

meaning no change, as we saw in the control group.

So, ideally if you, for yourselves, as, as trainers and evaluators,

you would want to see, did the training actually make a difference?

To the individuals, not just the whole group.

Because the whole group, you don't know

some may have lost knowledge for some reason.

Some may have

gained more than others what was going on.

So, this being able to pair up the results and look at it

gives you more confidence that the

improvement did occur with each, individual trainee.

Another patent medicine vendor training program was conducted

in Kenya by a USAID project called the Quality Assurance Project.

Their goal was to equip the patent medicine vendors

with customized job aids to communicate

new malaria guidelines to put drug outlets.

So there are a number of private drug shops around in the community.

the country had changed from chloroquine

to sulfadoxine-pyrimethamine because of drug resistance.

And they wanted to make sure that patent medicine

vendors who dispense quite a lot of these countries'

drugs understood this change.

So again, the idea was that if you

train the patent medicine vendors, they would be in

a good position to educate their customers on the

new drugs, malaria recognition, the importance of prompt treatment.

But also the trainees who would, in turn,

be able to train other patent medicine vendors.

So this was sort of an outreach type of program.

They were tested on specific areas of knowledge about malaria.

And as we can see in the chart on this

table there was an intervention group, there was a control group.

And interestingly enough can see that there are some

elements of knowledge that are, must be common knowledge in the community.

The very first thing, that fever is a symptom

of malaria and see that both groups knew that.

Majority, we ask majority knew that.

the issue of the new drug SP is also known commercially as Fansidar.

You can see the items about Fansidar, which was the focus of training and how to

use this new drug, were all higher for

the intervention group compared to the control group that

did not have this training.

So, again, not only do we have a gross score that

we can compare, but we can also look at specific items.

And we can look at that and say, okay, well, overall at the end of training

a number of these items, 95% for you

shouldn't sell drugs below the exist, the correct dose.

Again, recognizing the, the symptoms noting

that the Fansidar is a very effective drugs.

At the same time you can see that still,

about a quarter of them didn't really believe or

understand that Fansidar could be sold in their shops,

that you didn't have to get it at a pharmacy.

And we can see that 17% still didn't know that Fansidar,

a single dose, was enough for treatment. That it's packaged

that way so you just take it once.

So there were still some gaps but again, you

know, the, this overall improvement compared to the control

group, but it does tell you that you may

need to follow up with some of the trainees.

So by having knowledge specific results, you know item specific results, you can

see where the training was successful and where there are still some knowledge gaps.

Another thing that this Kenyan

program showed was that there are different categories of trainees.

They all may have been in one room together, but it was found that the

response was different by educational level.

And even though in all cases, as you can

see the darker green bar shows that the control

of the control group, I mean the

intervention group, did better than the control group.

You can also see differential among the intervention and the control respondents

in terms of their level of knowledge by their level of education.

Std 1-8 is like primary school.

Form 4 is like, sort of like junior high. people that have let's

see, then you can see that then senior high.

Also whats interesting is that the people with

high school education were more knowledgeable to begin with.

And so the intervention group, although they gained over

the control group, it wasn't that great a gain.

Whereas, those who had less education, only primary education,

they really gained a lot more by this program.

So we have to recognize that people of,

of different levels of education perform differently and

we have to make sure that our training

materials and exercises are geared toward all these differences.

That with the program is comprehensible by people who have only finished primary

school as easy as it is for people who have finished high school.

As I mentioned

earlier, one of our MPH students for his dissertation did a

program on training primary school teachers to test for visual acuity.

He was an optometrist himself when he came to the program.

And he was interested in preventative work community-based work.

He developed both a knowledge test as well as performance test.

The 17-point knowledge test that has

things like what structure of the eye is responsible for image formation?

What are three common causes of blindness?

Name two ways of detecting visual problems in student.

And just as we have done before, we did a pre-test, post-test with both intervention

schools teachers and intervention schools and teachers in control schools.

And as you can see here at baseline, the scores were on average four

out of 17. At

post test the the trainees

achieved almost over 13 points on average out of 17.

But you can see also that the the control group

did increase their knowledge by, on average, about two points.

And this is one of the things that happens in,

in the testing process by exposing the intervention teachers to the

pre-test itself. They thought about these questions.

They became curious.

They may have asked people around them, well,

I had this test today, what is this about?

And so, the testing process itself stimulated some knowledge acquisition.

But clearly it was not of the level that the people who experienced

a real training program.

In terms of training affecting attitudes, we have an

example from the CDT community-directed treatment in onchocerciasis control.

What was involved was training local government,

district health workers, to carry out the program.

We've talked about this before.

But it involves organizing community meetings explaining the program to

the community encouraging the community to

take responsibility for distributing it's own ivermectin.

Which involves volunteers who would be

trained that involves community collecting as

drugs, community collecting the records, statistics,

reporting the community managing side effects.

These types of things. And so, the program hands over

much of the responsibility for the day-to-day

delivery of the program to the community.

Which is why it's called community directive.

This idea of handing over some control to the community

was seen as somewhat threatening to some of the healtworkers.

They were concerned that the community may not do it correctly, that they may not, if

anything went wrong, they would be blamed. And so during prior to the intervention

of this program we had a series

of attitude statements with the health workers.

The health workers were trained to

organize the program in their local government.

And then

[INAUDIBLE]

we followed up.

Some of the statements with they could agree-disagree included

communities are quite capable of managing the distribution of ivermectin.

And in contrast onchocerciasis control should best be run by district.

So these are some things that we asked them in addition.

Community involvement in ivermectin distribution saves time for the health

workers who could be doing other things.

So, again, seeing a different value, seeing it in a positive light.

Health workers in this community cannot handle

ivermectin because they're distribution because they're too busy.

They'r overworked.

And then, again, health workers do

not believe that community directed distribution

of ivermectin is the best way to make it available to people.

In contrast to programs that have had it based

in health centers per se, or the health workers going

out and handing it out themselves.

So their response to these statements showed

whether they favored the program or not.

And as we can see in the chart, we have a pre-intervention survey,

where we have all of the health workers in the study districts combined.

And this was the distribution of their attitude scores.

it was reconfigured

to be on a scale of minus 25 to

25 whereas the positive side indicated a positive attitude.

So, in other words, the mid, midline was set at zero.

We see that, after the intervention, we have two types of intervention.

One, where we train the health workers simply to go out

into the villages and do their normal mobilization and helping the village

setup it's own program and decisions.

And then, we had an enhanced version where we,

in addition to the village meetings, we brought villagers

together at central places to have stakeholder meetings where

they could interact more freely with the health workers.

In both of these situations, the health workers improve their attitudes toward

this community-directed distribution. But those

who had the additional intervention of a stakeholder's meeting where there

was more interaction with the community developed a more positive attitude.

But, just going through the process, experiencing the, the

training, then going out in the field and applying it.

Interacting with the community.

They recognize that the community didn't have the resources and, and

the willingness to help carry out the program, and their attitudes improved.

So, here is an example of measuring that.

Another variable that we often look at, in

terms of training programs, is not just enhancing

people's skills but people's self-confidence or self-efficacy that

they can carry out a skill or a program.

And this is something we can clearly

measure before people leave the training site.

The actual application

of the skill we may not know until they get

back to the classroom, to the community, wherever they're working.

The visual acuity testing looked at this question of self-efficacy.

And as you can see, we asked the teachers

did they, how confident did they feel in four things.

Recognizing a child with visual problems, using the

visual acuity chart, interpreting the chart results, and

knowing when to make a referral.

And so, of course the options are

they're very capable, capable, unsure, not capable.

And they could check what they, what they felt.

And we did this again before and after.

One thing that was very interesting is that

prior to the intervention, the groups were very similar.

The intervention group, basically, doubled their perception

of self-confidence.

Whereas, interesting enough, the control group decreased.

So the, confrontation of the test, thinking about

these things, thinking about what's involved, actually made those

who did not receive the training feel less

secure in their ability to, to identify visual problems.

So, they recognize their limitations more after taking the test.

So we did see changes in, because

of exposure, we had changes in people's confidence.

And, again, exposure to the test itself,

especially the skill test made the teachers

uncomfortable and, and feel less about their

ability to, to do this without training.

In summary, we can see that pre-/post-test document are knowledge gains.

If we match the results, we have an identification number of the trainees, we

can match pre- and post-test results and

be really sure that individuals did gain knowledge.

And then using a control group definitely helps us know whether the

changes we've seen were due to the effect of the training program and not some

other intervening factor.

For example, the training that was done for

the teachers was not done all in one session.

It was done over a couple weeks, so could have been something going

on in the environment where teachers could have learned more about visual acuity.

having a control group makes sure that, if we see changes,

it's likely to have occurred because of the, the training program.

The issue of control groups

especially if your implementing a program that is ultimately designed for

all of the health workers If you do it in phases,

your control group could be people coming for the next round

of training, so that your not excluding people from the program, ultimately.

Also, in terms of testing, we're looking at not just the overall

summary scores to see if there was improvement, but looking at individual

test questions to identify areas of strength

or weakness in the training program itself.

And finally, we need to consider the

results in terms of the trainee's background.

Does their educational level, for example, affect

their scoring, affect their ability to absorb

the knowledge that's being provided in the training, and how we can accomplish that?

We've talked about tests.

In our next section, we will be looking at other methods of gaining information.

Lecture 9B: Pre- and Post-Tests

Training and Learning Programs for Volunteer Community Health Workers

与讲师见面