This is also related to something that's been called the decline effect.

And this is the subject of repopularized in her New Yorker article a few years ago,

which is a really wonderful article.

And the idea is that film effects that appeared very large in initial papers

subsequently decrease in effect size when they are resampling.

And the lead example here is an example of a drugs that seem to work initially and

they're test over the years, they get smaller and smaller.

The effect sizes get smaller and smaller.

And there many reasons going to happen, of course, a drugs including

changing standards for how diseases are diagnosed and drugs are applied.

But a big part of this decline effect is likely just simply regression to the mean.

So anytime you have a false positive finding that you happen to find published,

and then people start trying to replicate it.

If it's not a true finding, or if those findings have been picked out of a larger

family of experiments that happen to work the best, then when it's replicated

the effects will get smaller and smaller, and it will appear to decline.

So not so mysterious, but important.

And then we'll talk about circularity.

And this is a way of thinking about brain analyses.

Especially and the kinds of analysis that lead to bias effects and

how we avoid them.

So circularity is also called double-dipping, in which was popularized

by an article a few years ago by Nikolaus Kriegeskorte and colleagues.

And the idea of circularity is that you can select voxels to look at

based on one effect or test, and then test those voxels on something

that's not independent of that selection criterion.

So this is sort of pernicious, and so we have

to really think through our results and analysis and be careful to avoid it.

But here's an example, a work through example.

So what you see here is a panel of voxels and this is a study with four conditions,

A, B, C and D.

So now we've got the blue area here, which shows some true effects and

the true effect is A & B activate, C & D don't.

So now we're going to select data, there's the truth, or

we're going to select data based on this contrast A versus D.

So we selecting on A versus D and

we're picking up voxels, that show those A versus D effect.

Now what's going to happen is,

any voxels I test later are going to tend to show noise that favors A versus D.

So then, if I tested independent data, I would get about the right answer.

A and B activate, the other ones don't.

But, if I use that same data where the noise is favoring the A versus D

hypothesis, then I'm going to get a biased effect.

A is going to be greater than B, on average.

Just by chance.

So, how can I select on A versus D and get an A versus B effect?

Well I'm conditioning on noise values that tend to be high for

A, and that's not true for B, so I'm creating a bias.

And that's one of the, that's essentially the circularity problem in a nutshell, and

that's one of the big dangers in terms of selecting ROIs and then testing them.

We have to make sure that the data are independent.

And one popular way of selecting regions of interest is through contrasts that

are orthogonal to the tests of interest.

And so nominally you might that that avoids the circularity problem.

So I might select on a main effect of A B versus C D and then test on A minus B.

And that seems on the surface, pretty okay.

It's safer than truly non independent tests but

there can still be bias if you test on the same data.

Why?

Because the design matrix, the regressors for A,

B, C, and D can be correlated and that can produce effects, and

also the noise characteristics might be auto correlated so

the noise characteristics might also create a selection bias.

So we do have to be careful even when we're applying orthogonal contrasts.

It's better to test on independent data.

This basic circularity phenomenon is one of the other issues that was raised in

the voodoo correlations debate.

And the idea is non-independent tests inflate this brain behavior

correlation estimates.

So this is a histogram from the literature of the correlation value for

reported correlations in your brain and behavior from the literature.

And as you can see it ranges from about .2 to correlations close to 1,

which seem like fair large effect sizes.

And what the authors here did, is they broke it down into those in which they

thought that the test were independent of a voxel selection of criteria and

those that were not independent.

And so you see independent in green and non-independent in red.

And that's one estimate of what the inflation of the apparent

effect size due to the circularity or non-independent testing might be.

So here are some solutions.

One solution that were going to advocate

quite a bit later in the course is data splitting.

Hold out independent test data if you actually want to estimate effects sizes.

And we should want to estimate effects sizes.

So this means perhaps holding out a sub section of participants for

a later exact test of the findings that you report.

And also maybe holding out runs

if you're interested in making inferences within an individual person.

So, with single hypothesis test of the model.

If you develop a model or

a pattern across regions that you can integrate into a single test.

Then if you're just doing one test on new independent data

then you can get an unbiased estimate of how big that effect is.

We'll look at an example of this later.

And this principle goes beyond voxel selection to encompass

all kinds of model-building.

Whether your designing a fancy connectivity or dynamic causal model or

a predictive model, which includes multimodal data, or anything else really,

the same principles hold.

One really effective strategy, we'll talk about later is called cross-validation.

And what it is, it's an efficient data splitting strategy

in which model development is done on one set of data, training data.

And then testing is done on another subset of the data, systematically.

And we'll talk about that

when we talk about machine learning later in the course.

So let's talk a little bit more about selection bias in a more concrete way.

And let's look at how some forms of bias can combine with voxel

selection bias to multiplicitably increase false positives.

So for example, if I test two contrast maps and

I've gotten voxelselection bias in each contrast map,

I get twice the false positives and the corresponding increase in effect size.

If I do two experiments, I get twice the false positives.

This doesn't mean that we should correct for

multiple comparisons across every test that we've done.

But what it means is we need to be mindful of this when we interpret the effects.

So let's look at our four levels of bias, publication, experiment,

model selection, and voxels and tests, and how they play out in neuroimaging.

So this is a illustration of the file drawer problem.

And it's not from neuroimaging but I think it's quite illustrative.

And what we're looking at is studies of antidepressants that

have been submitted now to the British Medical Counsel.

And, across the five drugs, we're looking at

the studies that have been published in the blue lines.

And the y axis shows the effects size of the antidepressant.

And so those are pretty high when we look at only the published studies.

But the nice thing about these drug studies is that there's a national

registry where they have to have all the data submitted.

So they can go back to the unpublished data as well and

look at all the studies that have been submitted to the registry.

And we see those in red.

And what you see here is across the drugs The effect sizes in

all of the studies are substantially lower than they are in the published studies.

And that's an example of the problem in action.

So here's the next section which will look at

flexibility in experiments and in the model.

And this flexibility in choosing which experiments and which models and

which outcomes that you want to look at after you've observed some effects.

And trying to optimize the chances of getting nice looking effects from

publication has been referred to as p-hacking in the literature.

And there are tests for p-hacking now, that people are interested in doing.

So this is one influential paper where they discuss this.

And I think this is the paper where they coin the term.

So Simmons Nelson and Simonsen.

And they point out that researchers have millions of decisions to make.

Whether to collect more data, which outliers to exclude,

which measures to analyze, which covariance to use.

And the newer imaging which types to pre-processing and modeling in correction.

And the idea of p-hacking is,

that you make the analysis decisions as the data are being analyzed.

And then you want to create findings in publish

that would create a bias just like voxel selection bias.

Some of the red flags for p-hacking are the use of median splits in the data,

high and low responders when it's actually a continuous distribution.

Why not use the continuous scores?

Exclusion of many data points without really good

principle reasons for doing so.

Unconventional analysis choices or internal inconsistent analysis choices.

So maybe the researchers looked through lots of different possibilities and

they just picked the thing that worked the best.

P-values close to the threshold of 0.05

is one of the red flags that they pointed out.

Not everybody's P-value can be just under 0.05.

That's a flag for some effect that really wasn't significant and

then you're trying to get it to be more significant.

And then finally, unusual numbers of subjects without explanation.

And so journals have started to really flag those issues.

This is a view of the preprocessing pipeline.

And this is a paper by Josh Carp from a few years ago.

And what he did is he analysed the same data set many different ways.

In fact, he analysed them 34,000 different ways.

Just by picking different analysis steps and several strategies for

each analysis step.

So that he has got almost 7,000 unique analysis pipelines,

five multiple comparisons correction strategies, leading to 34,000 maps.

And so this is the mean activation across all of the analysis maps,

and also the range.

So the point is that there's a lot of variability

according to these analysis pipeline choices.

Some of them are better than others.

But the solution here for us is to really be principled and

consistent in your analysis choices.

It's not bad to have a good pipeline or

to change things about your pipeline to make it better.

But we should really make those choices in advance of looking at the results

as much as possible.

So here finally are some do's and don'ts in terms of

what we should do to avoid selection bias problems.

So don't is thoughtless analysis.

I don't want to give you the idea that you should just choose everything and

advance in one analysis and then be done and not look at your data.

It's really important to explore the data,

to examine the data, examine the assumptions.

The point is to get the right answer.

Not just to get the answer that we wanted in the first place.

And getting the right answer really requires looking at the data and

making some smart choices.

And sometimes we do have to change what we do based on what the data

actually looked like.

But we should do this in a principled way.

Don't do uncorrected exploratory analysis with strong conclusions.

So we can do those analyses but don't go to town and sell the story and

yourself on a finding that comes from an uncorrected exploratory analysis.

Do really work in advance to choose principled a prior hypothesis.

We talked about using net analysis to do that and that's in the next module.

And tried to conduct adequately powered studies which often requires a lot of

investment and resources.

Don't do circularity analysis and do,

do testing on independent data or data splitting to estimate effect sizes.

Don't do P-hacking, and do make principle choices and

use standardized, reusable pipelines with sensible a priori choices.

And finally, don't make heuristic reverse inferences.

We see the insula activated, that must be disgust, we see the hippocampus activated,

that must be memory.

There's a lot of various inferences that can come from that.

But do learn the techniques to make quantitative reverse inferences and

really use those to understand what your brain maps are telling you.

So that's the end of this module, thanks for listening.

Module 2B: Pitfalls and Biases in Interpretation: Circularity, Reverse inference, Voodoo, and Beyond

Principles of fMRI 2

与讲师见面