The system right now of disseminating knowledge and being open and transparent about your research results, well it might be considered not that open at all actually. because you need a a subscription to a journal. >> That's a problem, yeah. >> A journal has only limited amount of words and space for you to >> Yeah. >> Tell others, about your research results. >> Yeah. >> and, of course, it's very important to publish, it's very important to get cited. >> Yeah. >> So the whole system is sort of I'm focused on, publishing, publishing a lot, and publishing positive results. >> Yeah. >> Right? >> There is different aspects to the story. Now we can not have all the data around, because we need to have some sort of categorization, some sort of filter, in order to make sense of it all. So this is why we study papers But normally we write, scientific articles, which are then peer reviewed, by peers, other researchers- >> And counted as, more important. >> Yeah [CROSSTALK] Nah, and this is also because we, if you submit an article, research article for a publication in scientific, journal then these journals they, they perform peer reviews, that have independent reviewers, normally, people who are knowledgeable of the fields, to judge whether what you did is any sound, methodologically, but also theoretically, or other, the conclusions are really sufficiently supported. So you have this check this balance in place which you already convinced at least the reviewers of being right on what you trade to show, now there's different problems here because scientist and any other system that cost a lot of money. It's a very expensive business, but we need to decide who's going to get the money. >> Yeah. >> And how do you do that? And there you, you also decide what papers to publish. Now you have the system, you have, have researchers doing their work. They're left alone, in doing whatever they feel is interesting. And then, you, they can do a study. And, and then it has to be, submitted to a journal, it has to be published. And those at the journal, which are publishers, is, em, well, they want to earn something as well because it costs money also to publish, and those, editors want to know which are the in, interesting and most important findings. >> Right. >> And there, you, the money comes in. And then the money comes in, because, there's more money to be made as a publisher, with a very novel, very interesting, very exciting new finding. These are the things that, that make, publishers rich, because this is what people read. This is what people, readers want to pay for. And this is what gets media attention or not, so this is good for business. >> Citations. >> And then for the scientists themselves, it will, it will garner more citations. Other people's, if it's good, if it's an interesting idea, other researchers will follow up on it, and this is a very important thing. Because then, after all it's not this only saw. Little finding, but it, it, it builds knowledge, in a sense that it's gives way to more research so and that's evidence if, if people refer to my paper, if I publish the paper. This is, it's, it's good indicator at least of, of the value at least in the sense that, people do more research on it probably and refer to your particular paper. >> So it must be good? >> Now the problem here, is that if we all do novel work, we cannot do all do novel work all the time. >> No. >> You can't, you also have to do some checks. If someone finds something, and I think mm, that's weird. It's, it's published there's it's in a peer reviewed paper that's fine. But there's the method section. And maybe I'm, I'm very skeptical of that result and I maybe feel inclined to replicate it to do it all over again, using the method section there maybe contacting the authors if necessary to get the materials or not, some materials that are not there and then we do it. And now let's suppose I don't find what the other researcher, did, what now? Now I can then go to this journal, lets say the same journal, the high rated journal. >> For example, yeah. >> But only, that only accepts 10% of the submissions that are sent there, the ten, top ten most important results. >> Yeah. >> And I will comment the journal and say, well, that's a, I didn't find it. And then the journal says, yeah well, that's your problem, that, that's just submitting a to a lower rank journal. >> Go find me an interesting innovative new result. >> I want new, having something new this is good for business. The editor may feel, well, the owner probably stays where I wo, woo where I, would we want to second guess this particular researcher's result? And there you have it. Then in the grant system as well so if you want to get a grant to do research, and if you only do research, trying to check other people's work, the grant organization will say, well, this is pretty secondhand research. There's no value to society [CROSSTALK] what's the value? Yeah, it's not directly, it doesn't immediately show that there's, this doesn't have to be, ideally will build new knowledge. It basically breaks down a little piece of what we thought was right and now comes the big problem. Journals, and these editors, and the publishers, and the reviewers who act as safe guards in deciding, what to publish, they dislike, and this is across many scientific fields. They dislike, null findings, negative findings, or checks of earlier findings.They dislike them, they don't publish them. >> Even if you say confirmed earlier findings? >> Even then, it, it may be more easy to get that published, especially if you, build on it. So you do check experiment you find the same result and do something a variant thereof, something extra. >> yeah. and that's pretty basically fine but let's just suppose for the sake of the argument that this big new finding is, is simple fluke. It's a it's statistical accident. >> Just an accident. >> It can happen. >> Yeah. >> It happens f at least 5% of the time this is what happens, it can be, a chance finding. Now, all these researchers in the world are very excited about it. They all start trying to replicated it, and no one can replicate it. And now the problem is, do they want to then, write it down, write this paper, and send it to journal. If they feel that the journal is not very likely to accept this paper, the control experiment failed replication for publication. If they don't do that, if they feel that it has very little chance, to be rewarded, in the publication system. It has very little, to add to one's resume, because one's resume as a scientist, whether you do, submit grants, or you try to get tenure fixed position, or you want to become a professor or not, it all depends on your resume. >> It depends [CROSSTALK] on your number, of publications. >> Basically, your number of published articles. >> That's, that's still mostly the case and so, left, the original finding, it was very exciting. Ten labs in the world try to replicate it. Maybe one accidentally stumbled upon the same effect. It's just, there's chance involved. There some noise there. For many, many reasons. And the nine others don't. Now there's nine won't submit it for publication, because they don't feel that it has a chance to become published. Why would they invest their [CROSSTALK] precious time to do it? >> As the file drawer problem, right? >> This is the file drawer problem. So the nine would then put it in a file drawer. The one that does. To find this, some corroboration of the original that will submit. The journal will happily accept it because, yeah, after all, we all have this, exciting find. Now we have some independent replication of this finding. And novel, well, corroboration in another sample, another study. Wow [CROSSTALK] Yeah, then we are a year further along the road. We have the original finding of one sole publication that corroborates it and the other nine, they are just gone, and this is a big problem. And this is called publication bias it's called bias because it biases the facts, to a very severe degree. the, what we read in literature, because we, if you don't look at the literature and you go so to Google scholar, and find this particular exciting effect, you'll find the two studies and it may feel well. Two, two, two studies are better than one, so it must be- >> Yeah.
