By: Brian Nosek, Center for Open Science
In the Fall of 2011, Sarah Mackenzie, the maid of honor at my wedding, was diagnosed with a rare form of ovarian cancer. Sarah and her family were motivated to learn as much as they could about the disease to advocate for her care. They weren’t scientists, but they started searching the literature for relevant articles. One evening, Sarah called us, angry. Every time she found an article that might be relevant to understanding her disease, she ran into a paywall requiring $15-$40 to access it. Public money had paid for the research, yet she was barred from making any use of it. Luckily, she had us. Most people in Sarah’s position don’t have the luxury of friends at wealthy academic institutions with subscriptions to the literature.
During this time, I was pursuing an interest in the business models of scholarly communication. I wanted to understand the ways in which these models interfered with the dissemination of knowledge that could improve quality of life. Sarah’s experience illustrated one key part of the problem–the outcomes of research should be public goods, but the business models of publishing make them exclusive goods. Lack of access to published literature limits our ability to apply what we know to improving others’ quality of life. If doctors can’t access the literature, they can’t keep up with the latest innovations for care. If policy makers can’t access the literature, they can’t create evidence based policies. To advance solutions and cures, the outcomes of research must be open.
During the next few years, Sarah went through the standard treatments with mixed success. She needed new solutions and was fortunate to qualify for a couple of clinical trials testing new therapies. They were difficult. Sometimes the side effects would have her in the hospital for weeks at a time, or at home but unable to interact with her young boys.
Simultaneously, I was learning a lot about publication bias. For example, despite a requirement that results be reported within 24 months of the end of a trial, less than 50% of trials were so reported. Trials were funded. Patients were put through treatments often with terrible side effects. And, by failing to report the study results there is zero return on investment and needless suffering for vulnerable people.
Moreover, evidence shows that there is bias in which results are reported. Positive results are more exciting and therefore more publishable, negative results are boring or disappointing and more likely to be left out. But, negative data is immensely valuable for getting to important solutions faster and for making evidence-based decisions about what not to invest in next. Failing to share negative data fuels tragic cycles of dead-end research investments. To advance solutions and cures, the data from research must be open.
The pressures for finding positive results produce additional problems for the content and credibility of research that does get reported. Outcome switching is rampant. In other words, what researchers observe is often different than what they report. Evidence suggests that rampant selection bias, often occurring unintentionally, produces reported findings that are much rosier than what researchers actually observed. For the doctors, policymakers, child care providers, elder care providers, educators, members of national security apparatus and so many others who depend on published research to inform their choices and strategies, selective reporting results in false hope and ineffective action. To advance the human condition, the process of generating, analyzing, and reporting the data must be open.
Sarah, and many many others like her, can’t access the literature that their tax dollars funded, are being used for research that never gets reported, and are being misled by a literature that looks more beautiful than the actual evidence suggests. This is unethical, both in terms of the use of research dollars and the sacrifices experimental patients like Sarah make to test possible treatments.
Sarah died two days before we hosted the Society for the Improvement of Psychological Science (SIPS) meeting at COS this past July. At the meeting, we discussed the negative impacts of inaccessible outcomes, unavailable data, and doctored results in advancing science in general, and I couldn’t help but think of that impact in very concrete, personal terms. As a society, we didn’t do enough to transform science’s creativity, rigor, resources, and talent into the greatest possible benefit for people like Sarah. There are thousands of others like Sarah who are waiting for science to help them. And, there are going to be thousands more next year, and the next, and the next. Closed science is an impediment to progress.
We can change this. To improve quality of life, advance more effective cures, and more efficient solutions, we must build the means and create the incentives for scientists to appreciate and embrace two “simple” objectives:
Show your work and share your work.
Really and truly, that’s it. If we can open the marketplace of ideas to the actual evidence and how that evidence was acquired, scientists are smart, they will figure out what is reliable and what needs to be studied next. Open science is the key to reducing waste, accelerating meaningful solutions to the biggest problems faced by our communities, states, nations, business and civic institutions, and to save the lives of millions of people around the world including people like my friend Sarah who desperately need science to survive.
We can improve decisions and quality of life by making the content of research and the process of producing that content transparent, accessible, and reusable.
Researchers produce lots of research. What proportion of it is open? Let’s compare 1999 Brian with 2017 Brian. In 1999, I might have produced 25 studies. At the time, none of the materials, none of the data, and none of the code was made publicly accessible to others. One reason for this lack of access was that the Internet was not what it is today–technology makes a difference. But technological change is not enough. Achieving our key outcome requires a strategy that integrates new technological opportunities with a deep understanding of the academic cultures and expectations that work against open research.
1999 Brian was a product of that culture. I didn’t write down my predictions in advance. I just hoped for a positive result that I could publish. If I got a negative result, I was encouraged to toss it aside and try again. I had no incentive to record my predictions or save my “failed” trials. I have a few papers of mine from that time period. I don’t really know how many studies I ran. I don’t know where the data or materials are — or even if they still exist. I don’t remember why I picked some studies to write up for publication and not others. Did I leave out some inconvenient results? Did I p-hack? I did the work, and I don’t know that. No one knows. And, there is no way to find out. This reduces the credibility of the research that was published and is a waste of all the research that was lost.
In 2017, I will probably produce about the same total of 25 studies (now “I” is generous because others do most of the work). All 25 are on the OSF. My collaborators preregister the design and upload the materials in advance. After data collection, anonymized datasets and analytic code are added. All of the content and key process elements of my research are preserved and accessible. If I leave out studies from my reports, that is discoverable. If I p-hack my results, that is discoverable. If someone wants to reproduce my findings, they can. If someone wants to build on my work, they can. If I forget 16 years from now what work I did and how I did it, it doesn’t matter. It is there. Some parts of the workflow, e.g., email correspondence about ideas, is not yet integrated into OSF, but most of the research is preserved and accessible.
The immediate ROI of my 2017 research will exceed the ROI of my 1999 research because it is done more rigorously and more transparently. Moreover, the long-term ROI of my 2017 research will vastly exceed by 1999 research because it can be reused, reanalyzed, and extended by anyone and for purposes beyond what I envisioned. My 1999 research is gone and can provide no additional benefit.
Every part of the COS strategy moves researchers toward 100% open. That might start as simple as sharing a poster from a conference on OSF, a poster that would otherwise disappear within weeks. It might be as easy as sharing a paper on a preprint service, accelerating the date that others can become aware of the work by two years or more, and ensuring that it is openly available forever, even if it is never published in a journal. These little steps are ways to inculcate a change of mindset — a mindset that making our work available to others is a good thing. With technology, incentives, and training, those initial steps can accumulate into a new ethic of openness that influences planning, execution, and reporting of all research.
This article originally appeared on the Center for Open Science’s blog. Read the full article.
ECS is moving toward a future that embraces open science to future advance research in our field. Programs established by the Society that support that shift include the upcoming launch of ECSarXiv, a preprint server through a partnership with the Center for Open Science.
