The need to share reproducible research results is centrally important to the research enterprise.
The Journal of Water Resources Planning and Management recently adopted a novel policy to explicitly incentivize reproducibility. It now offers free open access publishing to papers that have verified reproducible results – the first journal of the American Society of Civil Engineers (ASCE) to use such an approach.
“The incentive is really a key to get people motivated,” says David Rosenberg, associate editor of the peer-reviewed journal and professor of civil and environmental engineering at Utah State University.
The journal’s editorial board now encourages authors to add a reproducible results section immediately after the data availability statement section in their manuscripts. The board calls this action a “moral and ethical imperative” in science and engineering.
A new crew of associate editors for reproducibility has been named to handle the review process. In addition to content editors, the reproducibility editors review the code and data to be sure it can be understood by other scientists hoping to expand on the results. The hope is to get researchers to think about reproducibility early in their projects, so it’s not a heavy
lift or an afterthought.
“For a decade, we’ve talked about the importance of reproducible results, but it’s been a nebulous concept for a lot of scientists,” says James Stagge, assistant professor of civil engineering at the Ohio State University and associate editor for the journal. “The idea is to make it very concrete – and that makes it more attainable and more desirable from the researcher’s standpoint.”
Rosenberg says the approach was important to go beyond allowing readers to read article text and figures and let them dive into the research’s code, data, and models to potentially build upon it.
“This policy is really trying to invest in open access with papers where there is the largest potential benefit from people actually reading, working and using the research,” Rosenberg says. “They can do that because the authors have actually made their materials available.”
The reproducibility policy was about 1 ½ year in the making, requiring support of the existing editorial board and the funding to support publishing open access papers. In analyzing the landscape, Rosenberg and five other researchers assessed the availability and reproducibility of 360 articles published in six hydrology and water resources journals. They were only able to reproduce results for less than 2% of articles found. (See article for more details.). Because this practice is new, it will require some training and a culture change.
“This allows us to bridge the gap between research and practice – and it speeds up the process,” Rosenberg says. With materials readily accessible, academics, as well as practitioners (consultants or engineers who work in the public sector) go directly to data repositories rather than having to contact authors and wait for them to respond.
The need to rebuild the country’s crumbling infrastructure will take a large investment and an intensely collaborative effort. There is an urgency to ensure access to the best research – and quickly – as engineers respond to crises that affect public health, safety, or well-being, says Rosenberg.
“Being transparent about methods and results is really critical on challenging research topics,” says Stagge, citing controversy around climate science and medicine. “As our problems get larger, we’ve found the benefit of being able to rely on others’ work. It’s the democratization of research. If you publish reproducible research, you can actually ‘stand on the shoulders of giants’ and utilize some of that work so you can focus on answering really interesting questions.”
As the open practices catch on and new tools are available, open sharing is becoming part of the workflow.
“It makes life easier for researchers. When you put work in a repository, you have a permanent archive and you can always find it,” Rosenberg says. “It would be great to see new work that is sparked and catalyzed because researchers are making their materials available.”
Across the board in science, there is a low level of reproducibility and availability of work, but Rosenberg is committed to pushing for open practices.
“As practitioners of science, we uphold the principle that anyone can reproduce the results of the study,” Rosenberg says. “That’s what makes science, science. It’s not about the person. It’s about the method and the results.”
Response to the journal’s reproducibility policy so far has been positive and there have been eight submissions. The reproducibility policy comes on the heels of a data/model/code policy that the journal had previously adopted.
Dana Compton, managing director and publisher for ASCE, says that as she receives feedback on the new reproducibility policy, she will share the information with the editors and editorial boards of the society’s 34 other journals.
“What we tend to see is that, as one journal begins to see some successes with a new initiative, other editors and communities become more interested in and invested in these ideas,” says Compton. This was the case with the data availability policy, which was adopted by three journals before the publisher rolled it out to all ASCE journals in 2019. “I imagine this may take much the same path.”
Compton says the reproducibility policy has the potential to encourage researchers to think about next steps for their work and improve trust in science. “Reproducibility has become an issue that needs to be at the forefront of the minds of all publishers,” she said. “We want to provide an outlet for those researchers who value practices like data sharing and reproducibility. We are trying to provide options.”
To make it easier to find and track articles with reproducible results, the journal will feature these articles in a new special collection of articles with reproducible results. The editors have also announced that the journal will give two annual awards to authors of papers who make the best effort to make their result reproducible.
For more information, read policy details and guide for authors in an article recently co-authored by Rosenberg.