Problem

KI President Ottersen describes the problem concisely;

We are in the midst of what some have called a research reproducibility crisis. While scientific discovery and complexity are developing at an unprecedented speed, less than 50% of scientific research studies can be reliably replicated. Left unchecked, this troubling fact may threaten our ability to generate sound, evidence-based knowledge that meets society’s needs. It is time to look beyond the traditional measures of quality and re-examine the very concept of quality itself.

Recently highlighted by Hanne Kjöller, previously by Lorena Barba, National Academies of Sciences (USA)

Illustration: In school one may not only give the answer to a problem - "show your work!". In research it is ofter not possible to see or reproduce how the answer was derived or constructed, why is it so?

Solution

Johan Jansson at KTH has driven the development of digitalization in the form of Open Source ("Digital Math" with FEniCS as core component) as a solution to the reproducibility crisis since 2003, the framework has become a de-facto world standard for advanced mathematical modeling. Johan leads the largest online course DigiMat Pro at KTH based on this with 30000+ participants.

In the US there are now guidelines for requiring the publication of the "digital objects" (Open Source), in the US National Academies of Sciences, Engineering and Medicine. A leader of the development has been Lorena Barba, Professor at George Washington University.

Lorena Barba says in Physics World:

What we are calling for is changing those norms to give importance to the full set of digital objects that are part of a scientific study and acknowledging that the scientific paper is insufficient today in its methods section to include all of the information needed for another researcher to confirm the results or build from those results.

The technology exists to achieve this, there have been technical solutions since the 80s and 90s.

Program for the Panel Debate: A new perspective on reproducible research in the digital era

Date: Monday 14 December 14:00-15:00 CET
Location: Webinar via Zoom (the seminar will be held in english)

Participants

• Hanne Kjöller, journalist, gives a short presentation of her book
  
• Lorena Barba, Professor George Washington University, background about US digitalization guidelines and experiences
  
• Johan Jansson, Docent, KTH, Director Center Digital Math, about digital methods for solving the problem
  
• Lina Nordquist, Member of Swedish Parliament (L)
  
• Marie-Louise Hänel Sandström, Member of Swedish Parliament (M)
  
• Ilona Szatmari Waldau, Member of Swedish Parliament (V)
  

Debate led by moderator Johan Jansson.

Agenda

1. (5 minutes) Johan Jansson welcomes to the debate and says just a few sentences about the purpose of the debate, with proposed digital Open Source solutions, with one perspective being Digital Math, with specific mentions of FEniCS and Jupter as frameworks, for the problem.
2. (5 minutes) Hanne Kjöller gives a short presentation of her book on the problem, which put this on the Swedish agenda, and her perspective.
3. (5 minutes) Lorena Barba gives a background about US digitalization guidelines and experiences on reproducibility, where she has had a leading role.
4. (35 minutes) Round-table debate, round-robin statement and perspective, starting with the Members of Parliament. Johan Jansson has prepared questions and background material to help the discussion, see below.
5. (10 minutes) Questions and discussions with the audience.

Statements

A short summary of Lorena Barba's position and statement is this:

The widespread use of computation and large volumes of data have transformed most disciplines of science and enabled new and important discoveries. But this revolution is not yet reflected in the ways that scientific findings are published and shared with the relevant communities. Extending the scholarly record to data, software, and computational environments and workflows is a must to ensure the robustness of science in this digital era. Even if current practices in this regard are insufficient, the NASEM report conveys a reassuring message of ‘no crisis, no complacency.’ The crisis narrative that has unfolded in recent years is unhelpful, even if improvements are needed. A science policy trend for more transparency is surely underway, while more work needs to be done in regards to infrastructure, and training of researchers on computational methods, statistics, and open science practices.

A short summary of Johan Jansson's position and statement is this:

What I'm proposing is that _all computations_ in _all papers_ (simulations, statistical analysis, etc.) should be available in Digital Math form as part of the paper together will necessary raw data, so the reader can test, edit, try to falsify, test for sensitivity of parameters, methods, etc. This includes experimental papers, where the computational/statistical part should be available in this way.

Proposed questions for the discussion

The EU report (in the references) serves as a good background for the discussion, below are some relevant excerpts:

One key driving question for the discussion can be: What is done, and what can be done in Sweden to drive the development forward, to realize or evaluate the proposals which are given in the EU report, and to collaborate with the EU Commission?

Excerpts from the EU report:

Second, there is a perceived deliberateness, or at least carelessness, in scientific production due to competitive pressures. A growing proportion of scientists are perceived as – willingly or unwittingly – bending some of the basic premises of the scientific method to produce ‘fast science’ or even ‘make believe science’ – facts and theories that are declared true but are dubious or even false. This rests more on the structure of incentives of science-making, embedded in culture and practice, than on deliberate attempts to ‘cheat’. The need for results to be reproducible, and the tangible steps needed to make them so, may help results be trustworthy and keep scientists honest.

Possible remedies:
[...]
Sharing of data, protocols, materials, software, codes, and other tools underlying publications; Transparency of analysis and modelling;

Possible actions:
[...]
17. Fund the testing and R&I development of automatic systems of compliance for reproducibility before publication;
[...]
24. Ensure that Horizon Europe provisions encourage and support
reproducibility (see list of possible actions, above);
25. Employ and police guidelines early in the grant application phase to anchor journal practices;

etc.

Activities

• A short summary/abstract of each participant's perspective and constribution will be published here in good time before the debate.
• Join the associated online course! Easily accessible "Digital Math" online course on reproducible research which everyone globally can participate in is active before, during and after the debate.

Organizers

• Johan Jansson (KTH)
  
• Robert Lagerström (KTH)
  
• KTH Reproducibility Workgroup

Registration

Please register for the debate below!

Your email:


(Optional) Comment:

Send

Swedish description

Datum: måndag 14 december 14:00-15:00 CET
Plats: Webinar via Zoom (seminariet hålls på engelska) http://digimat.tech/paneldebate-kth

Problem

KIs Rektor Ottersen beskriver koncist problematiken

We are in the midst of what some have called a research reproducibility crisis. While scientific discovery and complexity are developing at an unprecedented speed, less than 50% of scientific research studies can be reliably replicated. Left unchecked, this troubling fact may threaten our ability to generate sound, evidence-based knowledge that meets society’s needs. It is time to look beyond the traditional measures of quality and re-examine the very concept of quality itself.

Nyligen belyst av Hanne Kjöller, tidigare av Lorena Barba, National Academies of Sciences (USA)

Illustration: I skolan får man inte bara ge svar på en uppgift - "show your work!" I forskning är det ofta inte möjligt att se, reproducera hur man kom fram till svaret, varför är det så?

Lösning

Johan Jansson på KTH har drivit digitalisering i form av Open Source ("digital matematik") som en lösning på reproducerbarhetsproblemet sedan 2003 på KTH och Chalmers, ramverket har blivit en de-facto världsstandard för avancerade matematiska beräkningar, Johan leder KTHs största onlineutbildning baserat på detta med 30000+ deltagare.

I USA finns riktlinjer för att kräva publicering i Open Source, i US National Academies of Sciences, Engineering and Medicine. Drivande i arbetet har varit Lorena Barba, professor vid George Washington university.

Lorena Barba säger i Physics World:

What we are calling for is changing those norms to give importance to the full set of digital objects that are part of a scientific study and acknowledging that the scientific paper is insufficient today in its methods section to include all of the information needed for another researcher to confirm the results or build from those results.

Tekniken finns för att göra detta, det har funnits tekniska lösningar sedan 80-talet och 90-talet.

Upplägg Paneldebatt: A new perspective on reproducible research in the digital era

Datum: mån 14 dec kl. 14-15
Plats: Webinar via Zoom (seminariet hålls på engelska)

Program:
Hanne Kjöller, ger en kort presentation av sin bok
Lorena Barba, förinspelat material om USAs riktlinjer och erfarenheter
Johan Jansson, docent, KTH, föreståndare Center för digital matematik, om digitala metoder att lösa problemet
Lina Nordquist, riksdagsledamot (L)
Marie-Louise Hänel Sandström, riksdagsledamot (M)
Ilona Szatmari Waldau, riksdagsledamot (V)
Debatt under ledning av moderator Johan Jansson.
En kort sammanfattning av varje deltagares perspektiv och bidrag publiceras i god tid innan debatten.
En lättillgänglig onlinekurs i "Digital Math" reproducerbar forskning som alla kan delta i kommer vara aktiv före, under, och efter debatten.

Organisatörer:
Johan Jansson
Robert Lagerström
KTH Reproducibility Workgroup

Registration

Please register for the debate below!

Your email:


(Optional) Comment:

Send

References

1. "Science needs to improve the transparency of research results, says report", PhysicsWorld, 2019
2. "Reproducibility and Replicability in Science", Lorena Barba, et. al., US National Academies of Sciences, Engineering and Medicine, 2019
3. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8). (9000+ citations in 2020)
4. ”Kris i forskningsfrågan” , Hanne Kjöller, Fri Tanke, 2020.
5. "Reproducibility of scientific results in the EU", Directorate-General for Research and Innovation (European Commission), 2020
6. "Redefining research quality: Reproducibility and beyond", Ottersen, O. P., 2019
7. Events in the Macchiarini case, Karolinska Institutet, 2020
8. Digital Math, Johan Jansson, et. al., 2020