Research publication can both communicate and miscommunicate. Unless research is adequately reported, the time and resources invested in the conduct of research is wasted. Reporting guidelines such as CONSORT, STARD, PRISMA, and ARRIVE aim to improve the quality of research reports, but all are much less adopted and adhered to than they should be. Adequate reports of research should clearly describe which questions were addressed and why, what was done, what was shown, and what the findings mean. However, substantial failures occur in each of these elements. For example, studies of published trial reports showed that the poor description of interventions meant that 40–89% were non-replicable; comparisons of protocols with publications showed that most studies had at least one primary outcome changed, introduced, or omitted; and investigators of new trials rarely set their findings in the context of a systematic review, and cited a very small and biased selection of previous relevant trials. Although best documented in reports of controlled trials, inadequate reporting occurs in all types of studies—animal and other preclinical studies, diagnostic studies, epidemiological studies, clinical prediction research, surveys, and qualitative studies. In this report, and in the Series more generally, we point to a waste at all stages in medical research. Although a more nuanced understanding of the complex systems involved in the conduct, writing, and publication of research is desirable, some immediate action can be taken to improve the reporting of research. Evidence for some recommendations is clear: change the current system of research rewards and regulations to encourage better and more complete reporting, and fund the development and maintenance of infrastructure to support better reporting, linkage, and archiving of all elements of research. However, the high amount of waste also warrants future investment in the monitoring of and research into reporting of research, and active implementation of the findings to ensure that research reports better address the needs of the range of research users.
Introduction
In 2006, Lang and Secic1 warned that “The problem of poor research documentation and statistical reporting in the biomedical literature is long-standing, worldwide, pervasive, potentially serious, and not at all apparent to many readers”. Bradford Hill2 suggested that reports of research should answer four questions: what questions were addressed and why, what was done (the materials and methods), what was shown (direction, size, and uncertainty of effects), and what the findings mean (in the context of other research). Answers should be readable, complete, and make allowances for different audiences. However, most research reporting falls far short of these ideals. Carp3 noted that the methods in 241 reports of functional MRI studies often did not have sufficient detail needed for replication, deficits were common in the reporting of parameters used, data acquisition methods, and preprocessing and analysis methods. More than a third of studies did not describe the number of examinations, examination duration, and the range and distribution of intertrial intervals, and less than half reported the resolution, coverage, and slice order of images. These deficits make interpretation risky, and replication—an essential element of scientific progress—nearly impossible. Such problems occur in all types of research, as documented in a series in the journal Nature about the challenges of irreproducible research. The need for replicability underpins Nature's new requirement for inclusion of relevant details about several elements of experimental and analytical design.
Although concern about research fraud and misconduct is appropriate (a pooled estimate of 18 surveys showed that 2% of scientists admitted to having fabricated, falsified, or modified data or results at least once4), these issues represent the tip of a much larger iceberg of misrepresentation and waste in research that we have attempted to document and address.5 Findings from a 2009 article suggested that at least 50% of research reports were sufficiently poor or incompletely as to make them unusable, which represented a waste of tens of billions of pounds.5 In this Series paper, we extend that analysis with a more detailed breakdown of the problems in reporting of research. Although publication is essential for communication between researchers and others, it can also be a vehicle for miscommunication. The problems in reporting extend beyond missing details—eg, when the protocols and publications of 102 trials were compared, 62% had at least one primary outcome changed, introduced, or omitted.6
Recommendations
1
Funders and research institutions must shift research regulations and rewards to align with better and more complete reporting
•
Monitoring—when assessing research (or researchers), funders and research institutions should consider the accessibility of research protocols, study materials, study data, and their use by others
2
Research funders should take responsibility for reporting infrastructure that supports good reporting and archiving
•
Monitoring—funders and research institutions should regularly report expenditures for reporting infrastructure and archiving
3
Funders, institutions, and publishers should improve the capability and capacity of authors and reviewers in high-quality and complete reporting
•
Monitoring—researchers should use reporting guidelines, registries, archives, etc; and take up training opportunities
Discussions of waste in the reporting and usability of research articles commonly focus on peer-reviewed journals, but should look more broadly at the multiple and various forms in which research processes and findings are reported (figure 1). The efficiency of any reporting system depends on the quality of both the individual documents and the linkages between them. The complete documentation of a study includes several elements which often present the same information in different ways (panel 1). For some studies, such as the CRASH trial of corticosteroids after head injury, investigators provide most of these documents in one website. However, a problem that became apparent during the preparation of this review is the dearth of research on the usability of research reports. One (inadequately assessed) format is often assumed to be best, and the various needs of different research users are disregarded.7 Across the set of documents problems in reporting can be subdivided into missing or incomplete information (eg, missing details of treatments, outcomes in methods not shown in results, or selective presentation of findings); incorrect or misleading information in reports (eg, misleading figures, incorrect statistical analyses, a change of primary outcome, or spin in conclusions); inconsistent information (eg, differences between report summaries in trial registers vs articles in peer-reviewed journals); poorly written text, and poor use of figures and tables; and information presented in obscure or less-than-optimum formats (eg, non-searchable PDF files).
A large systematic review update in 2012 involving 50 studies and reports of more than 16 000 randomised trials that assessed the effect of journal endorsement of the CONSORT checklist showed that, despite improvements in the completeness of reporting for 22 of 25 checklist items, there are still major reporting deficiencies in journal publications.8 Although the adoption of reporting guidelines such as CONSORT, STARD, and PRISMA has helped to improve the quality of research reports, all guidelines remain much less adhered to than they should be. In the previous paper in this Series, Chen and colleagues9 examined the large amount of non-publication or delayed publication of research. In the fifth paper of the Series, we document problems in each of Bradford Hill's four areas and examine some options to improve reporting of research.
Section snippets
What was planned and done: the question and methods
Researchers, clinicians, and other users of research findings often want to apply and act on research that has shown benefit or promise. To do so, they need accurate and adequate descriptions of the study objective, methods, populations, interventions, tests, markers or biomarkers, and context. However, many authors seem unaware of how to achieve these aims.
Several therapeutic assessment studies have shown that key information is often absent from reports. Incomplete reporting means that the
Introduction to the problem
In the results section of a paper, readers expect to find an informative description of what actually happened in the study and the answers to the study questions. The outcomes and analyses presented should correspond with those specified in the study protocol and the statistical analysis plan, if this plan exists, and should match what is stated in the methods section. They should not be presented selectively and should provide adequate data and detail to allow incorporation in future
The need for a multistage approach
Improvements in reporting could occur at the presubmission, reviewing, publication, or post-publication stages. Although early correction is likely to be preferable, an expectation that all problems can be identified and fixed at any one stage is unrealistic; therefore, a multistage approach will be needed, including informal and formal presubmission commentary. Additionally, responsible reporting of research should be taught as an essential component of research training.
Presubmission
Reporting guidelines,
What it means: setting results in the context of previous research
Reports of new research should set the new findings in the context of the body of other relevant research. Although apparently simple, the wide scatter and poor indexing of research makes this process difficult and so it is usually done poorly. For example, of 136 trials of methods to prevent pain from propofol injections published after January, 2002, 37 (27%) did not cite a systematic review that was published in 2000;61 and, because most contributors did not use the review to inform study
Reducing waste in reporting
A precise quantification of the amount of waste in the reporting of research is not possible. However, figure 3 summarises issues that relate to different parts of the standard primary publication. Although flaws are likely to be correlated, the numbers clearly suggest that most publications have elements that are missing, poorly reported, or ambiguous. The options for improvement that we have discussed suggest that most problems are remediable at fairly low cost.
How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data
PLoS One
(2009)
A-W Chan et al.
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles
JAMA
(2004)
E Wager
Publishing clinical trial results: the future beckons
PLoS Clin Trials
(2006)
L Turner et al.
Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review
Syst Rev
(2012)
A-W Chan et al.
Increasing value and reducing waste: addressing inaccessible research
Lancet
(2014)
P Glasziou et al.
What is missing from descriptions of treatment in trials and reviews?
BMJ
(2008)
JM Duff et al.
Adequacy of published oncology randomized controlled trials to provide therapeutic details needed for clinical application
J Natl Cancer Inst
(2010)
C Pino et al.
Inadequate description of educational interventions in ongoing randomized controlled trials
Trials
(2012)
PS Fontela et al.
Quality and reporting of diagnostic accuracy studies in TB, HIV and malaria: evaluation using QUADAS and STARD standards
PLoS One
(2009)
PM Bossuyt et al.
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
BMJ
(2003)
C Kilkenny et al.
Survey of the quality of experimental design, statistical analysis and reporting of research using animals
PLoS One
(2009)
C Kilkenny et al.
Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research
PLoS Biol
(2010)
J Little et al.
STrengthening the REporting of Genetic Association Studies (STREGA): an extension of the STROBE statement
PLoS Med
(2009)
W Bouwmeester et al.
Reporting and methods in clinical prediction research: a systematic review
PLoS Med
(2012)
LM McShane et al.
Reporting recommendations for tumor marker prognostic studies (REMARK)
J Natl Cancer Inst
(2005)
AC Janssens et al.
Strengthening the reporting of Genetic RIsk Prediction Studies: the GRIPS Statement
PLoS Med
(2011)
C Bennett et al.
Reporting guidelines for survey research: an analysis of published guidance and reporting practices
PLoS Med
(2010)
LM Schilling et al.
Inaccessible novel questionnaires in published medical research: hidden methods, hidden costs
Am J Epidemiol
(2006)
S Lewin et al.
Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study
BMJ
(2009)
S Michie et al.
Effective techniques in healthy eating and physical activity interventions: a meta-regression
Health Psychol
(2009)
S Woloshin et al.
Press releases by academic medical centers: not so academic?
Ann Intern Med
(2009)
FJ Molnar et al.
Assessing the quality of newspaper medical advice columns for elderly readers
Acupuncture overviews are increasing rapidly; however, their reporting quality is yet unclear. We aimed to investigate the reporting quality of relevant overviews according to the preferred reporting items for overviews of reviews (PRIOR) statement.
We systematically searched PubMed from inception to August 16, 2022 for overviews on acupuncture therapies. Reporting quality of included overviews was evaluated using the PRIOR statement, and the results were cross-checked. Multiple linear regression analysis was used to assess the predictors of the reporting completeness. GraphPad 9.4 was utilized to generate an evidence map, Excel 2019 was used to extract and manage data, and R 4.2.3 was used for data analysis.
A total of 49 overviews published from 2006 to 2022 were included, of which China ranked first with 38 overviews. The most frequently searched database was PubMed/ Medline (n = 48, 98%), and commonly used methodological quality assessment tool was AMSTAR-2 (n = 14, 29%). The overarching themes centered on acupuncture for obstetrics, gynecology, reproductive diseases, as well as depression, anxiety, and insomnia. Reporting quality needs to be improved involving the definition of systematic reviews (SRs), overlap of primary studies and SRs, methods for managing discrepant data across SRs, risk of bias in primary studies, heterogeneity, and sensitivity analysis of synthesized results, reporting bias assessment, and registration and protocol. Moreover, publication in recent years and receiving funding support were significantly associated with higher overall reporting quality score (P < 0.05).
Based on the PRIOR statement, this methodological study indicates that the reporting quality of the included acupuncture overviews is poor. In the future, authors of overviews are encouraged to use the PRIOR statement for standardized reporting. Furthermore, it is recommended that journal editors mandate the inclusion of this statement in authors’ reports and require a complete PRIOR checklist.
To develop a set of detailed definitions for foundational domains commonly used in OMERACT (Outcome Measures in Rheumatology) core domain sets.
We identified candidate domain definitions from prior OMERACT publications and websites and publications of major organizations involved in outcomes research for six domains commonly used in OMERACT Core Domain Sets: pain intensity, pain interference, physical function, fatigue, patient global assessment, and health-related quality of life. We conducted a two-round survey of OMERACT working groups, patient research partners, and then the OMERACT Technical Advisory Group to establish their preferred domain definitions. Results were presented at the OMERACT 2023 Methodology Workshop, where participants discussed their relevant lived experience and identified potential sources of variability giving the needed detail in our domain definitions.
One-hundred four people responded to both rounds of the survey, and a preferred definition was established for each of the domains except for patient global assessment for which no agreement was reached. Seventy-five participants at the OMERACT 2023 Methodology Workshop provided lived experience examples, which were used to contextualise domain definition reports for each of the five domains.
Using a consensus-based approach, we have created a detailed definition for five of the foundational domains in OMERACT core domain sets; patient global assessment requires further research. These definitions, although not mandatory for working groups to use, may facilitate the initial domain-match assessment step of instrument selection, and reduce the time and resources required by future OMERACT groups when developing core outcome sets.
This study evaluates the endorsement of open science practices by dental journals.
This was a meta-research study that included journals listed in the 2021 Journal Citation Reports under Dentistry. A comprehensive evaluation was performed by accessing journal websites to ascertain the availability of publicly accessible instructions to authors in Portuguese, English, or Spanish. A researcher extracted information from the "Instructions for Authors" section, encompassing the journal's impact factor, mention of any reporting guidelines, details on data sharing, acceptance of articles in preprint format, and information regarding study protocol registration. Descriptive data analysis was conducted using the Stata 14.0 program, and an Open Science Score (OSS) (ranging from 0 to 100 %) was calculated for each journal by considering five open science practices. Pearson's correlation test was conducted to determine the relationship between the OSS score and journal impact factor.
Ninety journals were included in the study. Most journals (70 %) indicated the mandatory use of reporting guidelines, while 60 % recommended data sharing. Conversely, 46.7 % did not provide information on study protocol registration, and 44.4 % stipulated them as mandatory for authors. Regarding preprints, 50 % of the journals did not provide any information, but 46.7 % confirmed their acceptance. The mean OSS was 52.9 % (standard deviation 26.2). There was a weak correlation (Pearson's correlation coefficient of 0.221) between the journal impact factor and OSS (P-value=0.036).
This study found varying degrees of endorsement of open science practices among dental journals.
Dental practitioners rely on high-quality, evidence-based research for informed decision-making. By assessing the endorsement of open science practices, our study contributes to improving the quality and reliability of dental research, ultimately enhancing the evidence base for clinical practice.
Core outcome sets (COS) are agreed sets of outcomes for use in clinical trials, which can increase standardization and reduce heterogeneity of outcomes in research. Using a COS, or not, is a behavior that can potentially be increased using behavioral strategies. The aim of this study was to identify behavioral intervention components to potentially increase use of COS in trials.
This project was informed by the Behavior Change Wheel framework. Two reviewers extracted barriers and facilitators to COS use from four recently published studies examining COS use in trials. Barriers and facilitators were coded to the Capability, Opportunity, Motivation-Behavior (COM-B) model, which forms part of the Behavior Change Wheel. COM-B findings were mapped to intervention functions by two reviewers, and then mapped to behavior change techniques (BCTs). Full-team Affordability, Practicability, Effectiveness/Cost-effectiveness, Acceptability, Side effects/Safety, Equity ratings were used to reach consensus on intervention functions and BCTs. BCTs were operationalized using examples of tangible potential applications and were categorized based on similarity.
Barriers and facilitators were identified for all capability, opportunity and motivation aspects of the COM-B model. Five intervention functions (education, training, enablement, persuasion, and modeling) and 15 BCTs were identified. Thirty-six BCT examples were developed, including providing information on benefits of COS for health research, and information choosing COS. BCT examples are categorized by approaches related to “workshops,” “guidance,” “audio/visual resources,” and “other resources.”
Study findings represent diverse ways to potentially increase COS use in trials. Future work is needed to examine effects of these behavioral intervention components on COS use. If effective, increased use of COS can improve outcome reporting and minimize outcome heterogeneity and research waste.
Since 2017, the UK government has made concerted efforts to ensure the dissemination of clinical trials conducted at public research institutions. This study aims to understand how stakeholders within these institutions responded to these pressures and modified internal policies and processes while identifying best practices and barriers to improved transparency practice.
Research governance and trial management staff from UK public research institutions (i.e., Universities and NHS Trusts) in England, Scotland and Wales participated in semi-structured interviews. Interviews were analysed using thematic analysis, aided by the framework method.
Between November 2020 and July 2021, 14 individual participants were recruited from 11 different institutions. They worked in research governance, administration, and management. Almost universally, new policies and procedures have been established to ensure investigators are aware of, and supported in, fulfilling their transparency commitments, however challenges remain. Trials of medicinal products, as the most closely regulated research, consequently received the most attention. National professional networks aid in sharing knowledge and best practice within this community.
Investment in the institutional governance of transparency is essential to achieving optimal transparency practices. Universities and hospitals share responsibility for ensuring research is performed and reported to regulatory standards. Facing political pressure, public research institutions in the UK have made efforts to improve their transparency practice which can provide key insights for similar efforts elsewhere.
2024, Archives of Physical Medicine and Rehabilitation
To determine the reproducibility of exercise therapy interventions in randomized controlled trials for rotator cuff-related shoulder pain (RCRSP).
Data sources included Medline, Cumulative Index to Nursing and Allied Health Literature, Cochrane Central Register of Controlled Trials, and SPORTDiscus from studies published from database inception to April 23, 2022.
Randomized controlled trials studying the use of exercise therapy for RCRSP.
Two reviewers extracted exercise reporting details from all studies using the Template for Intervention Description and Replication (TIDieR) and the modified Consensus on Exercise Reporting Template (CERT). The same 2 reviewers assessed risk of bias of all studies using Cochrane Risk of Bias Tool version 2.0.
For 104 studies meeting inclusion criteria, the average number of items reported on the TIDieR was 5.27 (SD 2.50, range 1-12 out of 12) and 5.09 (SD 4.01, range 0-16 out of 16) on the CERT. Improved reporting over time was seen on both the TIDieR and CERT dating back to 1993 and through April 23, 2022. When comparing groups of studies published before and after the TIDieR (2014) and CERT (2016) were established, a statistically significant increase in median scores was noted on the TIDieR (P=.02) but not the CERT (P=.31). Quality of exercise therapy reporting was highest in studies with “low risk” of bias, and lowest in studies with “high risk” of bias on the RoB-2.
Overall exercise reporting in trials for RCRSP is incomplete despite the development of the TIDieR and CERT checklists. This has implications for translating evidence into practice.