3 Reporting of Research Using Animals
For scientific, ethical and economic reasons, experiments involving animals should be appropriately designed, correctly analysed and transparently reported. This increases the scientific validity of the results, and maximises the knowledge gained from each experiment. A minimum amount of relevant information must be included in scientific publications to ensure that the methods and results of a study can be reviewed, analysed and repeated. Omitting essential information can raise scientific and ethical concerns. (Kilkenny et al. (2009))
You might expect that senior, practising scientists with decades of experience in animal research would be experts in statistics, experimental design, and accurate analysis and reporting of their experiments, and that their publications reflect this expertise.
But does the evidence in the literature support this assumption?
3.1 The 2009 NC3Rs systematic survey
The National Centre for the Replacement, Refinement, and Reduction of Animals in Research (NC3Rs) was established in 2004 as the UK’s national organisation for the 3Rs (Reduction, Replacement, Refinement). It works with scientists to replace the use of animals by developing new approaches and technologies or, where use of animals is unavoidable, to reduce the number of animals used in each experiment and to minimise any pain, suffering or distress that the animals may experience.
In 2009, the NC3Rs published a systematic survey (Kilkenny et al. (2009)) of the quality of reporting, experimental design, and statistical analysis of recently-published biomedical research using animals.
It did not make for pleasant reading.
3.1.1 Cause for concern in reporting of biomedical research involving animals
Detailed information was collected from 271 publications, about the objective or hypothesis of the study, the number, sex, age and/or weight of animals used, and experimental and statistical methods. Only 59% of the studies stated the hypothesis or objective of the study and the number and characteristics of the animals used. […] Most of the papers surveyed did not use randomisation (87%) or blinding (86%), to reduce bias in animal selection and outcome assessment. Only 70% of the publications that used statistical methods described their methods and presented the results with a measure of error or variability. (Kilkenny et al. (2009))
The papers collated by the authors in their study spanned a wide range of biomedical topics including: dietary research; behavioural studies; immunology work; drug and chemical testing; and so on. Some of the problems of basic reporting they found included:
- 35% of the studies gave different numbers of animals in the Methods and Results sections without explanation
- 18% of studies did not explain why they used the number of animals that they did
- 4% of papers didn’t even say how many animals were used!
Biological experimentation is difficult, and we should hope to design experiments that minimise bias. To do this we use basic statistical techniques like randomisation and “experimenter blinding.” Of the 271 papers they examined:
- 12% said they used randomisation
- 13% said they used blinding
- 10% of studies didn’t say what statistical approach they used
Reproducibility and transparency are important components of good scientific reporting, but:
- 51% of studies did not include the number of animals in each group
- 30% of papers reporting statistical methods did not describe their methods
None of the studies examined explained the sample size (number of animals used in each group) for the experiment.
As the review notes:
Power analysis or other very simple calculations, which are widely used in human clinical trials and are often expected by regulatory authorities in some animal studies, can help to determine an appropriate number of animals to use in an experiment in order to detect a biologically important effect if there is one. This is a scientifically robust and efficient way of determining animal numbers and may ultimately help to prevent animals being used unnecessarily. Many of the studies that did report the number of animals used reported the numbers inconsistently between the methods and results sections. The reason for this is unclear, but this does pose a significant problem when analysing, interpreting and repeating the results. (Kilkenny et al. (2009))
3.2 The ARRIVE guidelines
The authors of that systematic review published a paper the next year (Kilkenny et al. (2010)) describing the ARRIVE guidelines for reporting animal research. This framework was developed as a checklist to help researchers report their results transparantly and reproducibly.
Many journals now routinely request information in the ARRIVE framework, often as electronic supplementary information. The framework covers 20 items including the following:
- Objectives: primary and any secondary objectives of the study, or specific hypotheses being tested
- Study design: brief details of the study design, including the number of experimental and control groups, any steps taken to minimise the effects of subjective bias, and the experimental unit
- Sample size: the total number of animals used in each experiment and the number of animals in each experimental group; how the number of animals was decided
- Statistical methods: details of the statistical methods used for each analysis; methods used to assess whether the data met the assumptions of the statistical approach
- Outcomes and estimation: results for each analysis carried out, with a measure of precision (e.g., standard error or confidence interval).
As Kilkenny et al. (2009) note:
There are many opportunities for the scientific community to improve both the experimental design and the quality of reporting of biomedical research using animals. Serious efforts are needed to improve both the quality of experimental design and the quality of reporting in order to make research articles better suited to the needs of readership. […] Raising awareness that these problems exist will be the first step in tackling these fundamental issues.
A key step in tackling these issues is to ensure that the next generation of scientists are aware of what makes for good practice in experimental design and animal research, and that they are not led into poor or inappropriate practices by more senior scientists without a proper grasp of these issues.
3.3 It’s your turn…
The ARRIVE framework has undoubtedly improved the reporting of experiments involving animals. but it is not effective in isolation. In otder to use the framework effectively and report good ractice, researchers must first know what good practice is, and then implement it.
This workshop is designed to give you an introductory foundation in aspects of experimental design - specifically in calculating appropriate sample sizes for experiments, and using the NC3Rs Experimental Design Assistant (NC3Rs EDA). We cannot hope to give you a complete statistical grounding in the time available, but we aim to help you avoid some of the mistakes and omissions of those who have gone before you, and improve the the transparency, reproducibility, and reliability of your work, and any scientific publications that come from it.