22 Obtain an Analysis Suggestion
EDA provides an Analysis Suggestion tool that is able to recommend statistical tests compatible with the design of your experiment. It is able to do this because the diagramming and critique process essentially asks the same questions a biostatistician would ask you, when helping you design your experiment.
The EDA analysis suggestion tool is not infallible. It is designed to provide advice suitable for most common, relatively simple, experimental designs. If your experimental design is complex, it may advise that you consult a statistician for advice.
The EDA analysis tool will suggest approaches that are compatible with your design, but it may only suggest one of many possible or acceptable analyses, and not necessarily the best one.
In this part of the workshop, we will use this tool to get a suggestion from EDA for how to analyse data from the experiment.
22.1 Run the Analysis Suggestion Tool
- Click on the
Tools -> Analysis Suggestionmenu option (Figure 22.1) to start the tool running, to analyse your design.
Tools menu, showing the Analysis Suggestion option.
You can use the question mark icon to get help for the Analysis Suggestion tool.
The critique involves running analysis on a remote machine, and may take a couple of minutes, so please be patient.
- Note the Task Completed message. This should issue no warnings, and a single analysis suggestion for this experiment (Figure 22.2, Figure 22.3)
- Click on the Analysis Suggestion icon, on the Analysis node, to see the pop-up window describing the analysis suggestion, with a justification for the choice (Figure 22.4).
The Analysis Suggestion tool has recommended a one-way ANOVA with blocking factor(s) and we could now proceed to estimating sample sizes that would give us a desired level of statistical power.
22.2 Interpreting the suggestions
The Analysis Suggestion tool will recommend an approach consistent with the design and information provided, but it may not be completely comprehensive, and the advice might not be exactly right for your situation. The Analysis Suggestion tool does attempt to describe how you should approach the analysis carefully. For example, with this design the result repeats the assumptions it makes about the design and notes:
If this description is not accurate, please check your diagram and verify that all nodes are connected properly, all variables and variable categories are indicated and tagged to the relevant interventions or measurements and the information provided in the properties of each node is accurate; then critique it again.
The design tool is not a mind-reader and can only work with the information provided. If that information is incorrect or not accurate about your design, then the suggestion may be incorrect.
The recommendation goes on to list some caveats:
Statistical analysis methods compatible with this design include the one-way ANOVA with blocking factor(s).
The tool notes that this is only one method compatible with the design, but there are others (including two-way ANOVA, and regression models); just because they are not recommended does not mean they are not appropriate.
The ANOVA approach assumes that the data satisfies the parametric assumptions […] If you have reasons to think the data are not normally distributed, […] consider transforming the data.
It is the researcher’s responsibility to ensure that the data collected meet the assumptions of the statistical methods employed. The EDA tool cannot predict what data will be collected.
Note that a one-way ANOVA can answer the question: is there an overall difference between the groups; they do not provide information on which individual groups differ. To identify where any differences lie, it may be necessary to carry out a post-hoc test. Note that in this case the sample size might have to increase and the experiment should be powered to detect differences in the pairwise comparisons.
It is the researcher’s responsibility to ensure that the experiment is correctly powered for the comparisons they wish to perform, and that the statistical test is appropriate. For two categories, the one-way ANOVA should require the same sample size as a t-test, for the same statistical power. But if the design and questions become complex, it is usually best to consult a statistician.
Please note that an experiment with a single variable of interest might not make the most efficient use of the data. Consider using a factorial design.
The EDA tool cannot guess how many other experiments you might want to run, and is trying to be helpful by reminding you to consider this study in the broader scope of your research. A factorial design might make more efficient use of the data collected, and require the use of fewer experimental subjects for the same statistical confidence, if there are multiple factors of interest that could affect the outcome.