MP968 Experimental Design Workshop

Leighton Pritchard

University of Strathclyde

2025-11-24

Why do we need experimental design?

We should not cause unnecessary suffering

We should always minimise suffering

This may mean not performing an experiment at all. Not all new knowledge or understanding is worth causing suffering to obtain it.

Where there is sufficient justification to perform an experiment, we are ethically obliged to minimise the amount of distress or suffering that is caused, by designing the experiment to achieve this.

Why we need statistics

It may be easy to tell whether an animal is well-treated, or whether an experiment is necessary.

But what is an acceptable (i.e. the least possible) amount of suffering necessary to obtain an informative result?

No-one likes talking about animal experiments.
It’s a difficult, emotive topic that crosses people’s moral red lines.
Animal experiments may be the only practical way to gain essential scientific knowledge
Whatever you believe “suffering” means for an animal, our ethical premise is that this suffering should not be in vain
The first thing to ask is whether the question we’re trying to answer needs to be answered enough that we should cause any suffering
- Not all questions are worth answering in this way
If the question is important enough to ask in this way, we are ethically obliged to minimise the amount of distress and suffering caused.
- If we cannot minimise this sufficiently, it still may not be worth asking the question
We need statistics to help us understand objectively what the least number of individuals is, to answer our question

Challenge

Quiz question

Suppose you are running a necessary and useful experiment with animal subjects, where the use of animals is morally justified. You are comparing a treatment group to a control group. Which of the following choices will cause the least amount of suffering?

Use three subjects per group so a standard deviation can be calculated
Use just enough subjects to establish that the outcome is likely to be correct
Use just enough subjects to be certain that the outcome is correct
Use as many subjects as you have available, to avoid wastage

We carry out experiments to obtain answers to our scientific hypotheses, but the answers we obtain are rarely if ever 100% certain. We usually aim to obtain answers that are very likely to be correct (think about what a statistical hypothesis test means: that the explanation is more likely or less likely to be explainable by the null hypothesis than some threshold), rather than 100% certain.

If we use too few subjects we may still be able to perform a statistical test, but the results of the experiment will be more uncertain and may be more likely to be incorrect than correct. The use of animal subjects in an experiment that is unlikely to give a correct answer (e.g. because too few animals are used) causes unnecessary suffering.

If we attempt to obtain a 100% certain outcome - or nearly so - we may need to use many more subjects - possibly tens or hundreds more - than are required to obtain a result about which we are (say) 80% certain. The use of animal subjects to obtain a level of certainty greater than is needed to answer the question reasonably causes unnecessary suffering.

If we use the number of subjects that are available, just because it is convenient, then we may not know how likely the experiment is to give us a correct answer. The use of animal subjects in an experiment where you do not know how likely you are to get a correct answer is likely to cause unnecessary suffering.

How many individuals?

The appropriate number of subjects

The appropriate number of animal subjects to use in an experiment is always the smallest number that - given reasonable assumptions - will satisfactorily give the correct result to the desired level of certainty.

What assumptions are reasonable?
What is an appropriate level of certainty?

By convention¹ the usual level of certainty for a hypothesis test is: “we have an 80% chance of getting the correct true/false answer for the hypothesis being tested”

Design experiments to minimise suffering

Experimental design and statistics are intertwined

Once a research hypothesis has been devised:

Experimental design is the process of devising a practical way of answering the question
Statistics informs the choices of variables, controls, numbers of individuals and groups, and the appropriate analysis of results

Design your experiment for…

your population or subject group (e.g. sex, age, prior history, etc.)
your intervention (e.g. drug treatment)
your contrast or comparison between groups (e.g. lung capacity, drug concentration, etc.)
your outcome (i.e. is there a measurable or clinically relevant effect)

Once a research hypothesis has been devised, Experimental Design is the process by which the practical means of answering that question is constructed. The design should aim to exclude extraneous or confounding influences on the experiment such that the causal factors are isolated and measurable, and any difference in outcome as a result of changing those factors (the “signals”) can also be measured cleanly.

Statistics is the branch of applied science that allows us to make probabilistic inferences about our certainty in the “signal” - measurements, comparisons and experimental outcomes - even in the face of natural variations in processes and “noise,” and the way we choose small groups to represent populations.

You should design your experiment specifically for your combination of population/subject group, the intervention you’re applying, the contrast or comparison you’re making, and the outcome you’re expecting to see - specifically a measurable or clinically relevant effect.

The 2009 NC3Rs systematic survey

The importance of experimental design

“For scientific, ethical and economic reasons, experiments involving animals should be appropriately designed, correctly analysed and transparently reported. This increases the scientific validity of the results, and maximises the knowledge gained from each experiment. A minimum amount of relevant information must be included in scientific publications to ensure that the methods and results of a study can be reviewed, analysed and repeated. Omitting essential information can raise scientific and ethical concerns.” (Kilkenny et al. (2009))

We rely on the reporting of the experiment to know if it was appropriate

Causes for concern 1

“Detailed information was collected from 271 publications, about the objective or hypothesis of the study, the number, sex, age and/or weight of animals used, and experimental and statistical methods. Only 59% of the studies stated the hypothesis or objective of the study and the number and characteristics of the animals used. […] Most of the papers surveyed did not use randomisation (87%) or blinding (86%), to reduce bias in animal selection and outcome assessment. Only 70% of the publications that used statistical methods described their methods and presented the results with a measure of error or variability.” (Kilkenny et al. (2009))

We cannot rely on the literature for good examples of experimental design

Causes for concern 2

No publication explained their choice for the number of animals used

We cannot rely on the verbal authority of ‘published scientists’ or ‘experienced scientists’ for good experimental design

One of the most shocking pieces of information is that, of the 48 papers surveyed, not a single one explained why they used the number of animals that they did.
We cannot therefore be assured that the number of animals used was chosen to minimise suffering, or to obtain a statistically justifiable result.
These papers are published. They are written, and the experiments conducted, by “experienced scientists”
Being an “experienced” or “published” scientist is clearly not a benchmark for good experimental design

Remember, these are “experienced,” “published” scientists in receipt of tens of thousands, maybe hundreds of thousands of pounds worth of funding. If you query their methods, they are likely to claim that having performed these experiments, published the outcomes, and received funding to do so, makes their opinion right and yours wrong. Resist this bullying. Always question how these numbers are decided upon and how the experiment is designed and performed. You are sometimes the experimental subjects’ only voice.

Very strong cause for concern

“Power analysis or other very simple calculations, which are widely used in human clinical trials and are often expected by regulatory authorities in some animal studies, can help to determine an appropriate number of animals to use in an experiment in order to detect a biologically important effect if there is one. This is a scientifically robust and efficient way of determining animal numbers and may ultimately help to prevent animals being used unnecessarily. Many of the studies that did report the number of animals used reported the numbers inconsistently between the methods and results sections. The reason for this is unclear, but this does pose a significant problem when analysing, interpreting and repeating the results.” (Kilkenny et al. (2009))

Important

As scientists, you - yourselves - need to understand the principles behind the statistical tests you use, in order to choose appropriate tests and methods, and to use appropriate measures to minimise animal suffering and obtain meaningful results.

You cannot simply rely on the word of “experienced scientists” for this.

The ARRIVE guidelines

The following year Kilkenny et al. (2010) proposed the ARRIVE guidelines: a checklist to help researchers report their animal research transparently and reproducibly.

Good reporting is essential for peer review and to inform future research
Reporting guidelines measurably improve reporting quality
Improved reporting maximises the output of published research

ARRIVE guidelines highlightes

Many journals now routinely request information in the ARRIVE framework, often as electronic supplementary information. The framework covers 20 items including the following (Kilkenny et al. (2010)):

ARRIVE guidelines (highlights)

1. Objectives: primary and any secondary objectives of the study, or specific hypotheses being tested
1. Study design: brief details of the study design, including the number of experimental and control groups, any steps taken to minimise the effects of subjective bias, and the experimental unit
1. Sample size: the total number of animals used in each experiment and the number of animals in each experimental group; how the number of animals was decided
1. Statistical methods: details of the statistical methods used for each analysis; methods used to assess whether the data met the assumptions of the statistical approach
1. Outcomes and estimation: results for each analysis carried out, with a measure of precision (e.g., standard error or confidence interval).

A vital step

Warning

“A key step in tackling these issues is to ensure that the next generation of scientists are aware of what makes for good practice in experimental design and animal research, and that they are not led into poor or inappropriate practices by more senior scientists without a proper grasp of these issues.”

Recommended reading

Bate and Clark (2014)

Some Statistical Concepts

Random variables

Your experimental measurements are random variables

Important

This does not mean that your measurements are entirely random numbers

Caution

Random variables are values whose range is subject to some element of chance, e.g. variation between individuals

Tail length (e.g. timing of developmental signals, distribution of nutrients)
Blood concentrations (e.g. circulatory heterogeneity, transient measurement differences)
Survival time (e.g. determining point of death)

When you take a measurement - tail length, concentration of something in blood, survival time, whatever - you are recording the value of a random variable
- This doesn’t mean that the number is chosen randomly
What it means is that the number is subject to the influence of some kind of random variation
- So tail length might be partially determined by variations in influences on overall growth of an individual (like nutrition), genetic propensity, and some random redistribution of nutrients within the individual or random timing of developmental signals
- Blood concentration might not be entirely uniform, so the measurement is subject to random variations throughout the circulatory system, or transient effects on how you measure the concentration
- Survival time might not be measured 100% accurately - it can difficult to measure the point of death exactly, so there may be some random variation around the actual time
There are numerous ways that random effects (“noise”) can impinge on the “signal” we’re trying to measure

Probability distributions

The probability distribution of a random variable $z$ (e.g. what you measure in an experiment) takes on some range of values¹

The mean of the distribution of $z$

The mean (aka expected value or expectation) is the average of all the values in $z$
- Equivalently: the mean is the value that is obtained on average from a random sample from the distribution
Written as $\mu_{z}$ or $E(z)$

The variance of a distribution of $z$

The variance of the distribution of $z$ represents the expected mean squared difference from the mean $\mu_z$ (or $E(z)$) of a random sample from the distribution.
- $\textrm{variance} = E((z - \mu_z)^2)$

A probability distribution describes the range of values that a random variable - let’s call it $z$ - takes.
- So if you’re measuring tail length, and that’s “constant” for a population, but under the influence of the random variation in how much nutrition an individual can get, the distribution is representing that random effect around the “constant” value.
- In more traditional statistical terms, you can think of $z$ being a single ball drawn from a bag containing an infinite number of balls, each ball with a number written on it
Probability distributions can describe many measures, including
- heights of men, incomes of women, political party preference, and so on
The mean of a probability distribution - its expected value or expectation - is the average of all numbers in the distribution
- There may be an infinite amount of numbers in the dataset (in principle), and there is an equivalent definition: the mean is the value obtained on average in a random sample taken from the distribution
The variance of a probability distribution expresses how much the individual values might differ from that mean.
- Variance is defined as the mean of the square of the difference between each individual value in the distribution and the mean of the distribution
- So, for each ball we draw from the bag, we treat the number as $z$ and calculate the sum of squares of differences from the mean - this is the variance.

Understanding variance

A distribution where all values of $z$ are the same

Every single value in the distribution ($z$) is also the mean value ($\mu_z$), therefore

\[z = \mu_z \implies z - \mu_z = 0 \implies (z - \mu_z)^2 = 0\] \[\textrm{variance} = E((z - \mu_z)^2) = E(0^2) = 0\]

All other distributions

In every other distribution, there are some values of $z$ that differ so, for at least some values of $z$

\[z \neq \mu_z \implies z - \mu_z \neq 0 \implies (z - \mu_z)^2 \gt 0 \] \[\implies \textrm{variance} = E((z - \mu_z)^2) \gt 0 \]

To get some intuition about the idea of variance, imagine that you have a set of values $z$ that are all the same, so there’s no randomness
- So you have a bag of balls, and all the balls in the bag have the number three on them, say
If the value are all the same, then they all have the same value as the mean of the distribution, and $z - \mu_z = 0$
This means that, for all $z$, $(z - \mu_z)^2$ is also zero, and the variance is zero.
But if any value of $z$ is different then, for at least that value, $z - \mu_z \neq 0$, so the square of that value is greater than zero. It follows that the variance must then be greater than zero: $E((z - \mu_z)^2) \gt 0$
In any dataset you meet, you are unlikely to have all values be identical, and so the variance is going to take a positive value
- All measurements you make will have variability in them! This variability must be accounted for.

Standard deviation

Standard deviation is the square root of the variance

\[\textrm{standard deviation} = \sigma_z = \sqrt{\textrm{variance}} = \sqrt{E((z - \mu_z)^2)} \]

Advantages

The standard deviation (unlike variance) takes values on the same scale as the original distribution
- Standard deviation is a more “natural-seeming” interpretation of variation

Note

We can calculate mean, variance, and standard deviation for any probability distribution.

Normal Distribution 1

\[ z \sim \textrm{normal}(\mu_z, \sigma_z) \]

Note

We only need to know the mean and standard deviation to define a unique normal distribution

Tip

Measurements of variables whose value is the sum of many small, independent, additive factors may follow a normal distribution

Important

There is no reason to expect that a random variable representing direct measurements in the world will be normally distributed!

You may see the normal distribution written like this, as “$z$ is distributed as the Normal distribution with mean $\mu_z$ and standard deviation $\sigma_z$.”
- We only need to know the mean and standard deviation to define a unique normal distribution
Values we measure in the real world should follow an approximate normal distribution if each measured value is the sum of many small, independent, additive factors
- This is a consequence of the Central Limit Theorem
- Many real-world variables do follow normal distributions
But there is no reason to expect that any random variable representing any direct measurement you make will have this property, or follow a normal distribution
- This is especially the case if there is a large factor affecting variation of the variable (e.g. sex)

Normal Distribution 2

Tip

For a normal distribution, the mean value is the value at the peak of the curve
The curve is symmetrical, so standard deviation describes variability equally well on both sides of the mean

As an example, we can look at some real data, the heights of men and women in the US.
- Here we have counts of individuals whose heights are measured to the nearest inch - they’re shown as counts in the histograms
These are two different random variables: one for men, one for women
For both distributions, we can calculate a mean, and a standard deviation (or variance)
- Therefore we can calculate a normal distribution representing both men’s and women’s heights (orange curve)
- We calculate the mean heights for men and women (63.7 and 69.1 inches), and also the standard deviations (2.7 and 2.9 inches)
The distributions of heights for each sex, separately, follow an approximate normal distribution
For a normal distribution, the mean value is the value at the peak of the curve
The curve is symmetrical, so the standard deviation describes variability equally well on both sides of the mean

(Non-)Normal Distribution 3

Tip

Here, the mean may not be the same value as the peak of the curve (i.e. the mode)
The curve is asymmetrical, so standard deviation does not describe variation equally well on either side of the mean

Binomial Distribution 1

Suppose you’re taking shots in basketball

how many shots?
how likely are you to score?
what is the distribution of the number of successful shots?

Tip

This kind of process generates a random variable approximating a probability distribution called a binomial distribution.

It is different from a normal distribution.

I mentioned earlier that a normal distribution arises when a process produces the a sum of many independent random variables.
- In general statistical distributions are idealised repreentations of the outcomes of distinct kinds of real-world processes
Suppose that instead of measuring height, weight, concentration or something like that, you’re measuring event outcomes
- These are not the result of summing independent random variables, and do not follow a normal distribution
If you take a bunch of basketball shots (equivalent to our experimental events), each one has some probability of succeeding
The number of successful shots is going to depend on the number of shots you take, and how likely you are to score
- Michael Jordan is much more likely to score any individual attempt than I am
The number of successful shots is a random variable with a probability distribution
This kind of process generates a probability distribution that approximates the binomial distribution
- It’s the same one you get for coin tosses (or any yes/no process)
It is different from a normal distribution
- If your underlying biological process resembles coin tosses or basketball shots, you need to design your experiment and analysis to be using an appropriate statistical test, such as one suitable for a binomial distribution

Binomial Distribution 2

\[ z \sim \textrm{binomial}(n, p) \]

Tip

number of shots, $n = 20$, probability of scoring, $p = 0.3$

\[z \sim \textrm{binomial}(20, 0.3) \]

mean and sd

\[ \textrm{mean} = n \times p \] \[ \textrm{sd} = \sqrt{n \times p \times (1-p)}\]

Design note

You need to design your experiments and analyses to reflect the appropriate process/probability distributions of your data. E.g., does $p$ differ between two conditions?

Poisson distribution 1

In prior experiments the frequency of calcium events in WKY was 3.8 $\pm$ 1.1 events/field/min compared to 18.9 $\pm$ 7.1 in SHR

This is not normal (or binomial)

Something that happens a certain number of times in a fixed interval generates a Poisson distribution.

This is different from a normal or binomial distribution.

Poisson distribution 2

\[z \sim \textrm{poisson}(\lambda)\]

Poisson distribution

\[ \textrm{mean} = \lambda \] \[ \textrm{sd} = \sqrt{\lambda} \]

Expectation ($\lambda$)

Only one parameter is provided, $\lambda$: the rate with which the measured event happens
Suppose a county has population 100,000, and average rate of cancer is 45.2mn people each year

\[z \sim \textrm{poisson}(45,200,000/100,000) = \textrm{poisson}(4.52) \]

Design note

You need to design your experiments and analyses to reflect the appropriate process/probability distributions of your data

E.g., does $\lambda$ differ between two conditions?

Binomial and Poisson distributions

Some important features

All measured values (and $n$) are positive whole numbers or zero; $\lambda$, $p$ may be positive real numbers or zero
The distributions may not be unimodal
The mean is not always the peak value (mode)
The distributions are not always symmetrical (so sd may not describe variation equally either side of the mean)

Distributions in Practice

Distributions are starting points

Distributions arise from and represent distinct generation processes (relate this to your biological system)
- Normal distributions are generated by sums, differences, and averages
- Poisson distributions are generated by counts (per unit interval)
- Binomial distributions are generated by success/failure outcomes
Design experiments with analyses that reflect these processes

Warning

All statistical distributions are idealisations that ignore many features of real data
No real world data should be expected to exactly match any statistical distribution
Poisson models tend to need adjustment for overdispersion

The different distributions we’ve looked at arise from distinct generation processes that have parallels in the real world
- Normal distributions arise when independent values are summed, or we take differences or averages of them
- Poisson distributions arise from counts, or rates (i.e. counts per unit)
- Binomial distributions arise from success/failure outcomes like tossing a coin
We need to take care though, as these distributions are idealised outcomes from those processes
In the real world there are many other influences that cause collected data to deviate from these idealisations
- There is no reason to expect real world data to exactly match any statistical distribution
- Though there are some principles: we ought not to use a normal distribution to model count data (as the normal distribution can drop below zero where counts cannot)
In general we approximate our real data with these idealised distributions in order to obtain more statistical power

Normal Distribution Redux

Probability mass

approximately 50% of the distribution lies in the range $\mu \pm 0.68\sigma$
approximately 68% of the distribution lies in the range $\mu \pm \sigma$
approximately 95% of the distribution lies in the range $\mu \pm 2\sigma$
approximately 99.7% of the distribution lies in the range $\mu \pm 3\sigma$

Estimates, standard errors, and confidence intervals

Parameters

Parameters are unknown numbers that determine a statistical model

A linear regression

\[ y_i = a + b x_i \]

Parameters are:
- $a$ (the intercept)
- $b$ (the gradient)

A normal distribution representing your data

\[ z \sim \textrm{normal}(\mu_z, \sigma) \]

Parameters are: $\mu_z$ and $\sigma$

Estimands

An estimand (or quantity of interest) is a value that we are interested in estimating

A linear regression

\[ y_i = a + b x_i\]

We want to estimate values for:
- $a$ (the intercept)
- $b$ (the gradient)
- predicted outcomes at important values of $x_i$

These are all estimands, and estimates are represented using the “hat” symbol: $\hat{a}$, $\hat{b}$, etc.

A normal distribution representing your data

\[ z \sim \textrm{normal}(\mu_z, \sigma) \]

Estimands are: $\mu_z$ and $\sigma$
- Maybe you want to determine the 95% confidence interval - this is also an estimand

Standard Errors and Confidence Intervals

The standard error is the estimated standard deviation of an estimate
- It is a measure of our uncertainty about the quantity of interest

Note

Standard error gets smaller as sample size gets larger
- You know more about the most likely value, the more data/information you collect
- Standard error tends to zero as sample size gets large enough

The confidence interval (or CI) represents a range of values of a parameter or estimand that are roughly consistent with the data

Important

In repeated applications, the 50% confidence interval will include the true value 50% of the time
- A 95% confidence interval will include the true value 95% of the time

Tip

The usual 95% confidence interval rule of thumb for large samples (assuming a normal distribution) is to take the estimate $\pm$ two standard errors

Now I’ve mentioned confidence intervals we’ll need to talk about them
We’ll also need to talk about standard errors
The standard error is the estimated standard deviation of an estimate
- It represents our estimate of our uncertainty in the estimate of the quantity
You will probably have heard of the standard error of the mean (SEM)
- We estimate the mean of the population by calculating the mean of the representative sample
- SEM represents our estimate of our uncertainty in our estimate of the mean
- The more measurements we have in the sample, the less uncertainty we have in our estimate - and so SEM tends to decrease as the number of individuals in the sample increases (and, if the sample gets large enough, the uncertainty can approach zero)
A confidence interval represents our uncertainty differently: as a range of values that are roughlly consistent wiht the data
- The confidence interval has upper and lower bounds
- The size of a confidence interval is given as a percentage
A 50% confidence interval (CI) is expected to contain the true value about 50% of the time
- That is, half of all 50% confidence intervals do not contain the true value
A 95% confidence interval (CI) is expected to contain the true value about 95% of the time
- That is, 5% of all 95% confidence intervals do not contain the true value
As a rule of thumb, for normal distributions and large samples, the 95% CI is bounded by values two standard errors from the estimate.
- The 95% CI for the mean is the range {(mean - 2xSEM), (mean + 2xSEM)}

Statistical significance and hypothesis testing

Statistical significance 1

Some scientists choose to consider a result to be “stable” or “real” if it is “statistically significant”
They may also consider “non-signifcant” results to be noisy or less reliable

Warning

I, and many other statisticians, do not recommend this approach (though we will not cover alternatives today)

However, the concept is widespread and we need to discuss it

Statistical significance 2

A common definition

Statistical significance is conventionally defined as a threshold (commonly, a $p$-value less than 0.05) relative to some null hypothesis or prespecified value that indicates no effect is present.
E.g., an estimate may be considered “statistically significant at $P < 0.05$” if it:
- lies at least two standard errors from the mean
- is a difference that lies at least two standard errors from zero
More generally, an estimate is “not statistically significant” if, e.g.
- the observed value can reasonably be explained by chance variation consistent with the null hypothesis
- it is a difference that lies less than two standard errors from zero

Most tests rely on probability distributions

We need to relate the measured values in the real world to an appropriate statistical distribution that approximates them

The way you’re likely to see statistical significance presented is that some threshold - usually a p-value of less than 0.05 - is applied in a statistical test, relative to some null hypothesis or prespecified value that indicates the absence of an effect
So you are likely to see an estimate be considered “statistically significant at P<0.05” if it lies at least two SEs from the mean, or is a difference that lies at least two SEs from zero
Conversely, you will see estimates be considered “not statistically significant at P<0.05” if they lie less than two SEs from the mean, or the observed value can reasonably be explained by chance variation alone
As you might have guessed from the talk of standard errors and distances from means, we map our measured values in the real world to statistical probability distributions to calculate these values.

A simple example: The experiment

The experiment

Two drugs, $C$ and $T$ lower cholesterol¹, and we want to compare their effectiveness
We randomise assignment of $C$ and $T$ to members of a single cohort of comparable individuals, whose pre-treatment cholesterol level is assumed to be drawn from the same distribution (i.e. be approximately the same)
We measure the post-treatment cholesterol levels $y_T$ and $y_C$ for each individual in the two groups.
We calculate the average measured $\bar{y}_T$ and $\bar{y}_C$ for the treatment and control groups as estimates for the true post-treatment levels $\theta_T$ and $\theta_C$.
- We also calculate standard deviation for the two groups, $\sigma_T$ and $\sigma_C$

Let’s consider a simple experiment
We want to compare the efficacy of two drugs, C and T, in their ability to lower cholesterol
- We assume a uniform pool of individuals, and randomly assign C and T to two equally-sized groups drawn from that cohort
- This implies that the starting cholesterol level of all individuals is drawn from the same distribution, and both groups should have the same mean and variation
We administer the drugs in the same way, for the same period of time, and measure the post-treatment cholesterol level in each individual
- We calculate the post-treatment mean and standard deviation for the two groups
- These allow us to estimate the underlying normal distribution of post-treatment cholesterol levels in each group
We can use the estimated means, $\bar{y}_T$ and $\bar{y}_C$ for the treatment and control groups as estimates for the true post-treatment levels $\theta_T$ and $\theta_C$.

A simple example: The hypotheses

We want to know if the treatments have different sizes of effect
- If they do, there should be a difference between the (average) post-treatment cholesterol level in each group
- The true post-treatment levels are $\theta_T$ and $\theta_C$
- We have estimated means, $\bar{y}_T$ and $\bar{y}_C$ for post-treatment levels

The hypotheses

We are interested in $\theta = \theta_T - \theta_C$, the expected true post-test difference in cholesterol between the two groups $T$ and $C$.
Our null hypothesis ($H_0$) is that $\theta = 0$, i.e. there is no difference ($\theta_C = \theta_T$)
Our alternative hypothesis ($H_1$) is that there is a difference, so $\theta \neq 0$, (i.e. $\theta_C \neq \theta_T$)

Our goal in this experiment is to determine whether the treatments have different effects on lowering cholesterol
- If they do, then there should be a difference between the (average) post-treatment cholesterol levels of groups T and C
There are true post treatment levels for each group: $\theta_T$ and $\theta_C$
- But we have only estimated them as $\bar{y}_T$ and $\bar{y}_C$
- So we turn to a hypothesis test to determine how likely it is our estimates support there being a difference in the true levels between the two groups
We set up a null hypothesis that there is no difference between the groups: $\theta = 0$
- In this hypothesis, any deviation from zero we see is considered the result of chance variation in the data, or error we cannot eliminate from measurements
We also define an alternative hypothesis that there is a difference between the groups: $\theta \neq 0$
- We’re not insisting that $\theta$ is greater than or less than zero, just that it’s not zero
- And also that any deviation from zero is not accounted for by the null hypothesis

A simple example: The distribution 1

To perform a statistical test, we may assume a distribution and parameters for the null hypothesis
- We can then test the observed estimate against that distribution to see how likely it is that the null hypothesis would have generated it

The distribution

We use a probability distribution reflecting generation of the null hypothesis: $\theta_C = \theta_T$
- This allows us to define a test statistic $T_\textrm{crit}$ (i.e. a threshold probability of “significance”) in advance
We test the estimated value from the experiment ($\bar{y}_T - \bar{y}_C$) to calculate a $p$-value for our estimate: $p = \textrm{Pr}(T(y_{\textrm{null}}) > T_\textrm{crit} = T(\bar{y}_T - \bar{y}_C))$

At this point we haven’t defined what our null hypothesis distribution should be, but we need to do so
- Once we have this, we can test our observed estimate of $\hat{\theta} = \bar{y}_T - \bar{y}_C$ against the distribution to see how likely it is to be explained by the null hypothesis
We need to choose a probability distribution that reflects the process generating the null hypothesis
- In this case, we could choose a normal distribution, which is typically appropriate for this kind of measurement
Then we can define a test statistic $T_\textrm{crit}$ in advance
- $T_\textrm{crit}$ represents a threshold value in the null distribution - one where we would consider difference giving a larger value of $T$ to be meaningful
- You may be familiar with setting P<0.05, where the probability of seeing some value in the null distribution is 0.05 or less, and we might choose to set a value of $T_\textrm{crit}$ that corresponds to this
We then calculate the test statistic corresponding to our actual estimate, and calculate the $p$-value
- This is the probability of observing our estimate, or a more extreme value, under the assumptions of the null model distribution

A simple example: The null hypothesis

The null hypothesis

Assume that the true difference $\theta$ is normally-distributed with $\mu_\theta=0$, $\sigma_\theta=1$

A simple example: The estimated difference

Observed between post-treatment levels: $\bar{y}_T - \bar{y}_C = -1.4$

Is this an unlikely outcome given the null hypothesis?

A simple example: A significance threshold

We choose a significance threshold in advance

Suppose we set a threshold $T$ corresponding to the 90% confidence interval (i.e. $P<0.1$)
- If the estimate is not in the central 90% of the distribution, we’ll say it’s “significant”

A simple example: Compare the estimate

Compare the estimate to the threshold

The estimate lies outwith the threshold, so we call the difference “significant”

A simple example: Another threshold

We choose a significance threshold in advance

Suppose we set the threshold $T$ corresponding to the 95% confidence interval (i.e. $P<0.05$) instead?

A simple example: Another outcome

Compare the estimate to the threshold

The estimate lies within the threshold, so the difference is “not significant”

A simple example: What changed?

What did not change

The null hypothesis was the same
The observed estimate of difference was the same

What changed

Our choice of significance threshold changed

Significance threshold choice

Once the estimate is known, it is always possible to find a threshold that makes it “significant” or “not significant”
It is dishonest to select a threshold deliberately to make your result “significant” or “not significant”
Always choose and record (preregister) your threshold for significance ahead of the experiment

Tailed tests: two-tailed

Use two tails if direction of change doesn’t matter

With a two-tailed hypothesis test, we do not care which direction of change is significant

Tailed tests: one-tailed (left)

Use one-tailed tests when direction matters

If we’re testing specifically for a significant negative difference/reduction, use a left-tailed test
e.g. if we wanted to know if $T$ reduced post-test levels with respect to $C$ at a threshold of $P < 0.05$

Tailed tests: one-tailed (right)

Use one-tailed tests when direction matters

If we’re testing specifically for a positive difference/increase, use a right-tailed test
e.g. if we wanted to know if $T$ increased post-test levels with respect to $C$ at a threshold of $P < 0.05$

Problems with statistical significance 1

Warning

It is a common error to summarise comparisons by statistical significance into “significant” and “non-significant” results

Statistical significance is not the same as practical importance

Suppose a treatment increased earnings by £10 per year with a standard error of £2 (average salary £25,000).
- This would be statistically, but not practically, significant
Suppose a different treatment increased earnings by £10,000 per year with a standard error of £10,000
- This would not be statistically significant, but could be important in practice

Problems with statistical significance 2

Warning

It is a common error to summarise comparisons by statistical significance into “significant” and “non-significant” results

Non-significance is not the same as zero

Suppose an arterial stent treatment group outperforms the control
- mean difference in treadmill time: 16.6s (standard error 9.8)
- the 95% confidence interval for the effect includes zero, $p ≈ 0.20$
It’s not clear whether the net treatment effect is positive or negative
- but we can’t say that stents have no effect

Problems with statistical significance 3

The difference between ‘significant’ and ‘not significant’ is not statistically significant

At a $P<0.05$ threshold, only a small change is required to move from $P < 0.051$ to $P < 0.049$
Large changes in significance can correspond to non-significant differences in the underlying variables

Problems with statistical significance 4

The difference between ‘significant’ and ‘not significant’ is not statistically significant

At a $P<0.05$ threshold, only a small change is required to move from $P < 0.051$ to $P < 0.049$
Large changes in significance can correspond to non-significant differences in the underlying variables

Standard errors, sample size, and statistical significance

Standard errors

Important

We cannot make an infinite number of measurements of $z$. We can only take a sample.

The mean and standard deviation we estimate in an experiment will not match those of the infinitely large population.

Standard Error (of the Mean)

The standard error of the mean reflects the uncertainty in our estimate of the mean.

When estimating the mean of an infinite population, given a simple random sample of size $n$, the standard error is:

\[ \textrm{standard error} = \sqrt{\frac{\textrm{Variance}}{n}} = \frac{\textrm{standard deviation}}{\sqrt{n}} = \frac{\sigma}{\sqrt{n}} \]

When we perform an experiment we measure a limited number of values
For statistical analysis, we consider that these values are randomly drawn from a random distribution containing an infinite number of values
The mean, standard deviation, and other values we measure in our experiment will not match the true mean, standard deviation, or whatever of the full infinite distribution
We can, though, estimate our confidence in our estimate of the mean by the standard error of the mean, which reflects the uncertainty in our estimate of the true mean of the infinite distribution.
- The standard error is the square root of the variance divided by the size of the sample
- This assumes that our sample is randomly selected from the population and all samples are independent - the more you violate those assumptions, the less accurate this estimate is
- Since standard deviation is the square root of the variance, we can write this as:
The standard error of the mean is the standard deviation divided by the square root of sample size

Standard error and sample size

Tip

Uncertainty in the mean estimate $\mu$ reduces proportionally to the square root of the number of samples, $n$

To visualise this, we can plot the distribution of our uncertainty in the estimate of the mean, where we’ve estimated that the mean is zero and the standard deviation is one, for a range of sample sizes, n
At $n=3$, the uncertainty in our estimate is quite flat and broad
As we increase the number of samples $n$, the uncertainty narrows, and the density of the distribution of estimates starts to gather more around the central value of our estimate of the mean.
This property holds regardless of our assumptions about the shape of the sampling distribution
- But it only reflects our confidence in the estimate of the mean, not the shape of the underlying distribution!
- The standard error is less informative about our distribution as the sampling distribution deviates more from a normal distribution

Standard error and hypothesis testing 1

Hypothesis test statistics

Test statistic $t$ is a point on the distribution representing a significance threshold

\[ t = \frac{Z}{s} = \frac{Z}{\sigma/\sqrt{n}} \]

$Z$ is some function of the data (difference between estimate and true value); $s$ is standard error of the mean

This is true for many hypothesis test methods

Standard error and hypothesis testing 2

One-sample $t$-test

\[ t = \frac{Z}{s} = \frac{\bar{X} - \mu}{\hat{\sigma}/{\sqrt{n}}} = \frac{\bar{X} - \mu}{s(\bar{X})} \]

$\bar{X}$ is the sample mean; $\mu$ is the hypothesised population mean (being tested)
$\hat{\sigma}$ is the sample standard deviation; $n$ is the sample size; $s(\bar{X})$ is the standard error of the mean

Wald test

\[ \sqrt{W} = \frac{Z}{s} = \frac{\hat{\theta} - \theta_0}{s(\hat{\theta})} \]

$\hat{\theta}$ is the estimated maximising argument of the likelihood function; $\theta_0$ is the hypothesised value under test
$s(\hat{\theta})$ is the standard error of $\hat{\theta}$

Standard error and hypothesis testing 3

Hypothesis test statistics

Test statistic $t$ is a point on the distribution representing a significance threshold

\[ t = \frac{Z}{s} = \frac{Z}{\sigma/\sqrt{n}} \]

$Z$ is some function of the data (difference between estimate and true value); $s$ is standard error of the mean

What happens if we hold $Z$ and $\sigma$ constant and vary sample size?

Sample size and hypothesis testing 1

We reject the null hypothesis when $t > $t_\textrm{crit}$

Suppose we set $t_\textrm{crit} = 2$ (and $\sigma=1$, $n=3$)
We can calculate $t = \frac{Z}{s} = \frac{Z}{\sigma/\sqrt{n}}$ for any value of $Z$

So suppose we set a confidence interval that gives us a test statistic: t-critical=2
- So any difference $Z$ between our estimated mean and the test value giving a value of t less than 2 is “not significant” and we accept the null hypothesis
- Any difference $Z$ between our estimated mean and the test value giving a value of t greater than 2 is “significant” and we reject the null hypothesis
The plot now shows the relationship between $Z$, our measured difference from the hypothesised mean, and the value of the resulting test statistic $t$ for a sample size of three measurements
With three measurements, we need a Z value of about 1.15 or greater for the line to cross the critical value and for us to reject the null hypothesis
- If we saw a Z value of less than 1.15, we would accept the null hypothesis
- If we saw a Z value of greater than 1.15, we would reject the null hypothesis

Sample size and hypothesis testing 2

The difference ($Z$) we need to see to reject the null varies with sample size

Set $t_\textrm{crit} = 2$
$n=3 \implies Z_\textrm{crit} \approx 1.15$; $n=15 \implies Z_\textrm{crit} \approx 0.5$

Statistical significance and effect size

Statistical significance is not the point

We can meet any statistical significance threshold for a difference by sufficiently increasing the sample size

Statistical significance is not the same as practical importance

Suppose a treatment increased earnings by £10 per year with a standard error of £2 (average salary £25,000).
- This would be statistically, but not practically, significant

What matters is effect size

In all our experiments we should be concerned not with “statistical significance,” but with how likely it is that, if there is a meaningful effect of the treatment, our experiment will be able to detect it?

We need to be concerned with statistical power

Statistical power

Important

Statistical power is defined as the probability, before a study is performed, that a particular comparison will achieve “statistical significance” at some predetermined level (e.g. $P < 0.05$) given an assumed true effect size.

The process

Hypothesise an appropriate effect size (e.g. what effect will improve health?)
Determine the $p$-value threshold you consider “statistically significant”
Make reasoned assumptions about the variation in the data (e.g. what distribution? what variance?)
Choose a sample size
Use probability calculations to determine the chance that your observed $p$-value will be below the threshold (accept the null hypothesis) for the hypothesised effect size

Effect sizes

Power analysis depends on an assumed effect size

The true effect size is almost never known ahead of time
- Determining the effect size is usually why we’re doing the study

How to choose effect sizes

Try a range of values consistent with relevant literature
Determine what value would be of practical interest (e.g. improvement in outcomes of 10%)

How not to choose effect size

DO NOT USE AN ESTIMATE FROM A SINGLE NOISY STUDY!
- Noisy studies suffer from The Winner’s Curse

The Winner’s Curse 1

A low-powered pilot study

Suppose we ran a small pilot study with only a few individuals
The study, by design, has low statistical power
- The variance of the data is relatively large, compared to the true effect size

The Winner’s Curse 2

You get a statistically significant result!

You think you won, but you lost! (The Winner’s Curse)
- The estimate is either eight times too large (at least 16% instead of 2%) or
- The estimate has the wrong sign (a negative change instead of positive)

The Winner’s Curse 3

The trap

Any apparent success of low-powered studies masks larger failure

When signal (effect size) is low and noise (standard error) is high, “statistically significant” results are likely to be wrong.

Low-power studies tend not to replicate well

Warning

Low-power studies have essentially no chance of providing useful information

We can say this even before data are collected

Published results tend to be overestimates

Statistical power and ethics

It is unethical to under-power animal studies

Under-powered in vivo experiments waste time and resources, lead to unnecessary animal suffering and result in erroneous biological conclusions (NC3Rs Experimental Design Assistant guide)

It is unethical to over-power animal studies

Ethically, when working with animals we need to conduct a harm–benefit analysis to ensure the animal use is justified for the scientific gain. Experiments should be robust, not use more or fewer animals than necessary, and truly add to the knowledge base of science (Karp and Fry (2021))

So how should we appropriately power animal studies?

Statistical power and error 1

We often refer to two kinds of statistical error

Type I Error ($\alpha$)

Type I error is the probability of rejecting a null hypothesis, when the null hypothesis is true
- Also known as a “false positive error”
Represented by the Greek letter $\alpha$

Type II Error ($\beta$)

Type I error is the probability of accepting a null hypothesis, when the null hypothesis is false
- Also known as a “false negative error”
Represented by the Greek letter $\beta$

Statistical power is $1 - \beta$

Statistical power and error 2

Statistical power needs context: the expected error rates of the experiment at a given effect size, e.g.

The experiment has 80% power at $\alpha = 0.05$ for an effect size of 2mM/L

How to read this

“an effect size of 2mM/L”: we are aiming to detect an effect of at least 2mM/L (e.g. blood glucose concentration)
“$\alpha = 0.05$”: we are using a significance test threshold ($\alpha$, type I error rate) of $P < 0.05$
“80% power”: we expect the study to report a significant effect, where one truly exists, 80% of the time

Statistical power and error 3

The experiment has 80% power at $\alpha = 0.05$ for an effect size of 2mM/L

If the drug truly has no effect

The test has $\alpha = 0.05$, so we would expect to reject the null hypothesis incorrectly 5% of the time
If we ran the experiment 100 times, we would expect to see a result implying that the drug was effective five times

If the drug truly has an effect

The test has predicted power $1 - \beta = 0.8$, so the type II error rate $\beta = 0.2$ and we would expect to accept the null hypothesis incorrectly 20% of the time
If we ran the experiment 100 times, we would expect to see a result implying that the drug was effective eighty times

Statistical power and sample size 1

What we need, to calculate appropriate sample size

An acceptable false positive rate (type I error, $\alpha$)
An acceptable false negative rate (type II error, $\beta$)
- This is equivalent to knowing the target statistical power ($1 - \beta$)
The expected effect size and variance
The statistical test being performed

Important

We need this information to calculate an appropriate, ethical sample size

Statistical power and sample size 2

Typical funders’ requirements

False positive rate $\alpha = 0.05$
Power $1 - \beta = 0.8$ (80% power)
These are only a starting point - other values may be more appropriate depending on circumstance

Under experimenter control

Effect size and variance
The appropriate statistical approach

`G*Power` Walkthrough

`G*Power`

G*Power is a powerful software tool that can compute several types of power calculations for a very wide range of statistical analyses (Faul et al. (2007))

Note

It is beyond the workshop scope to consider all the possible uses of G*Power.

We will only be introducing you to basic operation of the program.

The Experiment 1

Is the average weight of a group of mice statistically different from 25g?

Which statistical test?

Means of real value measurements like height, weight, tend to follow a normal distribution. A $t$-test may be appropriate
We are testing the mean of a sample against a single hypothesised (null) value, so a one-sample $t$-test would be the appropriate variant
We want to know if the sample mean is different from the single hypothesised (null) value and are not concerned with direction, so we would use a two-tailed test

The Experiment 2

Is the average weight of a group of mice statistically different from 25g?

What do we want to know from the power calculation?

We want to determine an appropriate sample size for our experiment, given an expected effect size, variance, and choices of power and statistical significance threshold

The Experiment 3

Is the average weight of a group of mice statistically different from 25g?

What kind of power calculation should we perform?

We want to calculate an appropriate sample size for our experiment before we perform it, so we require an a priori power calculation

The Experiment 4

Is the average weight of a group of mice statistically different from 25g?

What statistical power should we use?

We will use standard funders’ requirements of 80% power at $P < 0.05$

\[\alpha=0.05, \beta=0.2, 1-\beta=0.8 \]

The Experiment 5

Is the average weight of a group of mice statistically different from 25g?

What is the expected effect size and variation in the sample?

There are no absolute values that constitute a meaningful difference in weight under all circumstances, but we will make the following assumptions

A 10% difference in weight from the expected 25g null would be meaningful, for an effect size of 2.5g
We assume that the weight of individuals in the sample has a standard deviation of 1.25g, so $\sigma = 1.25$, on the basis of having previously weighed groups of mice.

Let’s start!

Note

We now have enough information to calculate an appropriate sample size

parameters

two-tailed, one sample $t$-test
a priori power calculation to compute required sample size
$\alpha = 0.05$
Power $1 - \beta = 0.8$
Effect size = 2.5
Standard deviation, $\sigma = 1.25$
Cohen’s $d = \frac{\mu - \mu_0}{\sigma}$

Experimental Design Assistant

NC3Rs EDA

NC3Rs EDA website: https://eda.nc3rs.org.uk/

The Experiment 1

We will use NONcNZO10/LtJ mice (JAX) to represent a polygenic form of diabetes. Mice in the treatment group will receive a single subcutaneous injection of drug A (up to 30mg/kg). Mice in the control group will receive a single subcutaneous injection of vehicle. Mice will be randomly assigned to receive either drug A or vehicle only without regard to the sex of the animal. 48 hours after administration of drug or vehicle, the blood glucose level will be measured.

The Experiment 2

Our beliefs about the experiment

We are testing the effect of a new drug - drug A - on plasma glucose levels
- There is a single experimental variable of interest: whether drug A is present (treatment) or absent (control)
- The plasma glucose level is our outcome measure
Our experimental subjects are diabetic (NONcNZO10/LtJ) mice
We will divide individuals into two groups, by different pharmacological treatment
- Group 1 will receive the vehicle with no active drug (the control)
- Group 2 will receive the vehicle, containing active drug (the treatment)
Individuals will be allocated to each group randomly, by complete randomisation
We are not testing for the effect of sex on drug performance as an experimental variable of interest
- We are only monitoring for the effect of sex on drug performance as a nuisance factor

The Experiment 3

Causal relationships in the experiment

Plasma glucose level is potentially dependent on three causal relationships

The pharmacological effect of drug A
The pharmacological effect of the vehicle
Individual differences between experimental subjects (mice)

Non-causal effects

Plasma glucose level is potentially also dependent on other factors including

sex
time the drug is administered

Some of these may be systematic and important (nuisance variables), and others random and not so important (“noise”)

The Experiment 4

Treatment vs no treatment

The causal influence of administering drug A (or not) on plasma glucose levels of the experimental subjects is the central reason for performing the experiment

Presence or absence of drug A is our independent variable (aka explanatory variable)

Independent variables

The presence/absence of drug A is an independent variable as whether it is administered or not is entirely under our control, as experimenters.

The Experiment 5

Control vs treatment groups

We cannot administer the drug in isolation!

Like most drugs, drug A is carried in a vehicle - a substance expected to be inert in the contect of the treatment.

Controlling for variables

It would be unwise to assume we could administer drug A alone to one group of mice and apply no intervention in the other group.

Physiological responses may be affected by: - the vehicle - the procedure (e.g. injection)

So we must follow the same protocol in the control and treatment groups, with the only variable being presence/absence of drug A.

The Experiment 6

Random assignment to groups

Individual mice may respond differently to drug A, the vehicle, and the procedure, due to underlying differences between them.

We therefore need to randomise subjects into experimental groups, assuming that all individuals are drawn from the same population (a single pool whose response varies according to some common distribution).

This means we expect the underlying plasma concentrations, and responses to treatment, in each group to be comparable.
Differences in measured outcome between groups are then expected to derive only from our experimental interventions and presence/absence of drug A

An EDA diagram

Figure 3: EDA diagrams are composed of nodes representing aspects of an experiment, and links representing relationships between the nodes

Our target diagram

Figure 4: The final EDA diagram describing our experiment

Summary

Power calculations

Power calculations describe what can be learned from statistical analysis of an experiment

We defined statistical power, and how statistical power tells us what conclusions can reasonably be drawn from statistical analysis of an experiment.
- The goal of experimental design is not to attain statistical significance with some (high) level of probability. The point of designing an experiment and conducting power analysis is to have a sense, both before and after data have been collected, of what can reasonably be learned from the statistical analysis.

G*Power

Using G*Power we worked through a practical example of how to estimate sample size for a desired statistical power

Experimental design

The NC3Rs EDA can help us design better experiments

We worked through the complete design, critique, and analysis of an experiment using the NC3Rs EDA tool, including how to share the resulting design with others.

What next?

Experimental design is a large and diverse field, and to become competent in experimental design takes time, and requires practice (which also involves making mistakes, at times).

What no-one tells you

Even experienced scientists with many publications and long track records of research funding may never have received any formal training in experimental design. This can lead to entrenched and outdated ideas being propagated.

It is always advisable to consult with funding bodies and specialists in statistics, experimental design, and research ethics.

References

Bate, Simon T., and Robin A. Clark. 2014. The Design and Statistical Analysis of Animal Experiments. Cambridge University Press.

Faul, Franz, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. “G*Power 3: A Flexible Statistical Power Analysis Program for the Social, Behavioral, and Biomedical Sciences.” Behav. Res. Methods 39 (2): 175–91.

Karp, Natasha A, and Derek Fry. 2021. “What Is the Optimum Design for My Animal Experiment?” BMJ Open Sci. 5 (1): e100126.

Kilkenny, Carol, William J Browne, Innes C Cuthill, Michael Emerson, and Douglas G Altman. 2010. “Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research.” PLoS Biol. 8 (6): e1000412.

Kilkenny, Carol, Nick Parsons, Ed Kadyszewski, Michael F W Festing, Innes C Cuthill, Derek Fry, Jane Hutton, and Douglas G Altman. 2009. “Survey of the Quality of Experimental Design, Statistical Analysis and Reporting of Research Using Animals.” PLoS One 4 (11): e7824.

MP968 Experimental Design Workshop

Why do we need experimental design?

We should not cause unnecessary suffering

Challenge

How many individuals?

Design experiments to minimise suffering

The 2009 NC3Rs systematic survey

The importance of experimental design

Causes for concern 1

Causes for concern 2

Very strong cause for concern

The ARRIVE guidelines

ARRIVE guidelines highlightes

A vital step

Some Statistical Concepts

Random variables

Probability distributions

Understanding variance

Standard deviation

Normal Distribution 1

Normal Distribution 2

(Non-)Normal Distribution 3

Binomial Distribution 1

Binomial Distribution 2

Poisson distribution 1

Poisson distribution 2

Binomial and Poisson distributions

Distributions in Practice

Normal Distribution Redux

Estimates, standard errors, and confidence intervals

Parameters

Estimands

Standard Errors and Confidence Intervals

Statistical significance and hypothesis testing

Statistical significance 1

Statistical significance 2

A simple example: The experiment

A simple example: The hypotheses

A simple example: The distribution 1

A simple example: The null hypothesis

A simple example: The estimated difference

A simple example: A significance threshold

A simple example: Compare the estimate

A simple example: Another threshold

A simple example: Another outcome

A simple example: What changed?

Tailed tests: two-tailed

Tailed tests: one-tailed (left)

Tailed tests: one-tailed (right)

Problems with statistical significance 1

Problems with statistical significance 2

Problems with statistical significance 3

Problems with statistical significance 4

Standard errors, sample size, and statistical significance

Standard errors

Standard error and sample size

Standard error and hypothesis testing 1

Standard error and hypothesis testing 2

Standard error and hypothesis testing 3

Sample size and hypothesis testing 1

Sample size and hypothesis testing 2

Statistical significance and effect size

Statistical power

Statistical power

Effect sizes

The Winner’s Curse 1

The Winner’s Curse 2

The Winner’s Curse 3

Statistical power and ethics

Statistical power and error 1

Statistical power and error 2

Statistical power and error 3

Statistical power and sample size 1

Statistical power and sample size 2

G*Power Walkthrough

G*Power

The Experiment 1

The Experiment 2

The Experiment 3

The Experiment 4

`G*Power` Walkthrough

`G*Power`