The International Baccalaureate Examinations: Item Response Theory

To quote Wikipedia, item response theory is "a paradigm for the design, analysis, and scoring of tests, questionnaires, and similar instruments measuring abilities, attitudes, or other variables." These "other variables" are what we call latent traits - traits that cannot be measured, in contrast to traits like height and weight which can be quantified.

This theory is named after the fact that it examines the individual items, or questions, on the test, and is regarded as superior to the classical test theory which focuses on the entire test as a whole.

Item Response Model

$$P(\theta) = c+\frac{1-c}{1+e^{-a(\theta-b)}}$$

This model is called the 3-parameter logistic model (3PL) which represents the probability $P$ of getting a question correct with parameters $a$, $b$, $c$ for a student with an ability of $\theta$.

• $\theta$ is the independent variable which represents student ability
• $P$ is the dependent variable and represents the probability of getting the question correct, hence the values for $P$ are constrained from 0 to 1.
• $a$, $b$ and $c$ are question parameters that altogether affect the probability $P$
• $a$ represents the discrimination index which affects the slope around the inflexion point of the graph. This is a measure of how well the question is able to distinguish between candidates. For example, if a particular question was too hard that everyone got it incorrect, then the discrimination index is low. On the other hand, if a question was just hard enough such that only the top students got it correct, then the discrimination index is high. Note that $a$ must be positive (what happens if $a$ is negative?)
• $b$ represents the difficulty index which affects the horizontal shift of the graph. A harder question will be shifted to the right such that the skill required to have a 50% probability of answering the question is higher than it would be if the graph was shifted to the left.
• $c$ represents the guessing parameter which simply reflects the number of choices on a multiple-choice test. For example, on the SAT, which has 4 answer choices per question, $c$ is simply $0.25$. Note that for a written response question, this parameter is simply zero. Since this is a probability, the constraint is $0 \leq c \leq 1$.

It might sound super confusing, but please open the embedded Desmos graph below and play around with the sliders to experiment how the different parameters affect the shape of the graph. Try and relate it back to the real world context - why does changing this parameter in this way affect the graph in this manner?

There are other forms of the item response model that you may want to read more about.

Investigating the Logistic Model with Calculus

You can take $\lim_{\theta \to -\infty}P(\theta)$ to show that the variable $c$ indeed represents the horizontal asymptote as $\theta$ approaches negative infinity. Note that you can interpret $\theta$ approaching negative infinity as saying that the question respondent has no ability at all, so that the probability of getting the question correct is just the probability of getting the question correct by guessing, which is represented by the parameter $c$.

You can differentiate $P(\theta)$ to explore the various behaviour of the function such as showing that $P(\theta)$ is always increasing (since $P'(\theta)$ is always positive), the effect on the inflexion point as the various parameters are changed, etc.

Challenge: By exploring the parameters, find an expression for the point of inflexion of the curve (yes it's on the embedded graph above, but this time, use maths to actually derive it!)

Maximum Likelihood Estimation

You may have noticed that both the student ability $\theta$ and the values of the parameters $a$, $b$, $c$ are unknown. In fact, this is exactly like a chicken and egg problem - if we know either the question's parameters or the student's abilities, we can work out the other, but we know neither!

This is where maximum likelihood estimation comes in, a method of calculus (optimisation) and statistics. It works by choosing arbitrary values which are then calibrated iteratively using the model.

1. Randomly choose reasonable values of $\theta$, the student ability
2. Estimate question parameters
3. Use these values to reestimate $\theta$ values
4. Repeat!

In step 3, the parameters are optimised by employing the iterative Newton-Raphson method, and the goodness of fit is measured using the chi-squared test.

You can read more about this method in this link.

The IB

Below is the examiner's comment on a question from the multiple choice section on a chemistry paper.

The question proved surprisingly challenging, as indicated by a high number of blank responses and a difficulty index of 55%. This would seem to indicate that a disturbing number of candidates are not aware of the charges on the common ions. It was however a good discriminator with a discrimination index of 0.55.

As you can see, the quote mentions a "difficulty index" and a "discrimination index", which led me to believe that the IB is incorporating IRT into their evaluation. As an IB student myself, it is fascinating to see how even the IB is using mathematical models to design and evaluate their exams!