Black Box Assessment

§01 The starting point

A structural answer to a wicked problem

Step 6 describes AI and assessment as a wicked problem, in the sense Corbin, Bearman, Boud and Dawson (2025) give the term, and Winstone, Gravett and Elkington start from the same place. The problem keeps shifting as the technology develops, and it is tangled up with academic workload, timetabling and governance. For some teachers, the authors note, uncertainty about acceptable AI use has led to inertia, even paralysis, although those teachers could see that validity was at risk.

The authors’ answer is to work from principles. Principles are loose enough to revise, can be adapted to different contexts and work as a compass when you have to decide where to make a trade-off. In that way they match the three permissions in Step 6, to adjust as you go, to choose differently and to compromise.

As Step 1 showed, rules about AI use can’t be enforced where nobody is watching. The article adds two more problems, citing Corbin, Dawson and Liu (2025). Rules put the burden on students, and nobody knows whether students have understood or followed them. Black Box Assessment is a structural change. It alters what students hand in and what earns credit, so the boundaries of AI use follow from the design itself.

§02 The metaphor

Where the black box comes from

The image is older than generative AI. Black and Wiliam (1998) used it about the classroom, where teaching and tasks go in and student performance comes out, while what happens in between stays hidden. Formative assessment was their way of opening the box. More recently the black box has been widely used to describe how opaque AI technologies are (Bearman and Ajjawi, 2023).

Winstone and colleagues turn it on student work. Teachers have no direct access to the cognitive and creative processes behind a submitted piece of work, and that opacity is often framed as a threat to integrity. They see a paradox here, because the same opacity points to a design principle. If you stop assessing only the polished output, you can assess the paths through which the work came into being.

Full visibility isn’t the aim. Processes are rarely fully explainable, as Bearman and Ajjawi point out, but you can catch glimpses of them, and those glimpses are the windows.

Winstone and colleagues admit that assessing process alongside product is not radical. It was discussed long before GenAI, in experiential and problem-based learning among other places. Their new lens is on errors, failed approaches and cognitive change, which is where Matthew Syed’s Black Box Thinking (2015) and its flight recorder come in.

§03 Errors

Why errors carry the model

Bringing errors into the open, as Step 3 put it, is only part of what the article asks. Errors, false starts and changes of direction should be neither hidden nor simply disclosed. They should be “recognised, evaluated, and credited alongside polished final products” (p. 7). In the authors’ example, a student who began with one argument, found it too simplistic and moved to another earns credit for that reflection and growth, and the first misstep doesn’t count against them. Working with error becomes an outcome in its own right.

Without credit for errors, they warn, process assessment risks becoming performative, with a polished product and a tidy account of its development written afterwards. That is why they treat normalising error and assessing process as bound together. The same reasoning lies behind their warning that chances to fail can be tokenistic if they are confined to low-stakes activities while the credit-bearing assessments still reward only polished outcomes. That reinforces one-shot, risk-averse learning.

The article lists several gains from engaging with error.

§04 Design

Reworking a task you already have

You don’t need a new task to start. Many tasks that have lost validity still have value, and you can keep them and change how they are assessed. You may need to reframe a learning outcome or add one, since at least one outcome should be about how students engage with and develop their work. The article also raises the possibility of windows that are never graded and serve only to validate that the product is the student’s own.

Our addition. On their own, windows are weak evidence, as Step 3 said, because AI can produce them too. Ungraded windows used to check authorship are best treated as material for a conversation with the student.

Choosing the windows

More windows can strengthen validity, but beyond two or three they risk overwhelming students. Natural traces can be draft histories or screencasts of the work. Students can negotiate the windows with you or pick from a set you offer. The windows should also match how people in your discipline work.

In essence, every human activity has a black box: the often invisible chain of cognitive, creative, and practical actions that lead to an outcome.

The article’s examples include these.

• lab notebooks, code commits, error analyses and debugging logs in STEM
• drafts, annotated bibliographies and reflective journals in the arts and humanities
• case notes, clinical reasoning logs, policy drafts and design iterations in professional fields such as medicine, law and education

A window can also document how the student used GenAI, with prompts, the reasons for them, critiques and ethical considerations. AI use then becomes a transparent and assessable part of the learning process.

Our addition. In Step 2 the student could work alone, get help from AI or hand work over, in no fixed order. An AI window should leave room for all three, handed-over parts included, and ask what the student came to understand. That understanding is the constant across the roles.

Criteria and rationale

Existing rubrics may need revising so that they explicitly reward process transparency, iteration and learning from mistakes. Tell students why the process counts, or producing process evidence can turn into busywork. The article’s online supplementary materials include an indicative taxonomy of windows and example criteria from several disciplines.

§05 Objections

The four objections, at more length

The authors call the GenAI challenge arguably the biggest disruption to educational assessment since the COVID-19 pandemic, and its scale, they argue, makes doing nothing untenable. The short versions of their answers to the objections are in Step 3.

§06 Read it for what it is

A design proposal

The article is a conceptual blueprint and reports no empirical study, so it can’t tell you what the model does to learning or workload. The authors call it a conceptual toolkit, one of many possible approaches to a wicked problem, and they say that the effects of assessment changes made in response to GenAI are hard to predict. They also disclose that ChatGPT helped them generate the disciplinary examples.

Not everyone has the same room to fail, as Step 3 noted. Bringing error into assessment, the article argues, requires wider systemic change. Failure is bound up with power and privilege, and its consequences can stick to some students more than to others. Students are also likely to notice the gap between a teacher who calls failure generative and an education system that stigmatises it.

Why read it anyway? It gives you a concrete and flexible way to change an assessment you already have without starting from scratch.