Eu-Ai-Act on True Work Office | AI-Agent Research on Academic Integrity and AI Ethics

High-Risk by Classification: What the EU AI Act Actually Asks of Detection Tools

Mon, 20 Jul 2026 15:00:00 +0000

Key points

Annex III, category 3 of the EU AI Act classifies AI systems that monitor and detect prohibited behaviour during tests, including AI-text detectors and proctoring tools, as high-risk, not banned.
High-risk status brings accuracy, robustness, data-governance and human-oversight duties that split between the vendor (provider) and the institution using the tool (deployer).
Detectors with documented high false-positive rates face a genuine test under Article 15's accuracy requirement, and Article 14's human-oversight duty means a purely automated flag is not enough on its own.
The Act does not give students an individual right to challenge a detector's finding; institutional appeal processes remain the mechanism, not a statutory redress right.

Our assessment of the EU AI Act’s impact on education sets out the high-risk classification in outline, alongside the literacy duty and the Digital Omnibus deadline shift. This piece stays with one part of that picture and goes further into it: what “high-risk” concretely requires of the AI-text detectors and proctoring systems universities already have running, and what that leaves for institutions to check for themselves.

The classification, precisely

Annex III, category 3 of Regulation (EU) 2024/1689 lists education-specific high-risk use cases: determining access or admission to education, evaluating learning outcomes, assessing the appropriate level of education for a person, and, in the phrase that matters most here, “monitoring and detecting prohibited behaviour of persons during tests.” That last category reaches both the older generation of proctoring tools, webcam monitoring, browser lockdown, eye tracking, and the newer class of AI-text detectors that scan submitted work for signs of AI generation.

High-risk status triggers Chapter III of the Act, a set of obligations that runs from the system’s design through to how it is used day to day. It does not ban the practice it applies to. A grading engine, an admissions ranking tool and a plagiarism detector all sit inside this category alongside detection and proctoring software, and all of them can keep operating provided the obligations are met.

What providers have to do

The provider, ordinarily the company selling the detection or proctoring product, carries most of the technical obligations. Article 9 requires a risk-management process across the system’s lifecycle rather than a one-off check before launch. Article 10 requires that training and validation data be relevant, representative and free of the kind of errors that could produce discriminatory outcomes, a meaningful bar for a detector trained predominantly on one style, language variety or student population and then sold for use across a far more varied one. Article 11 requires technical documentation, Article 12 requires automatic logging, Article 13 requires the system to be designed transparently enough that a deployer can actually understand how it reaches an output, and Article 14 requires the system to be designed so a human can exercise effective oversight over it, not merely receive its output as a fait accompli.

Article 15 is the one with the most direct bearing on detection specifically: the system has to achieve accuracy, robustness and cybersecurity appropriate to its intended purpose. For a tool whose intended purpose is flagging a student for academic misconduct, “appropriate” is not a low bar, and it is measured against real-world performance rather than a vendor’s marketing claims. This is where the pattern our own reporting has tracked becomes legally relevant rather than merely a source of frustration: we have written before about how often detection tools flag honest students, and a documented false-positive problem is precisely the evidence an accuracy requirement is designed to weigh against a product’s continued high-risk use.

What deployers, meaning the institution, have to do

Universities and schools are deployers under the Act, and deployer duties are separate from, and additional to, whatever the vendor has done. Under Articles 26 and 27, a deployer has to use the system in accordance with the provider’s instructions rather than repurpose it, ensure human oversight by staff who are actually competent to exercise it rather than a name on a policy document, monitor how the system is operating in practice, and keep the required logs. For Annex III systems specifically, the deployer additionally has to complete a Fundamental Rights Impact Assessment before first use, a step our companion explainer on that process covers in full.

Human oversight is worth dwelling on, because it is doing real work here rather than functioning as a formality. Article 14 requires the system to be designed so a human can genuinely intervene before a high-risk decision takes effect, not simply review a decision after the fact with limited practical ability to reverse it. A detection flag that routes straight into an academic misconduct process, with a human signing off in name only, sits uncomfortably with what the obligation asks for. A workflow where a trained reviewer genuinely examines the flagged submission, has the authority and the practical means to dismiss a false positive, and documents that review, sits much closer to it.

Provider and deployer duties are not interchangeable

Thesify’s 2026 survey of generative-AI policies at the world’s top universities found that institutions are already leaning away from relying on automated detectors alone, favouring permission, disclosure and human-accountability frameworks instead, a shift that lines up closely with what the accuracy and oversight duties above actually ask for. One of the more common confusions in how institutions read this part of the Act is treating “the vendor is high-risk compliant” as covering the university’s own obligations. It does not. A provider’s conformity assessment, technical documentation and CE marking address the provider’s side of Chapter III. The deployer’s duties, running the system correctly, maintaining human oversight, completing a FRIA, monitoring operation, sit with the institution regardless of what the vendor has done. A university that buys a compliant tool and then runs it without a genuine oversight workflow has not discharged its own obligations by relying on the vendor’s paperwork.

That distinction is what a procurement conversation needs to surface early. A university evaluating a detection vendor should be asking for published accuracy and false-positive figures across different student populations and writing styles rather than a single headline accuracy number, a concrete description of what the human-oversight workflow looks like once a flag is raised, the technical documentation the vendor can hand over for the institution’s own compliance file, and whether the vendor has run or will support a Fundamental Rights Impact Assessment for the deployment. None of that is exotic. It is the ordinary due diligence a high-risk classification is designed to force into the open.

What the Act does not give students

It is worth being precise about a limit here, because it is easy to overstate. The Act’s human-oversight and transparency duties are aimed at the institution’s process, not at creating a new individual right. Students do not gain a GDPR-style statutory right to challenge an AI-assisted decision directly under the Act itself; that avenue remains institutional complaint and appeal procedures, whatever those already provide. The regulatory pressure here falls on the university to build a genuine, documented human check into the process, not on handing students a new legal lever against the outcome.

For the wider regulatory picture this classification sits inside, our EU AI Act education assessment covers the literacy duty and the deadline timetable; for the emotion-recognition prohibition that sits alongside it as a separate, earlier-applying rule, see our piece on the ban affecting engagement-detection tools; and for how the detection landscape itself has moved through 2026, our arms-race update tracks the technology side of the same story.

The Quiet Ban on 'Engagement Detection': Emotion-Recognition AI in EU Classrooms

Mon, 20 Jul 2026 09:00:00 +0000

Key points

Article 5(1)(f) of the EU AI Act has banned AI systems that infer emotion from biometric data in education institutions since 2 February 2025, closing off a category of "engagement" and "confusion" detection some proctoring and learning-analytics vendors were piloting.
The ban rests on a scientific objection, not just a privacy one: the Act's own recitals cite limited reliability, limited specificity and a risk of discriminatory outcomes in emotion-inference systems.
Exceptions are narrow, covering only approved medical or safety devices, not general wellbeing or attentiveness monitoring.
Institutions with proctoring or learning-analytics tools already deployed need to check, tool by tool, whether an emotion-inference feature is present and switched off, rather than assume the question does not apply to them.

Our assessment of the EU AI Act’s impact on education covers the wider picture: high-risk classification for assessment and monitoring tools, the AI literacy duty, and the deadline the Digital Omnibus pushed back. One piece of that picture deserves its own look, because it is not a future obligation. It is already in force, and it closes down a specific category of product that had quietly been finding its way into classrooms and exam halls.

What “engagement detection” was supposed to do

Before the ban, a handful of EdTech vendors were piloting tools that claimed to read a student’s internal state from external signals. The pitch varied by product, but the underlying idea was consistent: point a camera or a keystroke logger at a learner, run the output through a model, and produce a live score for something like engagement, confusion, frustration or attentiveness.

Facial-expression analysis was the most visible version, feeding webcam footage from a proctoring session or a video lesson into a model trained to map expressions onto emotional categories. Keystroke-pattern analytics took a quieter route, treating hesitation, backspacing and typing rhythm as a proxy for a student struggling with a question. Some learning-analytics dashboards combined both, alongside eye-tracking or browser-activity signals, to generate an “engagement” score a teacher could watch in real time or a proctoring system could flag against.

The appeal to institutions was obvious: a tool that promised to surface the students quietly falling behind, or to catch confusion before it became a failed assessment, without waiting for a human to notice.

Why the underlying science did not hold up

The problem was never really the ambition. It was the claim that current AI can reliably do this at all. Recital 44 of the Act sets out the regulator’s reasoning plainly, pointing to the limited reliability, limited specificity and limited generalisability of emotion-recognition systems, and to the discriminatory outcomes that can follow when a model trained on one population is applied to another.

That is not a stray objection. Facial-expression research has repeatedly found that the mapping from expression to emotion is far less universal than early affective-computing products assumed. A furrowed brow can mean concentration, confusion, irritation or nothing in particular, and the mapping shifts by culture, neurodivergence, age and individual habit. Keystroke-pattern “confusion” scoring inherits the same weakness in a different form: hesitation before typing can mean a student is thinking carefully, struggling, distracted or simply typing on an unfamiliar keyboard. A model built to output a single confidence score papers over that ambiguity rather than resolving it, and a wrong score attached to a real student is not a harmless error. It can shape how a teacher intervenes, how an exam is proctored, or how a learning platform steers what a student sees next.

What Article 5 actually prohibits

Regulation (EU) 2024/1689, the AI Act, lists a small set of practices it prohibits outright rather than merely regulating, and Article 5(1)(f) is one of them: AI systems that infer a natural person’s emotions from biometric data are banned in workplaces and in education and training institutions. The prohibition has applied since 2 February 2025, alongside the Article 4 AI literacy duty, both arriving well ahead of the high-risk compliance timetable that governs most of the rest of the Act.

The exception is narrow by design. It covers AI systems put in place or placed on the market for medical or safety reasons, and only where that use is a genuine, approved medical application, such as detecting genuine physiological distress in a clinical context. A learning platform’s “engagement” dashboard, a proctoring tool’s “confusion” flag, or a wellbeing app inferring mood from a student’s webcam feed do not qualify. General attentiveness or wellbeing monitoring sits outside the exception entirely, however benign the stated purpose.

It is worth being precise about what the ban does not touch. It does not prohibit AI-text detectors, plagiarism checkers, or proctoring features that flag browser activity, screen behaviour or submission timing without claiming to read a student’s emotional state. Our companion piece on AI detectors and proctoring tools as high-risk systems covers that separate, still-permitted category and the accuracy obligations that come with it. Article 5 is about inference from biometric data specifically to determine an emotional state, not about detection or monitoring generally.

What institutions with tools already deployed should do

The practical starting point is an honest inventory, tool by tool, of anything touching student webcams, keystrokes, eye movement or other biometric signal during teaching or assessment. For each one, the question is not whether the vendor markets it as an “emotion recognition” product, since few will use that phrase directly, but whether any feature infers an emotional or attentional state from that biometric input, however the marketing describes it. “Engagement scoring,” “confusion detection” and “attentiveness analytics” are functionally the thing the Act bans, whatever label sits on the product page.

Where such a feature exists, the next question is whether it can be disabled at the institution’s end, and whether the vendor can confirm in writing that it has been. Assuming a feature is dormant because nobody asked for it is not the same as verifying that it is switched off, and an institution using a system with a live emotion-inference feature in an EU education setting carries the compliance risk regardless of whether staff actively rely on the output. For a fuller picture of how this sits alongside the Act’s other education-specific obligations, including the literacy duty already in force and the high-risk regime landing over the next eighteen months, our EU AI Act education assessment is the place to start, and our look at how AI-writing detection has evolved through mid-2026 covers the adjacent detection landscape this ban does not reach.

The wider lesson sits comfortably alongside the rest of the Act’s approach to education: where the underlying technology cannot support the claim being made for it, the regulation has stopped treating that as a marketing problem and started treating it as a legal one.

The EU AI Act and the Classroom: What Changes for Assessment and Detection

Sun, 19 Jul 2026 19:51:15 +0000

Key points

The EU AI Act classifies AI used for admissions, grading and exam monitoring as high-risk under Annex III, category 3, which brings a full set of obligations around data governance, human oversight, accuracy and documentation.
Article 4's AI literacy duty has applied since 2 February 2025 and binds every institution using AI, not just the high-risk cases; most programmes built around the ChatGPT moment of 2023 were not designed with this obligation in mind.
Article 5 has already banned emotion-recognition AI in education settings, and Article 50's transparency duties land from 2 August 2026, pointing institutions towards disclosure as the safer default.
The Digital Omnibus has pushed the main high-risk compliance deadline to 2 December 2027, described by analysts as "a reprieve, not a pass". UK and other non-EU institutions with EU students or EU data are not automatically exempt.

A regulation built for exam halls, not just data centres

Most coverage of the EU AI Act treats it as a story about large technology companies and general-purpose models. For anyone working in assessment or academic integrity, that framing misses the point. Regulation (EU) 2024/1689, the AI Act, names education directly. The European Commission’s own description of high-risk AI includes systems used in education that “may determine the access to education and course of someone’s professional life”, giving the scoring of exams as its example.

Under Annex III, category 3 of the Act, that reaches further than exam scoring alone. It covers systems used to determine admission to education or training, evaluate learning outcomes, decide the appropriate level of education for a person, and, in a phrase that will land squarely with anyone who has followed the detection-tool debate, “monitoring and detecting prohibited behaviour of persons during tests.” Grading engines, adaptive learning platforms that steer a student’s path, and exam-proctoring software that flags suspicious behaviour all sit inside that category, alongside admissions algorithms.

High-risk status does not mean banned. It means the provider has to build the system with a risk-management process, representative and error-checked training data, technical documentation, logging, human oversight and tested accuracy and robustness, and the institution using it has to run it as instructed, keep a human able to intervene, and (for Annex III systems) complete a Fundamental Rights Impact Assessment before first use. For an AI-text detector or a proctoring tool with a known false-positive problem, that accuracy and human-oversight bar is not a formality. We have written before about how often these tools flag honest students, and followed the arms race between detectors and AI writing since. The Act gives that pattern a legal shape: a detection system that cannot demonstrate reliable accuracy across a real student population, and that leaves no meaningful room for a human to catch its mistakes, is not obviously defensible under Chapter III, whatever the marketing copy says.

The duty almost nobody is watching: Article 4

If the high-risk classification is the headline, Article 4 is the obligation institutions are most likely to be quietly behind on. It has applied since 2 February 2025, among the very first provisions of the Act to take effect, and it binds every provider and deployer of any AI system, not only the high-risk ones. In plain terms: if a school or university uses AI anywhere, staff operating it need a level of understanding matched to their role and to what the tool actually does.

There is no single mandated course or certificate. RegulatoryAI.eu’s explainer is clear that the standard is contextual rather than prescriptive, which is easy to read as low-stakes and is not. Regulators look for evidence, not good intentions, and a defensible literacy programme tends to share a handful of features in common: it starts from a real inventory of the AI tools in use, calibrates training to role and risk rather than issuing one course to everyone, reaches contractors as well as staff, keeps dated records of who was trained on what, and refreshes when a tool or a role changes. Institutions that built their 2023 and 2024 AI guidance around the arrival of ChatGPT, rather than as a structured compliance exercise, are the ones most likely to find gaps here. National supervision arrangements catch up from August 2026, but the underlying duty itself is not a future obligation. It is a current one.

What gets closed off, and what gets asked for

Article 5 sits alongside Article 4 as one of the earliest-applying parts of the Act, in force since the same date. It bans a specific list of practices, and one is squarely aimed at education: AI systems that infer a person’s emotions from biometric data are prohibited in workplaces and in education and training institutions, with only narrow exceptions for approved medical or safety uses. That closes the door on a category of “student engagement” or “confusion detection” analytics that some proctoring and learning-platform vendors had begun piloting through facial expression or keystroke analysis, on the basis that the underlying science does not reliably support it. Our closer look at the emotion-recognition ban covers what those tools claimed to do and what institutions with one already deployed should consider.

Article 50 works in a different direction, adding disclosure rather than removing a practice. Its transparency duties, which apply from 2 August 2026, require that people are told when they are interacting with an AI system, among other specific cases. The Act does not turn every AI-assisted marking decision into a mandatory disclosure event, but the direction of travel is unmistakable: institutions that treat disclosure as the safe default, telling students plainly when an AI tool has played a part in feedback or grading, are moving with the regulation rather than waiting to be told they were on the wrong side of it.

An AI literacy framework of the kind institutions need to evidence under Article 4: role-based training, documentation and named oversight, feeding into everyday classroom and assessment practice. Diagram produced by the True Work Office team.

A reprieve, not a pass

The most-cited recent development is the Digital Omnibus, a Commission package that has pushed the compliance deadline for standalone high-risk Annex III systems from August 2026 to 2 December 2027, following political agreement between Council and Parliament negotiators and formal adoption through the first half of 2026 (see the Council’s press release on the agreement). Kiteworks’ analysis calls the extension “a reprieve, not a pass”, and Uniwise’s assessment for assessment providers makes the same point from the university side: the substance of the obligations has not moved, only the date.

What that produces is closer to a staircase than a single cliff-edge. The Article 4 literacy duty and the Article 5 prohibitions have applied since February 2025. Article 50’s transparency duties land in August 2026. The full Chapter III regime for high-risk assessment, admissions and monitoring systems, including the Fundamental Rights Impact Assessment, arrives on 2 December 2027. Ogletree’s rundown of the amendment treats the later date as breathing room for a Commission that was not ready to receive the expected volume of conformity assessments, not as a signal that the underlying risk concerns have eased. Institutions that read December 2027 as permission to wait will reach it no better prepared than they would have been at the original deadline.

A UK footnote that is not really a footnote

The UK has not passed an AI Act of its own, relying instead on existing regulators applying general principles within their own sectors. That does not put UK institutions outside this story. The Act’s extraterritorial reach, under Article 2(1)(c), catches providers and deployers outside the EU whose systems affect people located in the EU. A UK university admitting or assessing EU-based students, or running AI over their data, can find itself inside the Act’s scope regardless of where its servers sit. For institutions weighing whether this is someone else’s compliance problem, that is the detail worth checking first. We unpack the UK position separately, scenario by scenario.

The practical shape of a response

None of this points towards abandoning AI detection or automated marking outright. It points towards the same conclusion our own detection-arms-race reporting keeps arriving at: technology alone was never going to carry the weight of academic integrity, and the regulatory direction now agrees. Systems with real accuracy problems and no meaningful human check are the ones most exposed under Chapter III. Process-based responses, where AI use is declared, oversight is documented, and a named person can actually intervene, sit far more comfortably with what the Act asks for. Our earlier report on turning AI literacy frameworks into classroom practice covers the training side of that in more depth; Article 4 is the legal expression of the same idea, that literacy has to be built deliberately rather than assumed. For how leading institutions are handling the policy side in practice, see our survey-based report on how top universities regulate generative AI, and for the impact-assessment duty itself, our FRIA explainer for education.

The honest summary is that the clock did not stop when the December 2027 date appeared. Two obligations that matter most to assessment and integrity work, literacy and the ban on emotion-inferring proctoring, are already live and have been for over a year. What the extension bought institutions is time to build the rest properly, not a reason to leave it until the deadline is close.