Privacy-Preserving Oral Verification of Submitted Work

A local-first workflow for education, vocational training, and professional certification

When a polished submission no longer proves competence

AI can help a learner produce convincing code, calculations, reports, notebooks, and explanations. That assistance may be permitted and useful. The problem begins when an education or certification provider can no longer tell whether the submitted artefact is accompanied by the understanding required for a qualification.

This matters acutely to SMEs. A small training provider cannot manually design an individual oral examination for every participant. At the same time, allowing a serious gap in competence to pass unnoticed can place a future employer, customer, production system, or the learner themselves at risk—especially in software administration, cybersecurity, industrial automation, engineering, laboratory work, and other technically consequential fields.

This workflow locally separates identities from submissions and uses an AI coding agent to prepare questions about each learner's actual work. A qualified instructor asks the questions and evaluates the answers. The workflow does not detect cheating, prove authorship, assign a grade, or make a misconduct finding. It makes focused human verification practical before a credential is awarded.

Overview

A local-first workflow that separates participant identities from submitted artefacts, prepares submission-specific oral-verification questions in a pseudonymised agent workspace, and restores identities only in private instructor outputs. It helps education and certification providers verify demonstrated understanding without relying on AI-authorship detectors or automated grading.

Outcome

Instructors receive focused questions and expected explanations grounded in each participant's actual submission. This makes individual oral verification practical at cohort scale while retaining human assessment authority and reducing unnecessary disclosure of directly identifying information.

Service Card

Category	AI products
Domain	Educational Technology, Competence Verification, Privacy-Preserving AI
Vertical	Education, Vocational Training and Professional Certification
Target users	Vocational Schools, Professional Certification Providers, Technical Academies, Universities, Continuing-Education Providers, Industry Training Organisations, SMEs Running Bounded Internal Training
Service type	Local-First AI-Assisted Human-Review Workflow / CLI Toolkit
Outputs	Pseudonymised agent workspace; submission associations; evidence-grounded oral questions; expected explanations for instructors; preprocessing report; private named instructor reports.
Requirements	Moodle/Canvas-style submissions, a local participant roster, assignment specification, supported readable artefacts, an isolated agent environment, approved data-processing arrangements, and a qualified human instructor.
Limitations	The workflow does not detect cheating, prove authorship, assign grades, or make misconduct findings. Pseudonymised submissions remain personal data. Identifier removal is best-effort, opaque documents are excluded by default, and every generated question requires human verification. Employment decisions are outside the intended use.
Availability	Research MVP available for controlled institutional pilots
Licensing	MIT License for the workflow software; external models and coding-agent services retain their own terms.

The problem: production is easy; demonstrated understanding is not

Take-home assignments are valuable because learners can solve realistic problems, use professional tools, research alternatives, and revise their work. Generative AI does not remove that value. It does, however, weaken a long-standing assumption: that a technically impressive submission necessarily reflects the submitter's own understanding.

Banning AI is neither realistic nor always educationally desirable. AI-assisted work can still demonstrate strong judgement when the learner can explain the design, defend parameter choices, diagnose failure cases, and modify the solution. Conversely, a weak learner may submit an excellent-looking artefact without understanding its central mechanism.

Generic oral questions do not solve this well. They reward memorised theory and may miss the exact parts of the submission that deserve examination. Designing individual questions manually is effective but expensive, particularly for small vocational academies, certification providers, continuing-professional-development programmes, and specialist courses with limited teaching staff.

The workflow changes the unit of verification. Instead of asking whether a detector thinks the work “looks AI-generated”, it asks:

Can the learner explain the concrete decisions, results, anomalies, and failure modes present in the work submitted under their pseudonym?

An SME-focused illustration

Consider a private vocational-training provider certifying technicians who will maintain networked industrial systems. A participant submits a polished automation project with configuration files, tests, and a concise technical report. Nothing in the files reliably reveals whether the participant developed the solution, used AI competently, collaborated within the rules, or submitted work they do not understand.

The instructor runs the local preparation step. The participant's name and LMS identifiers remain in a private mapping, while the agent receives Student 0007 and a pseudonymised textual representation of the submitted work. The generated report identifies three defensible aspects—for example, an unusual timeout value, an error-recovery branch, and a difference between the report and configuration—and proposes an oral question and expected explanation for each.

During the defence, the instructor can ask the participant to explain why the timeout was selected, what happens when the sensor becomes unavailable, and which configuration change would be required for a different device. A participant who understands the work can reason through those questions even if AI helped produce the original files. If the answers reveal a gap, the instructor follows the institution's normal academic or certification procedure. The AI output itself is never the verdict.

The point is not to catch a particular tool. It is to prevent unverified competence from slipping through the assessment net.

Intended users and settings

The primary users are organisations that must verify practical understanding from work completed outside direct supervision:

• vocational schools and private technical academies;
• professional certification and continuing-education providers;
• universities, colleges, and applied research-training programmes;
• coding bootcamps and specialist cybersecurity or data-science courses;
• industry associations validating sector-specific training outcomes; and
• SMEs conducting clearly bounded internal learning or apprenticeship assessments.

Good tasks contain inspectable evidence: source code, notebooks, Markdown or TeX reports, parameters, calculations, experimental results, configuration, and explanations. The workflow is especially helpful when every submission differs enough to support individual questions.

It is not positioned as a recruitment, candidate-ranking, worker-monitoring, or employee-performance system. Applying AI outputs to hiring, promotion, dismissal, work allocation, or other employment decisions is a separate regulated use and is outside the intended workflow described here.

The workflow

The implementation follows three explicit stages. Real identities are required only before and after the agent stage.

flowchart LR
    A[Private roster and submissions] --> B[Prepare locally]
    B --> C[Pseudonymised agent workspace]
    C --> D[Generate evidence-grounded questions]
    D --> E[Human-led oral verification]
    E --> F[Instructor judgement]
    D --> G[Finalise locally]
    G --> H[Private named reports]

1. Prepare locally

The instructor supplies one or more Moodle/Canvas-style ZIP exports, a private student roster, and the assignment specification. The prepare command:

• matches submission folders conservatively against the local roster;
• assigns deterministic labels such as Student 0001;
• writes the real-name-to-pseudonym map outside the agent workspace;
• concatenates supported readable submission files into pseudonymised text packages;
• removes known names, email addresses, student IDs, and LMS folder identifiers;
• creates pseudonymised students.csv and associations.csv files;
• generates a deterministic process.sh; and
• records unmatched, missing, and excluded material in a private preprocessing report.

Supported readable formats currently include common source-code files, Jupyter notebooks, plain text, Markdown, TeX, and Makefiles. Student-submitted PDF, DOCX, DOC, ODT, and RTF files are excluded by default because changing a filename does not remove identifying content or metadata. The instructor's assignment may still be copied under a canonical name such as task.pdf.

2. Prepare oral-verification material

The generated script runs one agent task per pseudonymised learner. For the target submission, the agent uses only files listed in associations.csv and the canonical assignment. It may compare resolved submissions to distinguish assignment-required overlap from aspects specific to one submission.

The required report structure produces up to three evidence-grounded aspects. Each contains:

1. the concrete part of the submission worth discussing;
2. why that part is informative;
3. one question for the learner; and
4. an expected explanation for the instructor.

If evidence is missing, the report must say so rather than invent a question. Reports remain pseudonymised throughout the agent stage.

3. Finalise privately and conduct the defence

The finalize command uses the separately stored private mapping to create named instructor reports locally. Expected answers are instructor-facing material and should not be shared wholesale with learners. The instructor selects appropriate questions, conducts the oral verification, listens to the learner's reasoning, and applies the institution's assessment rules.

This separation is central: the agent prepares evidence; a competent human interprets the conversation and retains decision authority.

Privacy boundary

The workflow reduces how much directly identifying information reaches the AI stage without pretending that the remaining material is anonymous.

flowchart TB
    subgraph P[Private local area]
        A[Real roster]
        B[Original LMS export]
        C[Pseudonym map]
        D[Named instructor reports]
    end
    subgraph W[Pseudonymised agent area]
        E[Student labels]
        F[Readable submission text]
        G[Question reports]
    end
    A --> C
    A --> E
    B --> F
    E --> G
    F --> G
    C --> D
    G --> D

This is pseudonymisation, not anonymisation. The instructor can reconnect a report to a person, and source code may contain indirect identifiers not present in the roster. Pseudonymised data therefore remains personal data. Operators still need a lawful purpose, transparent learner information, access and retention controls, appropriate processor arrangements, and an assessment of whether an external model provider may retain, transfer, or reuse submitted content.

The repository already implements useful safeguards: safe ZIP extraction, separation of private and agent directories, known-identifier scrubbing, workspace leak checks, conservative matching, explicit unresolved states, and default exclusion of opaque documents. These controls reduce risk but do not establish legal compliance by themselves.

Human oversight and European regulatory context

Education and vocational training are sensitive contexts under the EU AI Act. Systems intended to evaluate learning outcomes may be classified as high-risk. This workflow is deliberately limited to a preparatory function: it generates possible oral questions and supporting evidence, but does not grade, rank, recommend sanctions, or determine misconduct. The instructor must critically review every report and independently evaluate the learner's actual answers.

Whether a particular deployment remains a preparatory tool or materially influences an educational decision depends on its intended purpose and real use. Before it affects grades, progression, or professional certification, the deploying institution should document that classification, complete the appropriate data-protection and fundamental-rights assessment, establish an appeal or correction route, train instructors, and validate question quality across relevant learner groups. “Human in the loop” must mean genuine competence and authority, not automatic acceptance of an AI report.

Useful primary references include the European Commission's AI Act education and vocational-training guidance, classification rules for high-risk systems, and the European Data Protection Board's explanation of anonymisation and pseudonymisation.

Operational safeguards for a pilot

Learner submissions are untrusted input. They may contain secrets, malicious code, or text intended to manipulate an agent. A controlled pilot should therefore:

• use a disposable container or isolated virtual machine;
• mount submissions read-only and provide a separate output location;
• disable network access unless it is explicitly required and approved;
• prevent execution of submitted programs by default;
• keep credentials and unrelated institutional files out of the workspace;
• minimise cross-learner comparison to cases with a documented pedagogical need;
• inspect the pseudonymised workspace before starting the agent; and
• review every generated question against the underlying evidence.

The current command-line workflow invokes a coding agent in autonomous mode. This is convenient for an MVP but should not be treated as the final institutional security profile. Sandboxing, model-provider governance, logging, deletion, version pinning, and reproducible prompts should be specified before wider use.

What the workflow does—and does not establish

It supports	It does not establish
Individual questions grounded in submitted artefacts	That a learner cheated or used a prohibited tool
Verification of demonstrated understanding	Legal or factual proof of authorship
Instructor preparation at cohort scale	An automatic grade or pass/fail decision
Privacy risk reduction through pseudonymisation	Anonymous processing or exemption from GDPR
Documented human review	Compliance without institutional assessment

The distinction protects learners and strengthens the credential. A learner who used AI transparently and understands the result can demonstrate that competence. A learner who cannot explain critical choices receives human follow-up rather than an opaque accusation.

Current MVP and adoption path

The research MVP provides the complete prepare → agent → finalize path, synthetic privacy tests, deterministic pseudonyms, Moodle-style archive handling, structured reports, and private named outputs. It is suitable for a controlled evaluation with synthetic data followed by a small, formally approved pilot.

A responsible adoption begins with one course or certification task, a clear pedagogical purpose, representative synthetic testing, an approved privacy and security design, instructor training, and a rule that no consequential decision may be based solely on the generated report. The pilot should measure question relevance, unsupported claims, instructor preparation time, learner experience, and differences across submission types—not simply how many cases appear “suspicious”.

The implementation and operating instructions are available in the SLAIF Defense Grading repository. The repository name reflects its origin; the implemented workflow is more accurately understood as privacy-preserving oral-verification preparation rather than automated grading.