Standard Fields
Most test cases in this library share wording for how they are scored, what counts as a pass or a fail, what evidence a reviewer expects and what the lab looks like. Repeating that in every case made the cases long and hid what is different about each one. The shared wording now lives here, once. A case shows a field only when its text differs from the standard.
How to read a case. If a case does not show one of the fields below, the standard wording on this page applies. Find the right row from the case's own details: its Test Method in the first table, whether it has an Expected Detection field (a detection case) or an Expected Result field (a result case), and its layer or domain from its ID (TC-L06-007 is layer L06).
Copying a case. Each case ends with a closed box, Standard fields for this case, that lists the wording from this page that applies to it. Open it and copy it with the case, and the copy is complete without this page. The AI Security Test Case Tracker (.xlsx) has the same wording filled into every row.
A case that does show one of these fields has its own wording, and that wording replaces the standard for that case. Anyone adding a case should leave a field out when the standard fits and write it out only when it does not. The build checks that no case repeats a standard word for word.
Fields that are the same in every case
Compliance Evidence
| Applies to | Standard wording |
|---|---|
| Every case | Test execution log and captured evidence retained in the Test Case Execution Register for audit. |
Fields that depend on the kind of case
Audit Evidence
| Applies to | Standard wording |
|---|---|
| Cases with an Expected Result field | Record identifier, actor, timestamp, before and after values, approval reference and source evidence exportable for audit. |
Fail Criteria
| Applies to | Standard wording |
|---|---|
| Cases with an Expected Detection field | Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data. |
| Cases with an Expected Result field | Expected Result is not met; evidence is missing or not from the live product; the control action does not occur where required; or the applicable requirement and its parameters were not recorded before testing. |
Fields that depend on the test method and the kind of case
Scoring Criteria
| Applies to | Standard wording |
|---|---|
| Technical method, detection cases | 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope. |
| Evidence method, detection cases | 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope. |
| Attestation method, detection cases | 0 = no statement; 3 = statement without supporting detail; 5 = statement with technical detail and an offer to demonstrate. |
| Technical method, result cases | 0 = not demonstrated; 3 = demonstrated but with missing fields, manual workarounds or SLA exceeded; 5 = fully met in the live product with complete evidence; N/A = capability out of scope by design. |
| Evidence method, result cases | 0 = not demonstrated; 3 = partly shown or shown only from documentation; 5 = shown in the live product with exportable evidence; N/A = capability out of scope by design. |
Pass Criteria
| Applies to | Standard wording |
|---|---|
| Technical method, detection cases | Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC. |
| Evidence method, detection cases | Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC. |
| Attestation method, detection cases | Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC. |
| Technical method, result cases | Expected Result and Expected Control Action are met in full against the requirement recorded in Applicable Requirement, within the vendor's documented SLA, and the evidence under Evidence to Capture is obtained from the live PoC. |
| Evidence method, result cases | Expected Result are met in full against the requirement recorded in Applicable Requirement, within the vendor's documented SLA, and the evidence under Evidence to Capture is obtained from the live PoC. |
Reviewer Notes
| Applies to | Standard wording |
|---|---|
| Technical method, detection cases | Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested. |
| Evidence method, detection cases | Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3. |
| Attestation method, detection cases | Attestation scores below demonstrated evidence. Request a demonstration where possible. |
| Technical method, result cases | Confirm evidence comes from the live PoC tenant, not vendor-supplied demo data. Record the product version tested and any manual steps needed. |
| Evidence method, result cases | Confirm the evidence is taken from the live product. Documentation alone scores no higher than 3. |
Expected Alert / Log
| Applies to | Standard wording |
|---|---|
| Technical method, detection cases | Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA. |
| Evidence method, detection cases | Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it. |
| Attestation method, detection cases | Signed written statement from the vendor. |
Expected Record / Log
| Applies to | Standard wording |
|---|---|
| Technical method, result cases | Record with timestamp, actor, record identifier, decision, before and after values and approval reference, within the vendor's documented SLA. |
| Evidence method, result cases | Configuration state, record or export that supports the claim, with timestamp and the identity of the person who produced it. |
Fields that depend on the layer or domain
Expected Report / Dashboard Evidence
| Applies to | Standard wording |
|---|---|
| D08 AI Red Teaming and Continuous Testing | Finding, run and verdict visible in the red-team console within the documented refresh interval. |
| D09 Agent and Non-Human Identity Governance | Identity finding or governance record visible in the identity governance view within the documented refresh interval. |
| D10 Browser and Computer-Use Agents | Agent action and decision visible in the browser agent dashboard within the documented refresh interval. |
| D11 Multimodal and Voice Input Security | Media finding or decision visible in the multimodal or gateway dashboard within the documented refresh interval. |
| D12 AI Incident Response and Forensics | Incident, evidence and case state visible in the incident response workspace within the documented refresh interval. |
| D13 AI Cost and Abuse Controls | Spend, budget and abuse state visible in the cost and abuse dashboard within the documented refresh interval. |
| L01 Business & Use Cases | Record or report visible in the governance and compliance workspace within the documented refresh interval. |
| L02 Governance & Risk Mgmt | Record or report visible in the governance and compliance workspace within the documented refresh interval. |
| L03 Legal, Privacy & Compliance | Record or report visible in the governance and compliance workspace within the documented refresh interval. |
| L04 Human Interaction Layer | Event visible in the workforce/Shadow AI dashboard within the documented refresh interval. |
| L05 AI Applications | Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval. |
| L06 Agent Orchestration Layer | Agent, tool and decision visible in the agent governance dashboard within the documented refresh interval. |
| L07 Prompt & Context Layer | Detection and decision visible in the prompt-security dashboard within the documented refresh interval. |
| L08 AI Gateway & Security Controls | Policy decision and rule hit visible in the enforcement dashboard within the documented refresh interval. |
| L09 Identity & Access Mgmt | Identity finding or access decision visible in the identity or access dashboard within the documented refresh interval. |
| L10 Data Layer | Finding visible in the data security or data inventory dashboard within the documented refresh interval. |
| L11 Knowledge & Retrieval Layer | Finding or retrieval decision visible in the knowledge and retrieval dashboard within the documented refresh interval. |
| L12 Model Layer | Model finding or decision visible in the model security dashboard within the documented refresh interval. |
| L13 Training & Fine-Tuning Layer | Finding or decision visible in the training and data supply chain dashboard within the documented refresh interval. |
| L14 MLOps / LLMOps Layer | Finding or decision visible in the pipeline security dashboard within the documented refresh interval. |
| L15 Infrastructure Layer | Infrastructure finding visible in the infrastructure security dashboard within the documented refresh interval. |
| L16 Supply Chain & Third Party | Finding or inventory entry visible in the supply chain dashboard within the documented refresh interval. |
| L17 Monitoring, Detection & Response | Detection, alert and case state visible in the platform and SIEM within the documented refresh interval. |
Forensic Evidence
| Applies to | Standard wording |
|---|---|
| D08 AI Red Teaming and Continuous Testing | Attack identifier, library version, target version, full transcript, judge verdict and confidence, seed and settings, sink evidence and timestamp exportable for reproduction. |
| D09 Agent and Non-Human Identity Governance | Identity, sponsor, entitlement, credential reference, request or decision, approver and timestamp exportable for audit and incident reconstruction. |
| D10 Browser and Computer-Use Agents | Agent, user, site and element, action type, page or screen state, policy decision, approval record and timestamp exportable for incident reconstruction. |
| D11 Multimodal and Voice Input Security | Media hash, type, channel, extracted text or transcript excerpt, region or time offset, policy decision and timestamp exportable for incident reconstruction, with sensitive values masked per policy. |
| D12 AI Incident Response and Forensics | Incident, evidence and case identifiers with component versions, actors, decisions, approvals, hashes and timestamps exportable for investigation and audit. |
| D13 AI Cost and Abuse Controls | Request, key, user, application, team, model, tokens, cost, policy decision and timestamp exportable for investigation, chargeback and dispute handling. |
| L04 Human Interaction Layer | Full session metadata (user, device, browser, destination, network path, timestamp and policy decision) exportable for incident reconstruction. |
| L05 AI Applications | Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction. |
| L06 Agent Orchestration Layer | Agent identifier, requesting user, tool and arguments, delegation chain, policy decision, approval record and timestamp exportable for incident reconstruction. |
| L07 Prompt & Context Layer | Full prompt, context segment, source label, rule identifier, confidence, decision and timestamp exportable for incident reconstruction, with sensitive values masked per policy. |
| L08 AI Gateway & Security Controls | Rule identifier, policy version, direction (prompt or response), identity, destination and decision exportable for incident reconstruction. |
| L09 Identity & Access Mgmt | Identity (user, agent, service), token or key identifier, scope, resource, decision and timestamp exportable for incident reconstruction. |
| L10 Data Layer | Data object identifier, location, classification, accessing identity, action and timestamp exportable for incident reconstruction. |
| L11 Knowledge & Retrieval Layer | Query, rewritten query, retrieved chunk and document identifiers, source, requesting user, filters applied, decision and timestamp exportable for incident reconstruction. |
| L12 Model Layer | Model identifier, version, hash, source, requesting identity, action and timestamp exportable for incident reconstruction. |
| L13 Training & Fine-Tuning Layer | Dataset or checkpoint identifier, hash, source, training job, operator, decision and timestamp exportable for incident reconstruction. |
| L14 MLOps / LLMOps Layer | Pipeline, stage, artefact identifier and hash, actor, approval reference, decision and timestamp exportable for incident reconstruction. |
| L15 Infrastructure Layer | Asset identifier, location, configuration state, accessing identity, change or finding and timestamp exportable for incident reconstruction. |
| L16 Supply Chain & Third Party | Component identifier, version, hash, source, publisher, decision and timestamp, plus the vendor evidence received, exportable for audit and incident reconstruction. |
| L17 Monitoring, Detection & Response | Event, alert, case and action identifiers with actor, source layer, decision and timestamp, plus evidence hashes, exportable for incident reconstruction. |
Preconditions
| Applies to | Standard wording |
|---|---|
| D08 AI Red Teaming and Continuous Testing | Isolated PoC lab provisioned; seeded target applications, mock tools and sinks, canary strings and test users seeded per the Lab Prerequisites appendix; the vendor's red-team tooling licensed and pointed only at the lab targets. Seeded target applications with documented ground-truth weaknesses (a deliberately vulnerable lab chatbot, RAG application and tool-using agent), mock tools writing to a lab sink, registered canary strings and an isolated network; the vendor's red-team tooling licensed for the PoC; no production systems, real customer data or live third-party services are targeted. |
| D09 Agent and Non-Human Identity Governance | Isolated PoC lab provisioned; lab identity provider, fabricated agent identities and entitlements, mock cloud and SaaS tenants, lab secret store and test users seeded per the Lab Prerequisites appendix; platform connected with least-privilege test credentials. Lab identity provider, lab agents with fabricated identities and entitlements, a mock cloud account and SaaS tenant, a lab secret store and a mock ticketing system; no production identities, credentials or directories are connected. |
| D10 Browser and Computer-Use Agents | Isolated PoC lab provisioned; internal lab web sites, managed test browsers, desktop VM, test browser and computer-use agents and test users seeded per the Lab Prerequisites appendix; platform deployed for the agent type under test. Lab web sites hosted internally (intranet mock, shop with mock payment page, file-sharing mock, mock SSO with MFA, mock CAPTCHA, hostile-content test pages), managed test browsers and a desktop virtual machine, test browser and computer-use agents, and test users; no real websites, accounts, payment methods or credentials are used. |
| D11 Multimodal and Voice Input Security | Isolated PoC lab provisioned; multimodal test applications and assistants, mock or lab-hosted multimodal model, fabricated media files and test users seeded per the Lab Prerequisites appendix; platform deployed for the media channel under test. Multimodal lab: test applications and assistants that accept images, documents, audio and video; a mock model or lab-hosted multimodal model; fabricated media (images with synthetic text, documents with hidden layers, synthetic-voice recordings of fabricated personas), registered canary strings and a lab sink; no real people's images, voices or documents are used. |
| D12 AI Incident Response and Forensics | Isolated PoC lab provisioned; lab AI applications with versioned prompts, models and indexes, lab SIEM and case tools, scripted incident scenarios with ground truth and test responders seeded per the Lab Prerequisites appendix; platform connected with least-privilege test credentials. Lab AI applications (chatbot, RAG assistant, tool-using agent) with versioned prompts, models and indexes; a lab SIEM, ticketing and case tool; scripted incident scenarios with known ground truth; test responders, an investigator and a legal reviewer; no production systems, real incidents or real personal data. |
| D13 AI Cost and Abuse Controls | Isolated PoC lab provisioned; lab applications and gateway pointed at a mock provider with per-token pricing, fabricated keys and budgets, abuse scripts acting only on lab endpoints and a mock billing export seeded per the Lab Prerequisites appendix. Lab AI applications and a gateway pointed at a mock provider with per-token pricing and configurable rate limits; fabricated keys, accounts and budgets; load and abuse scripts that act only against lab endpoints; a mock billing export and expense feed; no real provider accounts, spend or customer traffic. |
| L01 Business & Use Cases | Isolated PoC tenant provisioned; fabricated use cases, policies, records and individuals seeded per the Lab Prerequisites appendix; test users in the roles named in the case; platform connected to the lab directory with least-privilege test credentials. Written evidence request issued to the vendor before the PoC; sample use cases, policies, registers and records are fabricated; the assessor has copied the applicable requirement text and any numeric parameters (timelines, retention periods, thresholds) from the adopted framework into the Applicable Requirement field before testing. |
| L02 Governance & Risk Mgmt | Isolated PoC tenant provisioned; fabricated use cases, policies, records and individuals seeded per the Lab Prerequisites appendix; test users in the roles named in the case; platform connected to the lab directory with least-privilege test credentials. Written evidence request issued to the vendor before the PoC; sample policies, risks, exceptions and records are fabricated; the assessor has copied the applicable requirement text and any numeric parameters (timelines, review periods, thresholds) from the adopted framework into the Applicable Requirement field before testing. |
| L03 Legal, Privacy & Compliance | Isolated PoC tenant provisioned; fabricated use cases, policies, records and individuals seeded per the Lab Prerequisites appendix; test users in the roles named in the case; platform connected to the lab directory with least-privilege test credentials. Written evidence request issued to the vendor before the PoC; sample processing activities, individuals, requests and records are fabricated; the assessor has copied the applicable requirement text and any numeric parameters (response timelines, notification periods, retention periods, thresholds) from the adopted framework into the Applicable Requirement field before testing. |
| L04 Human Interaction Layer | Isolated PoC lab provisioned; test users, managed and unmanaged test endpoints and synthetic data seeded per the Lab Prerequisites appendix; platform agent or integration installed for the channel under test. |
| L05 AI Applications | Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test. |
| L06 Agent Orchestration Layer | Isolated PoC lab provisioned; lab agent framework, mock tools, mock MCP servers and test users seeded per the Lab Prerequisites appendix; platform deployed in the documented mode for agent and tool traffic. Lab agent framework with a controllable model or scripted agent, mock tools and mock MCP servers writing to a lab sink, canary strings registered in advance, and attack-success baseline measured with the platform disabled. |
| L07 Prompt & Context Layer | Isolated PoC lab provisioned; test application, mock model or lab-hosted model, mock tools and test users seeded per the Lab Prerequisites appendix; platform deployed in the documented mode for the channel under test. Test application backed by a lab-hosted or mock model; mock tools writing to a lab sink; canary strings registered in advance; attack-success baseline measured with the platform disabled so only payloads that actually work are counted. |
| L09 Identity & Access Mgmt | Isolated PoC lab provisioned; lab identity provider, directory, secret store and mock AI services seeded per the Lab Prerequisites appendix; platform connected with least-privilege test credentials. Lab identity provider and directory with fabricated users and groups; lab secret store; mock AI services and agents using test credentials only; no production identities or secrets connected. |
| L10 Data Layer | Isolated PoC lab provisioned; test data stores, test tenants and fabricated records seeded per the Lab Prerequisites appendix; platform connected with least-privilege test credentials. Isolated data environment populated only with fabricated records; classification ground truth agreed and signed off before testing; no production data connected. |
| L11 Knowledge & Retrieval Layer | Isolated PoC lab provisioned; lab RAG application, test vector store, fabricated corpus and test users seeded per the Lab Prerequisites appendix; platform connected to the lab knowledge sources with least-privilege test credentials. Lab RAG application with a test vector store, a fabricated corpus of 200 documents with access control lists, registered canary content and test users; no production knowledge sources connected. |
| L12 Model Layer | Isolated PoC lab provisioned; lab model hosting, test models, mock registry and test users seeded per the Lab Prerequisites appendix; platform connected to the lab model environment with least-privilege test credentials. Lab model hosting with small open-weight test models, a mock model registry, harmless EICAR-style marker artefacts (a file that only writes a marker file when loaded, never real malware), and test users; no production models connected. |
| L13 Training & Fine-Tuning Layer | Isolated PoC lab provisioned; lab training environment, fabricated datasets, small open-weight model, mock registry and test users seeded per the Lab Prerequisites appendix; platform connected with least-privilege test credentials. Lab training environment with fabricated datasets, a small open-weight model for fine-tuning, harmless trigger phrases, registered canary strings, a mock model registry and test users; no production data or production training pipelines connected. |
| L14 MLOps / LLMOps Layer | Isolated PoC lab provisioned; lab CI/CD runner, mock registries, notebook server, test cluster and test users seeded per the Lab Prerequisites appendix; platform connected with least-privilege test credentials. Lab MLOps environment with a CI/CD runner, mock model and container registries, notebook server, small training and serving pipeline, test Kubernetes cluster and test users; no production pipelines, registries or credentials connected. |
| L15 Infrastructure Layer | Isolated PoC lab provisioned; lab cluster with real or simulated GPU nodes, mock inference servers, test cloud account, network segments and test users seeded per the Lab Prerequisites appendix; platform connected with least-privilege test credentials. Lab Kubernetes cluster with real or simulated GPU nodes, mock inference servers, a test cloud account with fabricated resources, separate network segments, seeded misconfigurations and test users; no production infrastructure, accounts or data connected. |
| L16 Supply Chain & Third Party | Isolated PoC lab provisioned; mock model hub, mock package index, test applications and test users seeded per the Lab Prerequisites appendix; written vendor evidence requests issued before testing; platform connected with least-privilege test credentials. Lab environment with mock model hub, mock package index, test applications, fabricated vendor documents and test users; vendor evidence requests issued in writing before testing; no production dependencies or contracts altered. |
| L17 Monitoring, Detection & Response | Isolated PoC lab provisioned; lab SIEM and SOAR, the platform's event feed, simulated incident scenarios, test analysts and mock ticketing seeded per the Lab Prerequisites appendix; no production alerting connected. Lab SIEM and SOAR (open-source or vendor-neutral), the platform's event feed, simulated incident scenarios, test analysts and a mock ticketing system; no production alerting, ticketing or on-call routing connected. |