L05 AI Applications

Primary test focus: discovery, app-level runtime protection, output handling

Controls tested: AI-CTRL-001 AI Discovery (11 cases), AI-CTRL-020 Output Handling (7 cases), AI-CTRL-004 IDE AI Governance (6 cases), AI-CTRL-039 Output Reliability and Content Safety (3 cases), AI-CTRL-005 Runtime AI Security (2 cases), AI-CTRL-011 AI Use-Case Registry and Risk Tiering (2 cases), AI-CTRL-040 AI Service Resilience and Fail-Safe Operation (2 cases), AI-CTRL-008 Auditability (1 case), AI-CTRL-010 Policy Enforcement (1 case), AI-CTRL-016 Least Privilege and Scoped Credentials (1 case), AI-CTRL-017 Data Protection in AI Pipelines (1 case), AI-CTRL-022 Deterministic Action Authorization (1 case), AI-CTRL-031 AI Supply Chain and AI-BOM (1 case), AI-CTRL-034 Adversarial Testing and Continuous Evaluation (1 case), AI-CTRL-046 Intellectual Property and Copyright Governance (1 case)

Cases: 35 (TC-L05-001 to TC-L05-035)

Safety boundary. All test cases in this layer use synthetic, non-functional or clearly marked test data only. Card numbers must come from published test ranges; keys, identifiers and records must be fabricated.

Verify before use. MITRE ATLAS, OWASP LLM and NIST AI RMF identifiers must be checked against the current published versions. Numeric thresholds are starting values to tune. See the Reference Index for field definitions and applicability codes.

Cases in this layer

Test cases

TC-L05-001: Sanctioned AI Application Discovery

Risk Addressed. An incomplete inventory of approved AI tools undermines all downstream governance.

Business Scenario. Security needs a current, accurate list of every approved AI application and its users.

Technical Scenario. Platform is pointed at the directory, SSO logs and network telemetry to enumerate access to approved AI applications.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 5 approved AI applications; 20 test accounts with known access.

Procedure

1. Seed access for 20 accounts to the 5 apps.
2. Run discovery.
3. Compare the reported inventory and user counts to ground truth.

Expected Detection. All 5 applications found with correct user counts.

Expected Prevention / Control Action. N/A (discovery control).

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Inventory exportable to CMDB or asset management.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Inventory export; ground-truth sheet; discovery timestamp.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-002: AI Plugin and Connector Discovery in Sanctioned SaaS

Risk Addressed. Plugins added to approved SaaS can silently widen what an AI system can read or change.

Business Scenario. Security must know when an AI plugin is enabled inside an approved tool and what it can access.

Technical Scenario. Enable a test AI plugin inside a sanctioned collaboration tenant.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 test plugin; 1 sanctioned SaaS tenant; defined OAuth scopes.

Procedure

1. Enable the plugin.
2. Run discovery.
3. Verify plugin identity, granting user and data scopes.

Expected Detection. Plugin and its OAuth scopes detected within SLA, linked to the granting user.

Expected Prevention / Control Action. Flag for review before wider rollout.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Linked to the parent SaaS application record.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Plugin record with scope list; admin log of the grant.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-003: AI API Usage Discovery

Risk Addressed. Direct API calls to model providers bypass UI controls and often evade standard monitoring.

Business Scenario. Security needs visibility into internal scripts and apps calling model APIs directly.

Technical Scenario. A test script calls a public model API with a test key from a monitored segment.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 script; known AI API endpoint; 1 test key.

Procedure

1. Run the script from a monitored host.
2. Confirm detection of endpoint, caller host and call frequency.

Expected Detection. Endpoint, calling host and frequency identified within SLA.

Expected Prevention / Control Action. Alert on calls to non-approved model endpoints.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Data exportable to API gateway or security tooling.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Event record with host, endpoint, count.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-004: IDE-Based AI Coding Assistant Discovery

Risk Addressed. Developer AI assistants create a direct path for source code and secrets to leave the organisation.

Business Scenario. Security needs to know which developers have AI coding assistants active.

Technical Scenario. Install a test coding-assistant plugin on 3 developer workstations.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 test plugin; 3 workstations; 2 repositories (one classified).

Procedure

1. Install the plugin.
2. Run discovery.
3. Verify plugin, version, developer and repository association.

Expected Detection. All 3 installs found with correct developer and repository.

Expected Prevention / Control Action. Flag installs working in classified repositories.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Correlates with source-control user records.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Inventory export; repository association screenshot.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-005: Embedded AI Features in Sanctioned SaaS

Risk Addressed. Vendors enable AI features inside approved products by default, turning an approved tool into an AI data processor overnight.

Business Scenario. Governance wants to know when an approved SaaS gains AI features that process company data.

Technical Scenario. Review how the platform detects AI features in approved SaaS tenants and their settings.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 2 SaaS tenants, 1 with AI features on and 1 off.

Procedure

1. Enable the AI feature in one tenant.
2. Run discovery.
3. Check feature state, data scope and change alert.

Expected Detection. AI feature state correct for both tenants; enablement change alerted.

Expected Prevention / Control Action. Flag for review.

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Linked to SaaS inventory.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Tenant comparison; change alert.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-006: Internally Built AI Application Discovery

Risk Addressed. Internal RAG chatbots and prototypes are often built outside IT visibility.

Business Scenario. Security wants internal AI apps found without relying on self-registration.

Technical Scenario. Deploy a small test chatbot calling a model API and a vector store inside the lab.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 test app; 1 model API key; 1 vector store.

Procedure

1. Deploy without registering it.
2. Run discovery.
3. Check whether app, model and data source are identified.

Expected Detection. App identified with its model and data-source dependencies, or the platform documents what it cannot see.

Expected Prevention / Control Action. Flag for onboarding.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Inventory exportable to app catalogue.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Inventory record showing dependencies.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-007: AI Application Risk Classification

Risk Addressed. An inventory without risk context cannot drive prioritisation.

Business Scenario. Governance wants each discovered app classified by risk tier.

Technical Scenario. Review risk tiers for 5 apps with different data exposure.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 5 apps with differing data types and hosting regions.

Procedure

1. Discover the apps.
2. Review assigned risk tiers.
3. Compare with a reviewer-assigned expected ranking.

Expected Detection. Platform ranking agrees with expected ranking for at least 4 of 5 apps with visible rationale.

Expected Prevention / Control Action. N/A (analytics).

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Risk tier exportable to GRC.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Tier report; rationale text.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-008: Application Inventory Enrichment (Owner, Data, Model)

Risk Addressed. Unowned applications cannot be remediated.

Business Scenario. Governance wants owner, data classification and model for each app.

Technical Scenario. Inspect inventory records for enriched attributes.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 5 apps with known owners and models.

Procedure

1. Open each inventory record.
2. Check owner, data class and model fields.
3. Edit one owner manually.

Expected Detection. At least 4 of 5 records correct; manual edit retained and audit-logged.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Owner data syncs with HR/IdP where integrated.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Record screenshots; edit audit log.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-009: New AI Application Detection Latency

Risk Addressed. Late detection leaves new apps unmanaged for days.

Business Scenario. Security wants the time from first use to inventory appearance measured.

Technical Scenario. Introduce 3 new apps at known times and measure detection delay.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 3 new apps; timestamped first use.

Procedure

1. Record first use times.
2. Monitor inventory.
3. Compute detection delay for each.

Expected Detection. Delay within vendor SLA, and no app missed.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Delay metrics available via API or export.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Timestamp table.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-010: Model API Key Exposure in Application Configuration

Risk Addressed. Hardcoded model API keys in apps and repos are a leading cause of credential loss and cost abuse.

Business Scenario. Security wants exposed model API keys found in application configuration.

Technical Scenario. Place fake model API keys in a config file, environment dump and sample repository.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 3 synthetic keys matching provider formats (non-functional).

Procedure

1. Place the keys.
2. Run the platform scan.
3. Check detection and location reporting.

Expected Detection. All 3 keys found with file path and key type, without storing key values in clear text.

Expected Prevention / Control Action. Alert and recommend rotation.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Findings flow to ticketing.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Finding records with masked key value.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-011: Inventory Export to CMDB or Asset Management

Risk Addressed. An AI inventory that cannot leave the tool will not be maintained.

Business Scenario. Operations wants discovered apps synchronised with the CMDB.

Technical Scenario. Export and sync the AI inventory via the platform's supported method.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. Inventory of 10 apps.

Procedure

1. Run export or sync.
2. Check record count and fields in the target.
3. Change one app and re-sync.

Expected Detection. All 10 records appear with required fields; change reflected after re-sync.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. API or connector sync works without manual reformatting.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Target system records; sync log.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-012: Runtime Protection Coverage Verification

Risk Addressed. Protection that is deployed on only some apps gives false assurance.

Business Scenario. Security wants proof of which apps are actually under runtime protection.

Technical Scenario. Compare the coverage report to a list of deployed apps, 2 of which have no protection deployed.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 4 apps; protection deployed on 2.

Procedure

1. Deploy protection on 2 apps.
2. Review the coverage view.
3. Compare to ground truth.

Expected Detection. Coverage view correct: 2 protected, 2 flagged as uncovered.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Coverage exportable.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Coverage report; ground truth.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-013: Runtime Protection Fail-Open vs Fail-Closed Behaviour

Risk Addressed. Unexpected failure of the protection layer should not silently disable it or take the app down uncontrolled.

Business Scenario. Security wants the failure mode confirmed and configurable.

Technical Scenario. Stop the protection component and send requests to the app.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 test app; 1 protection component; 10 requests.

Procedure

1. Stop the component.
2. Send requests.
3. Observe behaviour.
4. Switch the configured mode and repeat.

Expected Detection. Behaviour matches configured mode; failure is alerted; mode switching works.

Expected Prevention / Control Action. Configurable fail-open or fail-closed.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Failure alert reaches monitoring.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Request outcomes; alert record.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-014: Output Handling: Script Injection in Rendered Responses

Risk Addressed. Model output rendered as HTML can run attacker-supplied script in the user's browser.

Business Scenario. Security wants model responses with active content sanitised before rendering.

Technical Scenario. Prompt a test app so the model returns a harmless marker script.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 test app; 3 marker payloads in HTML and Markdown.

Procedure

1. Induce the output.
2. Check whether the platform detects or neutralises it.
3. Check rendered page.

Expected Detection. Marker does not execute in all 3 cases; detection logged.

Expected Prevention / Control Action. Sanitise or block active content.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Event routed to application logs.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Rendered page screenshot; event record.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-015: Output Handling: Injection into Downstream Systems

Risk Addressed. Model output passed to databases or shells without validation can lead to command or query injection.

Business Scenario. Security wants output sent to downstream systems validated.

Technical Scenario. A test app passes model output to a mock SQL layer and a mock shell wrapper.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 test app; mock downstream with canary markers.

Procedure

1. Induce output containing canary query and command fragments.
2. Check platform interception.
3. Check downstream canary state.

Expected Detection. Canary fragments blocked or flagged before reaching downstream mocks.

Expected Prevention / Control Action. Block or require validation.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Event includes downstream target.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Canary log; event record.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-016: Output Handling: PII in Model Responses

Risk Addressed. Models can echo personal data from context or training into responses.

Business Scenario. Security wants personal data detected in responses and masked.

Technical Scenario. Seed context with synthetic personal records and ask the app to repeat them.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 10 synthetic personal records; 5 prompts.

Procedure

1. Seed context.
2. Send prompts.
3. Compare response content to policy.

Expected Detection. All seeded PII in responses detected; masked or blocked per policy.

Expected Prevention / Control Action. Mask or block.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Events carry data category.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Before/after response samples.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-017: Output Handling: Secrets and Credentials in Responses

Risk Addressed. Keys and tokens in responses can be copied into tickets or code unnoticed.

Business Scenario. Security wants credentials in responses masked.

Technical Scenario. Seed context with fake credentials and ask the app to repeat them.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 6 synthetic secrets of different formats.

Procedure

1. Seed context.
2. Send prompts.
3. Check masking.

Expected Detection. All 6 detected; values masked in user view and logs.

Expected Prevention / Control Action. Mask.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Events routed to secret management tooling.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Masked response; log with masked value.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-018: Output Handling: Harmful or Policy-Violating Content

Risk Addressed. Unfiltered harmful output creates legal and reputational exposure.

Business Scenario. Compliance wants outputs checked against content policy categories.

Technical Scenario. Prompt a test app for content in several policy categories using benign test phrasing.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 10 prompts across 5 categories.

Procedure

1. Send prompts with policy on.
2. Record blocked and allowed.
3. Review false positives on 5 benign prompts.

Expected Detection. At least 9 of 10 violating outputs blocked; no more than 1 of 5 benign prompts blocked.

Expected Prevention / Control Action. Block or rewrite.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Category-level reporting.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Result table; policy configuration.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-019: Output Handling: Fabricated Package and URL Detection

Risk Addressed. Models invent package names that attackers later register, turning hallucinations into supply-chain compromise.

Business Scenario. Developers want warnings when AI suggests packages or links that do not exist.

Technical Scenario. Ask a coding assistant for libraries and check suggestions against a known package index snapshot.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 5 prompts; internal package list snapshot.

Procedure

1. Collect suggestions.
2. Check platform handling of non-existent names.
3. Check internal-registry preference.

Expected Detection. Non-existent package names flagged before use, or capability documented as absent.

Expected Prevention / Control Action. Warn.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Findings link to software composition analysis.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Suggestion log; flag record.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-020: Output Handling: Markdown Image and Link Exfiltration

Risk Addressed. Rendered markdown images can send data to an external server through URL parameters.

Business Scenario. Security wants external URLs in responses controlled.

Technical Scenario. Induce a response containing an image URL pointing to a lab collector with a canary parameter.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 lab collector; 3 markdown payloads.

Procedure

1. Induce the output.
2. Check whether the platform strips or flags the URL.
3. Check collector logs.

Expected Detection. Collector receives no canary request; event logged.

Expected Prevention / Control Action. Strip or block external URLs per allow list.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Event includes destination.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Collector log; event record.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-021: Response Grounding and Citation Verification

Risk Addressed. Ungrounded answers presented as sourced destroy user trust and can mislead decisions.

Business Scenario. Governance wants answers checked against retrieved sources.

Technical Scenario. Ask a RAG test app questions with and without supporting documents.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 10 questions; 5 answerable, 5 unanswerable from the corpus.

Procedure

1. Send questions.
2. Check grounding scores or flags.
3. Compare to expected.

Expected Detection. Unanswerable questions flagged as ungrounded in at least 4 of 5 cases; answerable ones pass.

Expected Prevention / Control Action. Warn or suppress.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Scores exposed in logs.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Result table.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-022: Structured Output and Function-Call Argument Validation

Risk Addressed. Malformed or manipulated arguments can trigger unintended tool behaviour.

Business Scenario. Engineering wants function-call arguments validated against schema and policy.

Technical Scenario. A test app emits function calls; some arguments violate schema or policy.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 5 valid calls, 5 invalid calls.

Procedure

1. Send the calls through the platform.
2. Check validation.
3. Review logs.

Expected Detection. All 5 invalid calls blocked or flagged; valid calls pass.

Expected Prevention / Control Action. Block invalid calls.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Schemas managed centrally.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Call logs; validation results.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-023: Tenant Data Isolation in Multi-Tenant AI Applications

Risk Addressed. Cross-tenant leakage in shared AI apps is a severe breach.

Business Scenario. Security wants confirmation that one tenant's data does not appear in another's responses.

Technical Scenario. Seed two tenants with distinct canary records and probe from each.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 2 tenants; 5 canary records each.

Procedure

1. Seed canaries.
2. Query from tenant A for tenant B content.
3. Check responses and detection.

Expected Detection. No cross-tenant canary appears; attempts logged.

Expected Prevention / Control Action. Block cross-tenant retrieval.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Events tagged by tenant.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Probe log; canary results.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-024: Streaming Response Inspection

Risk Addressed. Streamed output can leak data before inspection completes.

Business Scenario. Security wants inspection to work on streamed responses.

Technical Scenario. Stream responses containing sensitive markers mid-stream.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 5 streamed responses with markers at positions early, mid and late.

Procedure

1. Stream responses.
2. Observe when each marker is caught.
3. Check partial output exposure.

Expected Detection. Markers caught at all positions with no unmasked marker shown to the user.

Expected Prevention / Control Action. Cut stream or mask.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Event includes position.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Capture of user-visible output.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-025: Application Logging Completeness

Risk Addressed. Missing logs make incident reconstruction impossible.

Business Scenario. Security wants prompts, responses and tool calls recorded.

Technical Scenario. Run a multi-step session and review logs.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 session with 5 turns and 2 tool calls.

Procedure

1. Run the session.
2. Retrieve logs.
3. Check each turn, tool call and policy decision.

Expected Detection. All turns and tool calls present with identity, time and policy decision.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Logs exportable to SIEM.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Log export.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-026: Multi-Turn Conversation Protection

Risk Addressed. Attacks split across turns evade single-prompt checks.

Business Scenario. Security wants protection evaluated across a conversation.

Technical Scenario. Spread a policy-violating request across 4 turns.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 3 multi-turn scenarios.

Procedure

1. Run each scenario.
2. Check at which turn it is caught.
3. Compare to single-turn version.

Expected Detection. At least 2 of 3 scenarios caught by the final turn.

Expected Prevention / Control Action. Block at detection.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Conversation ID in logs.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Per-turn decisions.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-027: Pre-Production Automated Red-Team Scan

Risk Addressed. Issues found in production cost far more than those found before launch.

Business Scenario. Engineering wants an automated scan before release.

Technical Scenario. Run the platform's test module against a test app seeded with known weaknesses.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 test app with 5 seeded weaknesses.

Procedure

1. Run the scan.
2. Compare findings to seeded issues.
3. Review report quality.

Expected Detection. At least 4 of 5 seeded weaknesses found with reproducible steps.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Reports exportable; CI hook available.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Scan report.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-028: Per-Application Policy as Code

Risk Addressed. Policies configured by hand per app drift and cannot be reviewed.

Business Scenario. Engineering wants policies stored as versioned code.

Technical Scenario. Export, edit and re-import a policy for one app.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 1 app policy.

Procedure

1. Export policy.
2. Change a rule.
3. Re-import and test.
4. View version history.

Expected Detection. Round-trip works; change takes effect; version history shows the change.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Policies stored in a version-control repository.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Policy diff; history.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-029: Orphaned and Decommissioned AI Application Detection

Risk Addressed. Abandoned AI apps keep keys, data and access alive.

Business Scenario. Governance wants apps with no recent use flagged.

Technical Scenario. Leave one app idle and review the report.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 2 apps; 1 idle.

Procedure

1. Set the idle threshold.
2. Wait or backdate usage.
3. Review the flag.

Expected Detection. Idle app flagged with last-use time and owner.

Expected Prevention / Control Action. Flag for review.

Expected Alert / Log. Configuration state, audit record or export that supports the claim, with timestamp and the identity of the person who produced it.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Link to decommission workflow.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = partially shown or shown only on documentation; 5 = shown in the live product with exportable evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Report entry.

Reviewer Notes. Confirm the evidence is taken from the live product in the PoC tenant. Documentation alone scores no higher than 3.

Back to layer index

TC-L05-030: Protection Overhead Under Load

Risk Addressed. High latency drives teams to remove controls.

Business Scenario. Engineering wants measured impact on response time.

Technical Scenario. Run a load test with and without protection.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test.

Test Data. 100 requests per minute for 10 minutes.

Procedure

1. Baseline without protection.
2. Repeat with protection.
3. Compare p50 and p95.

Expected Detection. Added p95 latency within the vendor's documented figure and agreed threshold.

Expected Prevention / Control Action. N/A.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Metrics exportable.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Latency table.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-031: AI Coding Assistant Extension Inventory and Version Risk

Risk Addressed. Unapproved or outdated assistant extensions run with full workspace access and are invisible to web-based discovery.

Business Scenario. Security needs a list of every AI coding extension by developer, IDE and version, with unapproved and outdated ones flagged.

Technical Scenario. Install approved, unapproved and outdated assistant extensions across IDEs and compare the discovery output with what was installed.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test. Approved-extension list with minimum versions loaded into the platform.

Test Data. 6 extension installs across 3 workstations and 2 IDE families: 2 approved and current, 2 approved but outdated, 2 unapproved (one side-loaded from a file).

Procedure

1. Record the installs as ground truth.
2. Run discovery.
3. Compare extension name, publisher, version, IDE and user against ground truth.
4. Check that unapproved and outdated installs are flagged distinctly.
5. Uninstall one extension and confirm the inventory updates.

Edge Cases / Variants. Extension installed but disabled; extension inside a remote development container; portable IDE build.

Expected Detection. 6 of 6 installs found with correct publisher and version; the 2 unapproved and 2 outdated installs are flagged.

Expected Prevention / Control Action. Unapproved extension blocked or removed by policy where the vendor claims enforcement; otherwise N/A (visibility control).

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Inventory exportable to CMDB or asset management.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Inventory export; ground-truth comparison table.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-032: Personal vs Corporate Account in AI Coding Assistants

Risk Addressed. A developer signed in to a coding assistant with a personal account sends corporate code to a tenant with consumer terms and no enterprise controls.

Business Scenario. Engineering wants corporate-licence use allowed and personal-account use on corporate repositories stopped.

Technical Scenario. Sign the same assistant in with a corporate test account and a personal test account and check that the account type is distinguished and policy applied.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test. One AI coding assistant with a corporate test tenant and a personal test account; fabricated restricted repository.

Test Data. 1 assistant; 2 test accounts (corporate tenant, personal); 5 prompts referencing the fabricated repository.

Procedure

1. Sign in with the corporate account and send the 5 prompts.
2. Sign in with the personal account and send the same 5 prompts.
3. Verify the account type and tenant or account identifier recorded for each prompt.
4. Apply a block-personal-account policy and repeat both runs.
5. Confirm the corporate run is unaffected.

Edge Cases / Variants. Two accounts signed in at once; bring-your-own API key entered in extension settings; account switch mid-session.

Expected Detection. 5 of 5 personal-account prompts identified as personal with a tenant or account identifier; corporate prompts labelled corporate.

Expected Prevention / Control Action. Personal-account requests blocked; corporate-account requests allowed.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Account type visible in SIEM events.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Event export showing account type; block evidence.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-033: Coding Assistant Workspace Context Scope and File Exclusion

Risk Addressed. Assistants gather context automatically from open files, neighbouring files and indexes, so secrets and restricted files are sent without the developer choosing to share them.

Business Scenario. Security wants named paths and file types (environment files, key material, restricted modules) never used as assistant context.

Technical Scenario. Configure exclusions, then use completion and chat in a workspace that contains excluded files and inspect what context leaves the workstation.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test. Outbound assistant traffic observable at the lab egress point or through the platform's own context log.

Test Data. Fabricated repository with 1 environment file holding fake secrets, 1 key file holding a non-functional key, 1 restricted module and 5 ordinary files; an exclusion policy covering the first three.

Procedure

1. Open the three excluded files in editor tabs.
2. Request completions in an ordinary file.
3. Ask the chat a question that the excluded content would help answer.
4. Capture the outbound context for each request.
5. Repeat with workspace indexing enabled.

Edge Cases / Variants. Excluded file referenced by explicit mention; exclusion defined in a repository-level ignore file that a developer edits; symlinked path.

Expected Detection. Any attempt to include an excluded file in context is detected and logged with the file path.

Expected Prevention / Control Action. 0 of 3 excluded files appear in the captured outbound context in any mode.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Exclusion policy managed centrally and versioned.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Outbound context capture; policy export; event log.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-034: Insecure Code Suggestion Detection

Risk Addressed. Assistants can suggest code with injection flaws, weak cryptography or hard-coded credentials that developers accept without review.

Business Scenario. Application security wants insecure suggestions flagged at the point of acceptance or caught before merge.

Technical Scenario. Prompt the assistant for code in patterns known to elicit insecure output and check whether the platform flags each suggestion.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test. Assistant pinned to a fixed model version; a commit hook or pull request check connected if the vendor claims downstream enforcement.

Test Data. 20 prompts: 5 on database query construction, 5 on cryptography, 5 on authentication and privileged access, 5 benign controls.

Procedure

1. Run each prompt and record the suggestion.
2. Record whether the suggestion is flagged and with which weakness category.
3. Accept 3 flagged suggestions and observe the commit hook or pull request check.
4. Compute the detection rate and the false-positive rate.

Edge Cases / Variants. Suggestion produced by inline completion; comments in a non-English language; a second programming language.

Expected Detection. At least 12 of 15 insecure suggestions flagged with a weakness category; no more than 1 of 5 controls flagged.

Expected Prevention / Control Action. Accepted insecure code is blocked or annotated at commit or pull request where the vendor claims enforcement.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Findings appear in the existing code scanning or pull request workflow.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Suggestion log with verdicts; rate calculation; commit or pull request check output.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index

TC-L05-035: AI-Generated Code: Licence and Public-Code Match Detection

Risk Addressed. Suggestions that reproduce public code can carry licence obligations the organisation has not accepted.

Business Scenario. Legal and engineering want suggestions that match public code flagged or filtered, with a record for audit.

Technical Scenario. Elicit suggestions that reproduce seeded licensed code and check matching, filtering and logging.

Preconditions. Isolated PoC lab provisioned; test applications, test tenants and synthetic data seeded per the Lab Prerequisites appendix; platform integration or SDK deployed on the applications under test. Mock assistant or lab model seeded to return known snippets; licence policy that denies copyleft terms.

Test Data. 15 seeded snippets from a fabricated corpus: 5 under copyleft terms, 5 under permissive terms, 5 original.

Procedure

1. Trigger each of the 15 suggestions.
2. Record which are flagged or suppressed.
3. Verify the licence family and source reference in each event.
4. Produce the audit report by repository.
5. Turn the filter off and confirm logging continues.

Edge Cases / Variants. Partial match with renamed identifiers; snippet shorter than the matching threshold.

Expected Detection. 10 of 10 matching snippets identified with licence family; 0 of 5 original snippets flagged.

Expected Prevention / Control Action. The 5 copyleft suggestions are suppressed or marked before acceptance.

Expected Alert / Log. Log entry with timestamp, user or application identity, device or host, destination, policy or rule matched, action taken and classification, within the vendor's documented SLA.

Expected Report / Dashboard Evidence. Finding visible in the application inventory or runtime protection dashboard within the documented refresh interval.

Expected Integration Evidence. Report exportable for legal review.

Forensic Evidence. Application, model, request and response identifiers, policy decision and timestamp exportable for incident reconstruction.

Compliance Evidence. Test execution log and captured evidence retained in the Test Case Execution Register for audit.

Scoring Criteria. 0 = not demonstrated; 3 = detected or enforced but one or more attribution or evidence fields missing, or SLA exceeded; 5 = fully met with complete evidence; N/A = architecture out of scope.

Pass Criteria. Expected Detection and Expected Prevention / Control Action are met in full within the vendor's documented SLA, and the evidence listed under Evidence to Capture is obtained from the live PoC.

Fail Criteria. Any seeded item is missed, the control action does not occur where required, SLA is exceeded, attribution fields are missing, or the result can only be reproduced with vendor-supplied data.

Evidence to Capture. Match report; event export.

Reviewer Notes. Confirm evidence comes from the live PoC environment, not vendor-supplied demo data. Record the build and policy version tested.

Back to layer index