GPU_WORKLOAD_MISMATCH Part II: From Detection to Runtime Enforcement for AI Infrastructure
How continuous GPU workload verification could turn an AI security finding into a runtime security control
By Carnell E. Smith — Founder, Champtron Systems LLC
Part II of the GPU_WORKLOAD_MISMATCH series
New to the concept? Start with:
Part I — GPU_WORKLOAD_MISMATCH: A Novel Security Finding Category for AI Container Workloads
Detection Is Only the Beginning
In Part I, I introduced GPU_WORKLOAD_MISMATCH as a security finding category for GPU-accelerated AI environments.
The underlying premise is simple:
A container being authorized to access a GPU does not mean that every GPU workload executed by that container should automatically be trusted.
Traditional container security controls can answer important questions:
- Is the container image approved?
- Does the image contain known vulnerabilities?
- Is the container running with excessive privileges?
- Has its configuration changed?
- Is the expected process running?
GPU-accelerated AI introduces another layer of runtime trust:
Is the workload actually using the GPU in the way we expected it to?
That question becomes increasingly important as organizations deploy:
- Local large language models
- Inference services
- Training and fine-tuning jobs
- AI agents
- Retrieval-augmented generation systems
- Multi-model services
- Concurrent GPU workloads
- Containerized AI applications
Detecting unexpected GPU behavior is useful.
But detection is only the beginning.
The next question is more important:
What should happen when observed GPU workload behavior no longer matches the workload that was authorized?
That moves GPU_WORKLOAD_MISMATCH from a detection problem toward a runtime enforcement and continuous-assurance problem.
1. The Runtime Trust Problem
Consider an approved AI container.
At deployment time, everything appears valid:
Image → Approved
Image Digest → Verified
Container Policy → Passed
GPU Access → Authorized
Expected Model → Known
Admission Controls → Passed
Runtime Start → Successful
From a traditional deployment perspective, the workload may be considered trusted.
But trust established at deployment should not automatically become permanent trust.
After execution begins, the environment can change.
A container might launch an unexpected process.
An AI agent might invoke a tool that starts an additional GPU workload.
A compromised dependency could initiate unauthorized computation.
A model-serving container could begin executing a workload outside its intended operational profile.
GPU activity could appear inside a container that was never expected to use GPU resources.
The container itself may still be legitimate.
The GPU workload may not be.
That creates an important distinction:
Container identity is not necessarily GPU workload identity.
For AI infrastructure, verifying the container is increasingly only one component of verifying the workload.
2. From Static Trust to Continuous Trust
A Zero Trust approach should not ask only:
Was this workload trusted when it started?
It should continuously ask:
Does this workload still satisfy the conditions under which we decided to trust it?
For GPU-accelerated infrastructure, that suggests a continuous verification lifecycle:
Expected Workload Identity
↓
Runtime GPU Observation
↓
Container / Process / GPU Correlation
↓
Expected vs. Observed Comparison
↓
GPU_WORKLOAD_MISMATCH
↓
Risk Evaluation
↓
Policy Decision
↓
Response
↓
Evidence Generation
↓
Continuous Re-evaluation
The important architectural change is that GPU_WORKLOAD_MISMATCH is no longer simply an alert.
It becomes a security signal capable of informing a policy decision.
3. Proposed GPU-Aware Runtime Security Architecture
The following conceptual architecture illustrates how this could work.
┌───────────────────────────────────────────────────────────────┐
│ EXPECTED WORKLOAD IDENTITY │
│ │
│ Container │ Image │ Process │ GPU │ Model │ Resource Policy │
└───────────────────────────────┬───────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────┐
│ RUNTIME OBSERVATION │
│ │
│ GPU Telemetry │ Container Runtime │ Process │ Model Activity │
└───────────────────────────────┬───────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────┐
│ CHAMP CONTAINERGUARD CORRELATION │
│ │
│ Expected State ←→ Observed Runtime State │
└───────────────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────┐
│ GPU WORKLOAD MATCH? │
└─────────┬───────────┘
│
┌──────────┴──────────┐
│ │
YES NO
│ │
▼ ▼
┌─────────────────┐ ┌────────────────────────┐
│ Continue Trust │ │ GPU_WORKLOAD_MISMATCH │
│ + Monitoring │ │ Finding Generated │
└────────┬────────┘ └───────────┬────────────┘
│ │
│ ▼
│ ┌──────────────────────┐
│ │ RISK EVALUATION │
│ │ │
│ │ Identity Confidence │
│ │ Behavior Deviation │
│ │ Workload Criticality │
│ │ Persistence │
│ │ Threat Context │
│ └──────────┬───────────┘
│ │
│ ▼
│ ┌──────────────────────┐
│ │ POLICY DECISION │
│ └──────────┬───────────┘
│ │
│ ┌────────────────┼────────────────┐
│ │ │ │
│ ▼ ▼ ▼
│ ALERT QUARANTINE TERMINATE
│ │ │ │
│ └────────────────┼────────────────┘
│ │
│ ▼
│ ┌──────────────────────┐
│ │ EVIDENCE GENERATION │
│ │ │
│ │ Workload Identity │
│ │ GPU Identity │
│ │ Runtime Observation │
│ │ Policy Decision │
│ │ Response Action │
│ │ Timestamp + Integrity│
│ └──────────┬───────────┘
│ │
└──────────────┬──────────┘
│
▼
CONTINUOUS RE-EVALUATION
│
└──────────────► LOOP
Three Questions Drive the Architecture
The model separates three questions that are often treated as one.
1. What should be running?
Defined through expected workload identity and policy.
2. What is actually running?
Determined through runtime observation of the container, process, GPU, model, and surrounding workload.
3. What should happen when those states diverge?
Determined through risk evaluation and policy.
The difference between the first two states is where GPU_WORKLOAD_MISMATCH becomes meaningful.
This changes the objective from:
Monitor the GPU.
to:
Continuously verify that GPU activity remains consistent with the identity and policy of the authorized AI workload.
4. Defining Expected GPU Workload Identity
Runtime verification requires an expected state.
A security platform therefore needs some representation of what legitimate GPU behavior looks like for a particular workload.
A simplified policy could look like this:
workload:
name: approved-inference-service
container:
image: registry.example.com/inference:v3
digest: sha256:EXPECTED_DIGEST
gpu:
access: required
expected_processes:
- python
- tritonserver
expected_models:
- approved-model
max_gpu_memory_percent: 75
allow_child_gpu_processes: false
response:
mismatch:
severity: high
action: quarantine
This is a conceptual example rather than a proposed universal schema.
The important principle is:
Expected GPU behavior becomes security policy data.
Runtime observations can then be compared against that expected state.
5. Scenario: Unexpected GPU Process
Assume an approved inference container normally launches:
python inference_server.py
The container is authorized to access the GPU.
Later, runtime monitoring observes another GPU-consuming process:
unknown_worker
Traditional monitoring might tell us:
GPU utilization increased.
GPU workload verification asks a different question:
Was
unknown_workerauthorized to consume GPU resources as part of this workload?
If the answer is no, the resulting security finding could conceptually resemble:
{
"finding": "GPU_WORKLOAD_MISMATCH",
"severity": "HIGH",
"container": "inference-api",
"expected_gpu_process": "python",
"observed_gpu_process": "unknown_worker",
"policy_result": "DENY",
"recommended_action": "QUARANTINE"
}
The security value is not merely GPU utilization.
The value is correlation between expected identity and observed activity.
6. Scenario: GPU Activity Where None Was Expected
Consider an application container whose policy states:
gpu:
access: prohibited
Runtime observation later identifies GPU activity correlated with that workload.
The important security event is not:
GPU utilization = 12%
It is:
Expected GPU usage = NONE
Observed GPU usage = PRESENT
That difference becomes the finding:
Finding: GPU_WORKLOAD_MISMATCH
Expected: NO_GPU
Observed: GPU_ACTIVE
Severity: HIGH
Potential explanations could include:
- Configuration drift
- Incorrect device exposure
- Unexpected child processes
- Compromised workload behavior
- Incorrect workload placement
- Policy misconfiguration
The mismatch does not automatically prove malicious activity.
It identifies a security-relevant divergence requiring explanation.
That distinction matters.
7. Scenario: Approved Container, Unexpected Model
AI infrastructure introduces another identity layer:
model identity.
A container may be authorized.
The GPU may be authorized.
The inference process may also be authorized.
But what if the model changes?
Suppose policy expects:
approved-model-v4
while runtime evidence indicates:
experimental-model.bin
Infrastructure monitoring may report a perfectly healthy system.
From a security and governance perspective, however, something important has changed.
This suggests that future GPU-aware security architectures may need to correlate:
Container Identity
+
Process Identity
+
GPU Identity
+
Model Identity
+
Runtime Behavior
↓
Continuous Workload Identity
This is fundamentally different from treating each telemetry source independently.
8. Agentic AI Changes the Security Model
Autonomous AI agents make continuous verification even more important.
Traditional applications generally follow relatively predictable execution paths.
Agents can dynamically:
- Select tools
- Generate code
- Invoke subprocesses
- Interact with APIs
- Start jobs
- Launch containers
- Modify files
- Invoke models
- Initiate additional AI workloads
An initially approved AI workload can therefore create new execution paths during runtime.
The security question changes from:
Is this AI application authorized?
to:
Are the actions and GPU workloads generated by this authorized AI application still operating within policy?
That is a substantially more dynamic security problem.
It is also one reason I believe continuous GPU workload verification will become increasingly relevant as agentic systems move into production.
9. Detection Should Inform Policy
A mature GPU workload security architecture should eventually do more than generate HIGH, MEDIUM, or LOW findings.
The finding should be capable of informing a policy engine.
For example:
IF
container_identity == approved
AND
gpu_process_identity != expected
THEN
severity = HIGH
recommended_response = QUARANTINE
Another policy:
IF
gpu_access == prohibited
AND
observed_gpu_activity == true
THEN
severity = CRITICAL
recommended_response = TERMINATE
Model identity could introduce another condition:
IF
expected_model_hash != observed_model_hash
THEN
severity = CRITICAL
recommended_response = BLOCK_MODEL_EXECUTION
Actual enforcement mechanisms will depend heavily on the runtime, orchestrator, GPU architecture, workload, and operational environment.
Potential responses include:
- Alert
- Increase telemetry collection
- Capture forensic evidence
- Revoke workload authorization
- Restrict GPU access
- Isolate the container
- Quarantine the workload
- Terminate a process
- Terminate a container
- Trigger re-attestation
- Require human approval
The key principle is:
Response should be proportional to confidence, context, and potential impact.
10. Automatic Enforcement Can Be Dangerous
Not every deviation is malicious.
AI workloads are particularly dynamic.
Model servers may legitimately create workers.
Frameworks may dynamically allocate resources.
Inference engines may load additional runtime components.
Agents may intentionally create subprocesses.
An overly aggressive policy could disrupt legitimate workloads.
A runtime policy engine therefore needs context.
Confidence
How certain is the system that observed behavior violates policy?
Context
Is this production, development, testing, research, or training?
Severity
Does the mismatch indicate unusual behavior or violation of a defined security boundary?
Persistence
Did the mismatch happen once, or does it continue?
Blast Radius
Could the workload affect other models, tenants, data, services, or GPU resources?
A conceptual risk model could look like:
Risk Score =
Identity Confidence
+ Behavioral Deviation
+ Resource Sensitivity
+ Workload Criticality
+ Persistence
+ Threat Context
The exact scoring methodology requires careful validation.
The larger point is that enforcement should become risk-informed rather than reflexively binary.
11. Preserve Evidence Before Responding
There is another problem with immediately terminating suspicious workloads.
Termination can destroy evidence.
Before taking destructive action, a security system may need to preserve:
- Container identity
- Image digest
- Process tree
- GPU process information
- GPU utilization
- GPU memory consumption
- GPU/device identity
- Model identity, where available
- Runtime timestamps
- Workload metadata
- Detection reason
- Policy decision
- Enforcement action
The resulting evidence record could then be integrity-protected.
Conceptually:
┌─────────────────────────┐
│ Evidence Record │
├─────────────────────────┤
│ Workload Identity │
│ GPU Identity │
│ Runtime Observation │
│ Policy Evaluation │
│ Enforcement Decision │
│ Timestamp │
└────────────┬────────────┘
│
▼
HMAC / Signature
This turns runtime detection and response into something potentially useful for:
- Incident response
- Forensic investigation
- Auditability
- Compliance evidence
- Security assurance
- Post-incident analysis
The objective should not simply be:
Stop the workload.
It should be:
Understand what happened, preserve trustworthy evidence, and then apply the appropriate response.
12. Where NVIDIA Telemetry Fits
GPU-aware runtime security does not require replacing the existing NVIDIA observability ecosystem.
It can build on it.
NVIDIA provides mechanisms for obtaining important GPU health, resource, and telemetry information. Container runtimes and orchestrators provide additional workload context.
A security correlation layer can combine those signals with expected workload policy.
Conceptually:
NVIDIA GPU Telemetry
+
Container Runtime Metadata
+
Orchestrator Metadata
+
Process Identity
+
Expected Workload Policy
│
▼
CHAMP ContainerGuard Correlation
│
▼
Expected vs. Observed State
│
▼
GPU_WORKLOAD_MISMATCH
│
▼
Risk / Policy Decision
│
▼
Evidence + Response
The distinction is important:
Telemetry tells us what happened.
Policy tells us what should have happened.
GPU_WORKLOAD_MISMATCH represents a security-relevant difference between the two.
13. Attestation and Runtime Verification Are Complementary
Hardware and platform attestation provide another important component of infrastructure trust.
Attestation can help establish that a platform or trusted execution environment is in an expected state before sensitive resources are released.
But another question remains after workload execution begins:
Does the workload continue behaving according to its authorized runtime policy?
These should be viewed as complementary security concepts.
ATTESTATION
│
▼
Can I trust the platform?
│
│
+
│
│
RUNTIME VERIFICATION
│
▼
Can I continue trusting the workload?
│
│
=
│
▼
CONTINUOUS AI INFRASTRUCTURE ASSURANCE
This distinction is important.
Establishing trust is necessary.
Maintaining trust is continuous.
14. Toward GPU-Aware Zero Trust
Zero Trust is commonly associated with the principle:
Never trust, always verify.
AI infrastructure may require that principle to extend deeper into accelerated computation.
Not simply:
Verify the user.
Verify the device.
Verify the host.
Verify the container.
But eventually:
Verify the process.
Verify the GPU workload.
Verify the model.
Verify runtime behavior.
Verify continuously.
That creates a broader trust chain:
Identity
↓
Device
↓
Host
↓
Container
↓
Process
↓
GPU
↓
Model
↓
Runtime Behavior
Each layer contributes additional context to the security decision.
This is the direction I describe as GPU-aware Zero Trust.
15. From Finding to Runtime Security Control
GPU_WORKLOAD_MISMATCH began as a detection concept.
The larger opportunity is a runtime assurance model for AI infrastructure.
The progression looks like this:
GPU Monitoring
↓
GPU Workload Correlation
↓
GPU_WORKLOAD_MISMATCH
↓
Risk Evaluation
↓
Policy Decision
↓
Runtime Response
↓
Evidence Generation
↓
Continuous Verification
That is more significant than adding another monitoring alert.
It suggests the beginnings of a:
GPU-aware runtime security control plane for AI infrastructure.
Such a control plane would not determine trust from a single signal.
Instead, it would continuously correlate workload identity, runtime activity, GPU behavior, model identity, policy, and evidence.
16. What Comes Next
The next stage of this research raises an even deeper question:
Can we prove what AI workload was actually running when a security decision was made?
That leads to several technical problems worth exploring:
- Can model identity become part of workload attestation?
- Can GPU workload evidence be cryptographically bound to container identity?
- Can runtime behavior trigger re-attestation?
- Can workload evidence remain trustworthy after the workload is terminated?
- Can security policy follow AI workloads across edge, workstation, and data-center GPU environments?
- Can organizations prove that an AI workload not only started in a trusted environment but remained within its authorized operating state?
Those questions lead directly to Part III.
Next in the Series
GPU_WORKLOAD_MISMATCH Part III
Attesting the AI Workload: Cryptographic Evidence for GPU Runtime Integrity
Part III will explore how workload identity, runtime evidence, cryptographic integrity, and attestation could contribute to a stronger chain of trust for GPU-accelerated AI systems.
The objective is no longer simply detecting unusual GPU activity.
It is establishing and maintaining:
Continuous trust in accelerated AI workloads.
Continue the Series
Part I
GPU_WORKLOAD_MISMATCH: A Novel Security Finding Category for AI Container Workloads
Introduces GPU_WORKLOAD_MISMATCH and the security gap created when container identity and observed GPU workload behavior diverge.
Part II
GPU_WORKLOAD_MISMATCH: From Detection to Runtime Enforcement for AI Infrastructure
Extends the finding into continuous verification, risk evaluation, evidence generation, and policy-driven response.
Part III — Upcoming
Attesting the AI Workload: Cryptographic Evidence for GPU Runtime Integrity
Explores how runtime evidence and attestation could contribute to continuous AI workload trust.
About the Author
Carnell E. Smith is the Founder of Champtron Systems LLC and developer of CHAMP ContainerGuard, an AI infrastructure and container security platform focused on GPU workload integrity, security evidence, Zero Trust principles, and AI workload assurance.
The GPU_WORKLOAD_MISMATCH series documents ongoing research and engineering into security controls for GPU-accelerated and agentic AI infrastructure.
Technical note: This article discusses both implemented security concepts and proposed architectural extensions. Example policies, enforcement actions, model-identity mechanisms, and runtime flows are conceptual unless explicitly identified as validated functionality. Capabilities should be independently validated against the hardware, container runtime, orchestration platform, GPU software stack, workload, and security requirements of the target environment.
Tags: #ai #cybersecurity #gpu #nvidia
