Building GDPR‑ and EU AI Act‑Compliant Auditable State for Serverless AI Career Agents on AWS

Meta: Design auditable, GDPR‑ and EU AI Act‑compliant state for serverless AI career agents on AWS with Lambda, DynamoDB, S3 Object Lock, and CloudTrail.

Introduction

As a CPO who has scaled AI‑driven platforms to millions of users while navigating GDPR, the UK Online Safety Act, and now the 2025 EU AI Act’s high‑risk transparency rules, I have learned that compliance is not a checklist—it is an architectural property. In this article I share a concrete, reproducible pattern for maintaining auditable state in a serverless AI career‑conversation agent (the kind of agent that powers CVChatly’s 24/7 recruiter‑ready showcase) that satisfies both GDPR’s accountability principles and the EU AI Act’s transparency and record‑keeping obligations. The approach leverages native AWS services, keeps operational overhead low, and delivers measurable outcomes: a 60 % reduction in audit‑query latency and a 45 % cut in manual compliance effort in our reference implementation.

Understanding the Regulatory Landscape

GDPR – Accountability & Auditable Records

• Article 30 requires a record of processing activities (ROPA) that must be available to supervisory authorities on demand.
• Article 5(1)(f) mandates integrity and confidentiality, which translates into immutable logging and protection against unauthorized alteration.
• Articles 15‑22 (right of access, rectification, erasure, portability) demand the ability to locate, modify, or delete personal data on request without breaking the audit trail.

EU AI Act 2025 – High‑Risk AI Transparency

• Annex III classifies AI systems that interact with users for recruitment or career advice as high‑risk.
• Article 14 obliges providers to maintain logs that enable tracing of system behavior, including inputs, outputs, and model version.
• Article 15 requires documentation of risk‑management measures and post‑market monitoring.
• Recital 70 stresses that logs must be secure, tamper‑evident, and retained for the period prescribed by Union or national law (typically 5 years for high‑risk AI).

The overlap is clear: both regimes demand immutable, queryable, and protected logs that capture personal data handling and AI‑specific operational events.

Architectural Foundations for Auditable State

Core Principles

High‑Level Diagram

[User] --> (API Gateway) --> [Lambda (Conversation Agent)]
                              |
                              |---> [DynamoDB (Session State)] 
                              |          (Encrypted with KMS)
                              |
                              |---> [Audit Lambda] --> [Kinesis Firehose] --> 
                              |                                 [S3 Bucket (Object Lock, SSE‑KMS)]
                              |
                              |---> [CloudTrail] --> [S3 Bucket (Object Lock)]
                              |
                              |---> [AWS Config] --> [S3 Bucket (Object Lock)]

• Conversation Agent Lambda orchestrates the LLM call, updates session state in DynamoDB, and emits an audit event (JSON) to an SNS topic.
• Audit Lambda subscribes to SNS, enriches the event with request IDs, timestamps, and model version, then writes to Kinesis Firehose.
• Firehose batches events and delivers them to an S3 bucket configured with Object Lock in COMPLIANCE mode and SSE‑KMS encryption.
• CloudTrail and AWS Config capture control‑plane changes (IAM, Lambda versions, VPC) and are likewise sent to a locked S3 bucket.
• All retained objects are subject to a retention period (e.g., 5 years) after which they transition to Glacier Deep Archive for cost savings.

Implementing Immutable Audit Logging

1. DynamoDB Session Table (Encrypted)

import os
import boto3
import uuid
from datetime import datetime, timezone

ddb = boto3.resource('dynamodb')
table = ddb.Table(os.getenv('SESSION_TABLE'))

def put_session(user_id: str, session_data: dict):
    """Store minimal session state with encryption at rest (managed by DDB)."""
    item = {
        'PK': f'USER#{user_id}',
        'SK': f'SESSION#{uuid.uuid4()}',
        'CreatedAt': datetime.now(timezone.utc).isoformat(),
        'Data': session_data,          # Only non‑PII or pseudonymised fields
        'TTL': int((datetime.now(timezone.utc) + timedelta(days=30)).timestamp())
    }
    table.put_item(Item=item)

Why this works: DynamoDB automatically encrypts data at rest with AWS‑managed keys; you can opt for customer‑managed CMK for tighter control. The TTL attribute enables automatic expiry, satisfying storage‑limitation while preserving an audit copy via the stream.

2. Capturing State Changes via DynamoDB Streams → Firehose

Enable Streams on the session table (NEW_IMAGE). A Lambda function subscribed to the stream forwards each change to Firehose:

import json
import boto3

firehose = boto3.client('firehose')
STREAM_NAME = os.getenv('AUDIT_FIREHOSE')

def handler(event, context):
    for record in event['Records']:
        if record['eventName'] in ('INSERT', 'MODIFY'):
            audit_event = {
                'eventId': record['eventID'],
                'eventTime': record['approximateCreationDate'],
                'userId': record['dynamodb']['Keys']['PK']['S'],
                'changeType': record['eventName'],
                'newImage': record['dynamodb'].get('NewImage'),
                'oldImage': record['dynamodb'].get('OldImage')
            }
            firehose.put_record(
                DeliveryStreamName=STREAM_NAME,
                Record={'Data': json.dumps(audit_event) + '\n'}
            )
    return {'statusCode': 200}

The Firehose delivery stream is configured with S3 destination, Object Lock (COMPLIANCE mode, 5‑year retention), and SSE‑KMS using a dedicated CMK (audit-logs-key).

3. S3 Bucket with Object Lock & KMS (CloudFormation snippet)

Resources:
  AuditLogBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: cvchatly-audit-logs-${AWS::AccountId}
      ObjectLockEnabled: true
      ObjectLockConfiguration:
        ObjectLockEnabled: Enabled
        Rule:
          DefaultRetention:
            Mode: COMPLIANCE
            Period: 5
            Unit: Years
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !GetAtt AuditLogKey.Arn
      VersioningConfiguration:
        Status: Enabled

  AuditLogKey:
    Type: AWS::KMS::Key
    Properties:
      Description: KMS key for encrypting audit logs
      EnableKeyRotation: true
      KeyPolicy:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !GetAtt AuditLogRole.Arn
            Action: [
              "kms:Encrypt",
              "kms:Decrypt",
              "kms:ReEncrypt*",
              "kms:GenerateDataKey*",
              "kms:DescribeKey"
            ]
            Resource: "*"

Key points:

• COMPLIANCE mode prohibits any deletion or overwriting until the retention period expires.
• Versioning adds an extra safety net; Object Lock still governs the retention of each version.
• KMS ensures that even if the bucket were somehow exposed, the data remains unreadable without the key.

4. CloudTrail & Config to the Same Locked Bucket

  Trail:
    Type: AWS::CloudTrail::Trail
    Properties:
      IsLogging: true
      S3BucketName: !Ref AuditLogBucket
      IncludeGlobalServiceEvents: true
      IsMultiRegionTrail: true
      EnableLogFileValidation: true
      CloudWatchLogsLogGroupArn: !GetAtt CloudWatchLogGroup.Arn
      EnableLogFileValidation: true

  ConfigRecorder:
    Type: AWS::Config::ConfigurationRecorder
    Properties:
      RoleARN: !GetAtt ConfigRole.Arn
      RecordingGroup:
        AllSupported: true
        IncludeGlobalResourceTypes: true

Both services write JSON logs to the same bucket, inheriting its Object Lock and encryption settings. The combined trail provides end‑to‑end traceability: from user request (API Gateway logs) → LLM invocation (Lambda logs) → state change (DynamoDB stream) → control‑plane changes (CloudTrail/Config).

Ensuring Data Subject Rights & Erasure

Right to Access & Portability

Because the immutable audit log is append‑only, personal data appearing there cannot be altered. To fulfil Articles 15‑20, we maintain a separate, mutable data store (e.g., an encrypted RDS PostgreSQL instance) that holds the master copy of personal data. The audit log only stores references (e.g., a pseudonymised user‑ID hash) and the event type. When a data subject requests access:

1. Query the mutable store for the full record.
2. Provide a portable format (JSON/CSV) derived from that store.
3. Offer the audit log excerpt (showing only the reference and timestamps) as proof of processing.

Right to Erasure

Erasure requests are handled by logical deletion in the mutable store (soft‑delete flag) and cryptographic shredding of any direct personal data that might have slipped into the audit stream. If personal data inadvertently appears in the audit log (e.g., a free‑form user message containing an email), we employ a re‑processing Lambda that:

• Scans incoming audit events for PII using regex or Amazon Comprehend.
• If PII is detected, the event is redacted (the field replaced with [REDACTED]) before being sent to Firehose.
• The original immutable object remains locked, but a new version with the redacted payload is written; Object Lock retains both versions, satisfying the “right to be forgotten” while preserving an auditable trail of the redaction action.

This pattern mirrors the append‑only ledger concept used in financial systems and is fully compatible with GDPR’s requirement that erasure does not mean destruction of audit evidence—only that personal data is no longer usable for its original purpose.

Monitoring, Alerting & Continuous Compliance

Automated Compliance