Critical Architecture Flaw: The vector database was implemented as a single shared index accessible to all employees. No access control filters were applied during document retrieval—every user query could retrieve content from any uploaded document regardless of who uploaded it or its sensitivity classification.

Why Security Was Bypassed: Project classified as “internal productivity tool” by IT leadership. Security team only consulted on network connectivity, not data access controls. Timeline pressure (6-week deployment mandate) eliminated time for security review. Board viewed AI as innovation priority requiring rapid execution.

Mon
09:00
November 12, 2025 – CFO Uploads Confidential Contracts

Mon
11:30
Vector Database Processes Contracts

Wed
14:30
November 14, 2025 – Unauthorized Access Via Query

Thu
10:15
November 15, 2025 – Follow-up Query Exposes Client Names

Fri
09:00
November 16, 2025 – Incident Discovery

1. Document Upload (No Access Control Tagging)

# Simplified actual process
def upload_document(file, user_id):
    text = extract_text(file)
    chunks = split_into_chunks(text, chunk_size=300)
    
    for chunk in chunks:
        embedding = generate_embedding(chunk)
        metadata = {
            "user_id": user_id,
            "upload_date": datetime.now(),
            "source_file": file.name
            # MISSING: sensitivity_level, authorized_users, department
        }
        vector_db.insert(embedding, metadata, chunk_text)
        

Problem: No classification of data sensitivity or authorized audience

2. Query Processing (No Permission Filtering)

# How the query actually executed
def process_query(query_text, user_id):
    query_embedding = generate_embedding(query_text)
    
    # Similarity search - NO user-based filtering
    results = vector_db.search(
        query_embedding, 
        top_k=10,
        # MISSING: filter={"authorized_users": [user_id]}
    )
    
    # All semantically similar content returned regardless of uploader
    context = [r.chunk_text for r in results]
    
    # LLM generates response using ALL retrieved context
    response = llm.generate(query_text, context)
    return response
        

Problem: Similarity search returns ANY matching content without checking if querying user has authorization

3. Response Generation (No Output Filtering)

LLM synthesizes response directly from retrieved chunks. No scanning of output for:

• Client names or identifiable information
• Financial figures or contract values
• Payment terms or commercial conditions
• Source document metadata (e.g., “this came from CFO’s upload”)

# Required classification interface
def upload_document(file, user_id):
    # STEP 1: User must classify data
    classification = prompt_user_classification(
        options=["Public", "Internal", "Confidential", "Restricted"]
    )
    
    if classification == "Restricted":
        # STEP 2: For restricted data, specify authorized audience
        authorized_users = prompt_authorized_users()
        authorized_departments = prompt_authorized_departments()
    
    # STEP 3: Automatic scanning for sensitive patterns
    auto_detected_sensitivity = scan_for_pii_financial_data(file)
    if auto_detected_sensitivity > classification:
        warn_user_insufficient_classification()
        require_confirmation()
    
    # STEP 4: Store with full metadata
    metadata = {
        "classification": classification,
        "uploader": user_id,
        "authorized_users": authorized_users,
        "authorized_departments": authorized_departments,
        "upload_date": datetime.now(),
        "retention_policy": get_retention_for_classification(classification)
    }
    
    chunks = process_document(file)
    for chunk in chunks:
        embedding = generate_embedding(chunk)
        vector_db.insert(embedding, metadata, chunk)
        

Result: CFO would have classified contracts as “Restricted – Finance Only,” preventing marketing analyst access.

def process_query_with_access_control(query_text, user_context):
    # Generate query embedding
    query_embedding = generate_embedding(query_text)
    
    # Retrieve potentially relevant documents
    candidate_results = vector_db.search(query_embedding, top_k=50)
    
    # CRITICAL: Filter by user permissions
    authorized_results = []
    for result in candidate_results:
        if check_user_authorization(user_context, result.metadata):
            authorized_results.append(result)
    
    # If no authorized results, return safe default
    if len(authorized_results) == 0:
        return "I don't have access to information matching your query. This may be because the relevant documents are restricted to other departments."
    
    # Generate response only from authorized content
    context = [r.chunk_text for r in authorized_results[:10]]
    response = llm.generate(query_text, context)
    
    # Log access for audit trail
    log_query_access(user_context.user_id, query_text, 
                     [r.metadata for r in authorized_results])
    
    return response

def check_user_authorization(user_context, doc_metadata):
    # Check 1: User is uploader
    if user_context.user_id == doc_metadata.uploader:
        return True
    
    # Check 2: Public or Internal documents
    if doc_metadata.classification in ["Public", "Internal"]:
        return True
    
    # Check 3: User in authorized list
    if user_context.user_id in doc_metadata.authorized_users:
        return True
    
    # Check 4: User's department authorized
    if user_context.department in doc_metadata.authorized_departments:
        return True
    
    # Check 5: User has required role
    if any(role in user_context.roles for role in doc_metadata.required_roles):
        return True
    
    # Default deny
    return False
        

Result: Marketing analyst’s query would have returned “No authorized information available,” preventing contract exposure.

def filter_response_output(response_text, user_context, source_documents):
    # Scan for sensitive patterns
    detected_issues = []
    
    # Check 1: PII detection
    pii_matches = scan_for_pii(response_text)
    if pii_matches:
        response_text = redact_pii(response_text)
        detected_issues.append("PII_REDACTED")
    
    # Check 2: Financial data
    financial_patterns = scan_for_currency_amounts(response_text)
    for amount in financial_patterns:
        # Verify user authorized to see this financial data
        if not user_authorized_for_financial_detail(user_context, amount):
            response_text = response_text.replace(amount, "[REDACTED]")
            detected_issues.append("FINANCIAL_REDACTED")
    
    # Check 3: Source document authorization
    for source_doc in source_documents:
        if not check_user_authorization(user_context, source_doc.metadata):
            # Should never happen if retrieval filtering works correctly
            # But defense in depth: remove this content
            response_text = remove_content_from_source(
                response_text, source_doc
            )
            detected_issues.append("UNAUTHORIZED_SOURCE_REMOVED")
            alert_security_team("Output filter caught unauthorized source leak")
    
    # Check 4: Client name detection
    known_client_names = get_confidential_client_list()
    for client_name in known_client_names:
        if client_name in response_text:
            if not user_authorized_for_client_info(user_context, client_name):
                response_text = response_text.replace(
                    client_name, "[CONFIDENTIAL CLIENT]"
                )
                detected_issues.append("CLIENT_NAME_REDACTED")
    
    # Log filtering actions
    if detected_issues:
        log_output_filtering(user_context.user_id, detected_issues)
    
    return response_text
        

Result: Even if access control failed, output filter would have redacted “Phoenix Bank” and specific contract values before delivery to analyst.

Automated Test Scenario

def weekly_access_control_test():
    """
    Automated test simulating the exact scenario that occurred
    """
    # SETUP: Test CFO uploads confidential contract
    test_cfo = create_test_user(role="CFO", department="Finance")
    test_contract = create_test_document(
        content="Client: Phoenix Bank, Value: £12.5M, Term: 36 months...",
        classification="Restricted",
        authorized_departments=["Finance", "Legal"]
    )
    
    upload_result = upload_document(test_contract, test_cfo)
    assert upload_result.success
    
    # TEST: Marketing user attempts to query for contract info
    test_analyst = create_test_user(role="Analyst", department="Marketing")
    test_queries = [
        "What are our Q4 contract values?",
        "Tell me about the Phoenix Bank deal",
        "Show me recent financial services contracts",
        "I need contract examples for a case study"
    ]
    
    for query in test_queries:
        response = process_query(query, test_analyst)
        
        # VERIFY: Response should NOT contain restricted info
        violations = []
        
        if "Phoenix Bank" in response:
            violations.append("Client name leaked")
        if "£12.5" in response or "12.5M" in response:
            violations.append("Contract value leaked")
        if "36 months" in response and "term" in response.lower():
            violations.append("Contract terms leaked")
        
        if violations:
            alert_security_team(
                f"ACCESS CONTROL FAILURE: {', '.join(violations)}",
                query=query,
                user=test_analyst,
                response=response
            )
            halt_ai_system()
            return TEST_FAILED
    
    return TEST_PASSED

# Schedule to run every Sunday at 02:00
schedule_weekly_test(weekly_access_control_test)
        

Result: This test would have detected the cross-contamination vulnerability BEFORE real data was exposed, allowing fix before production use.

Critical Point: FinancialTech Corp did ZERO testing of access controls before deployment. This single weekly automated test (takes 5 minutes to run) would have prevented £30M+ in losses.