System Design Document (SDD): AI Safety, Security, and Trustworthy AI

This document defines the system architecture, data models, algorithm specifications, and integration flows for the AI Safety, Security, and Trustworthy AI framework of Octo.

---

1. Overall Pipeline Architecture

The security architecture operates as an inline firewall wrapping all AI requests (Prompt Guardrails) and validation interceptors wrapping all responses (Output Moderation).

          [Student Input]
                 │
                 ▼
     ┌───────────────────────┐
     │ 1. Regex Sanitizer    │ ──► Drops control characters & raw instruction tags
     └───────────────────────┘
                 │
                 ▼
     ┌───────────────────────┐
     │ 2. Black/Whitelist    │ ──► Checks local SQLite cached keywords (Fast Path)
     └───────────────────────┘
                 │
                 ▼
     ┌───────────────────────┐
     │ 3. Safety Classifier  │ ──► Evaluates category risks using specialized model
     └───────────────────────┘
                 │
                 ▼
     ┌───────────────────────┐
     │  4. LLM execution     │ ──► Large Model / Medium Model routing
     └───────────────────────┘
                 │
                 ▼
     ┌───────────────────────┐
     │ 5. Output Moderator   │ ──► Scans generated text for accuracy, bias, safety
     └───────────────────────┘
                 │
                 ▼
     ┌───────────────────────┐
     │ 6. Reviewer Queue /   │ ──► Pushed to Human Approval or Client Response
     │    Client Delivery    │
     └───────────────────────┘

---

2. Database Schema (D1 SQLite)

To support accountability, auditability, human reviews, and dynamic safety configuration, the following tables are defined.

2.1 Audit Logging

Stores all telemetry and parameters for audit checks:

CREATE TABLE IF NOT EXISTS ops_ai_audit_logs (
    id TEXT PRIMARY KEY,
    organization_id TEXT NOT NULL,
    user_id TEXT NOT NULL,
    timestamp TEXT NOT NULL DEFAULT (CURRENT_TIMESTAMP),
    prompt_raw TEXT NOT NULL,
    response_raw TEXT,
    model_name TEXT NOT NULL,
    prompt_tokens INTEGER DEFAULT 0,
    completion_tokens INTEGER DEFAULT 0,
    latency_ms INTEGER DEFAULT 0,
    safety_status TEXT NOT NULL, -- 'allowed', 'flagged', 'blocked'
    safety_categories_flagged TEXT, -- JSON array of strings
    evidence_payload TEXT, -- JSON backing this decision
    confidence_level REAL, -- Bayesian confidence
    workflow_id TEXT
);

CREATE INDEX IF NOT EXISTS idx_audit_user ON ops_ai_audit_logs(user_id);
CREATE INDEX IF NOT EXISTS idx_audit_time ON ops_ai_audit_logs(timestamp);

2.2 Whitelist / Blacklist Tables

Supports rule caches for local matching:

CREATE TABLE IF NOT EXISTS ops_safety_rules (
    id TEXT PRIMARY KEY,
    organization_id TEXT NOT NULL,
    rule_type TEXT NOT NULL, -- 'blacklist', 'whitelist'
    pattern TEXT NOT NULL, -- regex or exact string
    context_domain TEXT, -- e.g., 'biology', 'math', 'all'
    severity TEXT NOT NULL DEFAULT 'high', -- 'low', 'medium', 'high'
    created_at TEXT NOT NULL DEFAULT (CURRENT_TIMESTAMP)
);

CREATE UNIQUE INDEX IF NOT EXISTS idx_safety_pattern ON ops_safety_rules(pattern, rule_type);

2.3 Human Review Workflows

Manages content moderation states:

CREATE TABLE IF NOT EXISTS ops_content_reviews (
    id TEXT PRIMARY KEY,
    organization_id TEXT NOT NULL,
    content_type TEXT NOT NULL, -- 'quiz', 'lesson', 'journey_mission'
    content_id TEXT NOT NULL, -- foreign key to target content table
    draft_payload TEXT NOT NULL, -- JSON draft generated by LLM
    moderator_id TEXT, -- reviewer user ID
    status TEXT NOT NULL DEFAULT 'pending', -- 'pending', 'approved', 'rejected', 'modified'
    feedback_notes TEXT,
    created_at TEXT NOT NULL DEFAULT (CURRENT_TIMESTAMP),
    updated_at TEXT NOT NULL DEFAULT (CURRENT_TIMESTAMP)
);

CREATE INDEX IF NOT EXISTS idx_reviews_status ON ops_content_reviews(status);

---

3. Explainability and Evidence calculations

Octo implements a Bayesian network model to track student mastery. When suggesting a next learning step (Stage or Mission), the Recommendation Engine operates under the following logic:

3.1 Bayesian Inference Score (evidence updating)

Let the student's mastery of skill $S$ before evidence $E$ be $P(S)$. The updated mastery $P(S|E)$ after submitting quiz answer $E$ (correct/incorrect) is computed as:

[P(S|E) = \frac{P(E|S) \cdot P(S)}{P(E|S) \cdot P(S) + P(E|\neg S) \cdot (1 - P(S))}]

Where:

• $P(E|S)$ is the probability of a correct response given mastery (accounting for a slip factor $s = 0.1$).
• $P(E|\neg S)$ is the probability of a correct response given lack of mastery (accounting for a guess factor $g = 0.2$).

3.2 Confidence Score (Threshold Requirements)

• Confidence Limit ($C_t$): $70%$
• If $P(S|E) < C_t$, a Remediation Activity is recommended. The engine selects prerequisite content blocks.
• If $P(S|E) \ge C_t$, the next logical concept in the graph is suggested.

---

4. Cost Control Architecture (Model Routing)

To prevent API cost explosion, the platform employs a dynamic router that chooses model tiers based on task complexity.

                  [AI Request Task]
                          │
                          ▼
            ┌───────────────────────────┐
            │   Task Complexity Classifier
            └───────────────────────────┘
                          │
            ┌─────────────┼─────────────┐
            ▼             ▼             ▼
       (Simple)       (Medium)       (Complex)
      Embedding       Text Query    Gen-Quiz/Code
          │             │               │
          ▼             ▼               ▼
      [Cache /      [Flash-Lite     [Pro Model]
     Vector DB]        Model]

Routing Thresholds:

1. Cache / Vector DB (Zero Cost): Standard definitions, basic translations, and repetitive requests are resolved using cached responses.
2. Flash-Lite Model (Low Cost): Used for daily student chats, chat classification, and raw spelling checks.
3. Pro Model (High Cost): Reserved for long-term curriculum planning workflows, multi-step math/code validation, and complex content generation.

---

5. Human-in-the-Loop Content Generation Flow

The state transitions for content draft review are governed by the following state machine:

[Draft Created] ──(Auto moderated)──► [Pending Review] ──► [Editor Review]
                                                                │
                                      ┌─────────────────────────┴─────────┐
                                      ▼                                   ▼
                              (Approved/Modified)                     (Rejected)
                                      │                                   │
                                      ▼                                   ▼
                                 [Published]                         [Discarded]

• Transition Rule:
  • No content item in database tables cf_learning_activities, cf_concepts, or cf_mini_concepts can have a status of 'active' without a matching ops_content_reviews row containing status = 'approved' or status = 'modified'.
  • If a reviewer edits a draft during review, the modified copy is saved in the target table, and the log documents the changes.