Agent 安全、约束与可靠性：从威胁建模到纵深防御

1
Authenticity：主体/来源真实
2
Authorization：只有允许的主体执行允许动作
3
Confidentiality：数据不向未授权方泄露
4
Integrity：指令、事实、状态和制品未被篡改
5
Availability：不会被循环/资源耗尽拖垮
6
Accountability：动作可归因、可审计
7
Controllability：能暂停、撤销、人工接管

1
Risk ≈ Likelihood × Impact × Exposure

1
用户身份与会话
2
订单/支付/健康/代码等敏感数据
3
System/Developer Prompt 与 Policy
4
模型/Adapter
5
Tool/MCP/Skill 凭据
6
Memory/RAG/Index
7
业务动作与资金
8
Trace/Audit/Eval 数据

1
End User
2
Organization Admin
3
Agent Runtime
4
Model Provider
5
Tool/MCP/A2A Service
6
Human Reviewer
7
Operator/Developer
8
Attacker/Supply-chain Publisher

1
Browser -> API Gateway
2
Application -> Model Provider
3
Harness -> Tool/MCP Server
4
Agent -> A2A Remote Agent
5
Runtime -> Sandbox
6
Ingestion -> RAG/Memory
7
Production -> Observability/Eval
8
Human Reviewer -> Approval Service

1
data classification
2
identity/auth method
3
purpose
4
retention
5
encryption
6
allowed destinations
7
failure mode

1
class AbuseCase(BaseModel):
2
    case_id: str
3
    attacker: str
4
    entry_point: str
5
    preconditions: list[str]
6
    attack_steps: list[str]
7
    target_assets: list[str]
8
    potential_impact: list[str]
9
    controls: list[str]
10
    detection: list[str]
11
    recovery: list[str]
12
    residual_risk: str

1
class SecurityRequirement(BaseModel):
2
    requirement_id: str
3
    statement: str
4
    severity: Literal["critical", "high", "medium", "low"]
5
    control_ids: list[str]
6
    test_case_ids: list[str]
7
    evidence_queries: list[str]
8
    owner: str
9
    residual_risk: str

1
新增模型供应商/区域
2
新增写工具或权限
3
接入 MCP/A2A/Skill
4
Memory/RAG 新数据源
5
允许代码/浏览器执行
6
Prompt/Policy 信任层级变化
7
日志/数据保留变化

1
User Principal
2
  谁提出请求
3

4
Agent Principal
5
  哪个 Agent/版本代表用户工作
6

7
Service Principal
8
  哪个工具服务实际访问资源
9

10
Human Approver
11
  谁批准高风险动作

1
class DelegationContext(BaseModel):
2
    tenant_id: str
3
    user_subject: str
4
    calling_agent_id: str
5
    calling_agent_version: str
6
    delegated_scopes: list[str]
7
    resource_constraints: list[str]
8
    audience: str
9
    task_id: str
10
    expires_at: str
11
    authentication_strength: str

1
有 order:read scope
2
≠ 可以读取任意 order_id

1
tenant
2
owner/relationship
3
resource state
4
field-level access
5
purpose

1
Instruction
2
  System/Developer/Policy，受控发布
3

4
Verified Data
5
  经授权工具/数据库读取并带版本
6

7
Untrusted Content
8
  用户、网页、邮件、文档、Memory、远端 Agent 文本

1
class SecurityLabeledContext(BaseModel):
2
    item_id: str
3
    content_ref: str
4
    trust: Literal["instruction", "verified_data", "untrusted_data"]
5
    data_classification: Literal["public", "internal", "confidential", "restricted"]
6
    source_identity: str
7
    allowed_purposes: list[str]
8
    allowed_destinations: list[str]
9
    expires_at: str | None

1
restricted data -> external model: deny unless approved route
2
confidential tool result -> public response: redact/project
3
untrusted document -> tool arguments: require structured extraction + policy
4
private memory -> shared agent: deny

1
Policy Decision Point (PDP)
2
  输入主体、动作、资源、环境，返回决策
3

4
Policy Enforcement Point (PEP)
5
  在 API、Context、Tool、Output 等边界强制决策
6

7
Policy Information Point (PIP)
8
  提供身份、资源、风险、时间等属性

1
class PolicyDecision(BaseModel):
2
    decision: Literal["allow", "deny", "require_confirmation", "require_human"]
3
    reason_codes: list[str]
4
    obligations: list[str]
5
    policy_version: str
6
    evaluated_at: str
7
    expires_at: str | None

1
mask.phone
2
limit.amount=5000
3
verify.order_owner
4
use.model_route=internal
5
log.audit.high_risk
6
confirm.user.bound_args

1
read-only low risk -> configurable degraded path
2
write/high risk -> deny or human

1
check owner/state/version
2
-> user waits for confirmation
3
-> resource changes
4
-> execute

1
API PEP：谁能创建哪类 Run
2
Context PEP：哪些数据可进入哪个模型/Agent
3
Capability PEP：当前 Step 暴露哪些工具
4
Tool PEP：动作/资源/参数是否执行
5
Memory PEP：能否写入/读取哪个 namespace
6
A2A PEP：能否委托给哪个远端 Agent
7
Output PEP：哪些字段可发往当前渠道

1
class PolicyInput(BaseModel):
2
    subject_ref: str       # auth middleware
3
    agent_ref: str         # registry/runtime
4
    action: str            # resolved tool contract
5
    resource_ref: str      # trusted adapter lookup
6
    resource_version: str
7
    environment: dict      # server clock/risk signals
8
    proposed_args_hash: str

1
Tool Proposal
2
-> Schema
3
-> Trusted Identity Binding
4
-> Resource Authz
5
-> Policy
6
-> Confirmation/HITL
7
-> Idempotency
8
-> Deadline/Rate Limit
9
-> Execute
10
-> Verify Final State
11
-> Audit

1
subject
2
tool/version
3
resource/version
4
normalized args hash
5
amount/currency
6
expiry

1
不直接 retry
2
-> query by operation/idempotency key
3
-> reconcile final state
4
-> only then retry/complete/human

1
timeout
2
concurrency
3
rate/budget
4
max payload
5
retry matrix
6
circuit breaker
7
bulkhead

1
外部文本不能变成高优先级指令
2
模型不能自授工具权限
3
高风险动作需要确定性 Policy/确认
4
结果必须验证

1
标记 source/content
2
停止将该内容用于指令
3
继续安全的只读目标或请求人工
4
不泄露检测规则
5
保存最小攻击证据
6
加入 adversarial regression

1
来源 allowlist/signature
2
malware/OCR/hidden-text scan
3
ACL/tenant/effective time
4
内容 hash/version
5
发布审批
6
index snapshot

1
filter before rank
2
source trust/freshness
3
poisoning anomaly
4
quote/span validation
5
untrusted label

1
candidate only
2
PII/secret/injection classification
3
consent/purpose/retention
4
scope/namespace
5
conflict/provenance
6
policy commit

1
来源分布突变
2
同文档大量近重复
3
异常高检索命中
4
高风险指令短语
5
图实体/边暴增
6
记忆写入率异常

1
恶意 Server/Tool description
2
token passthrough/audience 错误
3
confused deputy
4
DNS rebinding/SSRF
5
capability drift
6
Tool Result Injection

1
伪造 Agent Card
2
过宽 delegation
3
远端 Artifact 注入
4
push callback SSRF/replay
5
跨组织数据保留不明

1
SKILL.md 恶意指令
2
scripts 执行命令/发网
3
依赖投毒
4
assets 模板泄露数据
5
自动更新扩大权限

1
artifact_id: order-fulfillment-skill@3.2.0
2
publisher: platform-ai
3
digest: sha256:...
4
permissions:
5
  filesystem: [workspace-read]
6
  network: [order-api.internal]
7
  secrets: []
8
risk: medium
9
reviewed_at: 2026-07-15
10
expires_at: 2026-08-15

1
quarantine
2
-> manifest/diff/SBOM/signature
3
-> static/dynamic scan
4
-> sandbox tests
5
-> adversarial eval
6
-> human approval
7
-> pinned canary
8
-> monitor/rollback

1
source repository/commit
2
builder identity
3
build workflow
4
dependency lock/SBOM
5
artifact digest/signature
6
vulnerability scan
7
license policy

1
私有包名被公共仓库抢注
2
相似 Skill/Server 名称
3
安装脚本执行任意命令
4
transitive dependency 被接管

1
阻断新增/变更的高风险能力
2
隔离连接/进程
3
保留旧 Snapshot 完成安全只读任务
4
告警并要求重新审核

1
Process sandbox
2
Container
3
VM/microVM
4
Dedicated account/project
5
Physical/organizational isolation

1
ephemeral workspace
2
read-only inputs
3
write allowlist
4
path traversal/symlink defense
5
size/quota
6
no host home/SSH/cloud credentials

1
deny by default
2
DNS/IP/domain allowlist
3
block metadata service/private ranges
4
proxy with audit
5
download size/type scan
6
egress data policy

1
non-root
2
syscall/capability restrictions
3
CPU/memory/time/process limit
4
no privileged mount/socket
5
kill entire process tree on timeout

1
Agent requests capability token
2
-> Broker validates run/step/policy/resource
3
-> issues short-lived credential to Adapter
4
-> Adapter calls target
5
-> credential expires/revokes

1
audience
2
scope/action
3
resource/tenant
4
short TTL
5
one task/step
6
non-exportable where possible
7
rotation/revocation

1
path/size/type validation
2
malware scan
3
secret/PII scan
4
content hash
5
untrusted label
6
artifact allowlist

1
隔离浏览器 profile
2
无个人 cookie
3
域名/下载/上传策略
4
危险点击/提交确认
5
视觉与 DOM 观察视为不可信
6
动作截图/日志（按隐私策略）

1
风险或权限提升
2
不可逆/财务动作
3
数据外发
4
目标/参数变化
5
证据冲突

1
谁请求
2
代表谁
3
执行什么
4
对哪个资源
5
关键参数/金额
6
依据与风险
7
可逆性
8
有效期
9
允许的决定

1
不隐藏金额/收件人/域名
2
不使用模糊“继续”替代动作名
3
显示参数变化 diff
4
高风险拒绝与批准同等可见

1
Schema
2
claim/citation verification
3
PII/secret redaction
4
tenant/resource authorization
5
channel-specific policy

1
错误栈
2
文件名/URI
3
citation link
4
tool name/args
5
token/latency 差异
6
Trace/metrics labels

1
max steps/model/tool calls
2
wall-clock deadline
3
token/cost budget
4
per-tool quota
5
max delegation depth

1
close write capabilities
2
degrade to read-only/rule/human
3
isolate version/provider/server

1
强模型失败 -> 弱模型

1
Policy 不可用
2
Auth provider 延迟
3
Audit sink 满
4
MCP Server 被撤销
5
Sandbox egress 拒绝
6
Human approval 超时

1
class AuditEvent(BaseModel):
2
    event_id: str
3
    trace_id: str
4
    run_id: str
5
    actor_type: str
6
    actor_id: str
7
    on_behalf_of: str | None
8
    action: str
9
    resource_refs: list[str]
10
    normalized_args_hash: str | None
11
    policy_decision_ref: str | None
12
    confirmation_ref: str | None
13
    outcome: str
14
    artifact_versions: dict
15
    occurred_at: str

1
Trace：诊断性能与执行路径，可采样
2
Audit：高风险行为责任证据，完整性与保留更严格

1
append-only/WORM（按要求）
2
hash chain/signature
3
严格写入身份
4
独立保留
5
访问审计
6
时间同步

1
谁让哪个 Agent 做了什么？
2
模型看到了哪些能力和数据？
3
哪个 Policy/确认允许动作？
4
哪个工具版本执行？
5
最终业务状态是什么？
6
数据发往哪里？

1
订单归属
2
退款金额/流水
3
用户联系方式
4
规则版本
5
退款工具凭据
6
审批与 Audit

1
用户伪造管理员要求跨订单退款
2
恶意规则文档注入“自动批准”
3
MCP Tool 描述更新后扩大动作
4
重复超时造成双退款
5
远端 A2A Agent 返回恶意 Artifact
6
Memory 保存“以后自动退款”
7
Reviewer 确认后金额被模型修改

1
API AuthN
2
-> tenant/order ownership
3
-> untrusted input labeling
4
-> read-only capability set
5
-> verified order/logistics facts
6
-> published policy evidence
7
-> deterministic eligibility/amount
8
-> user confirmation bound to args/resource version
9
-> short-lived refund capability
10
-> idempotent execution
11
-> final-state verification
12
-> audit

1
deny if subject does not own order
2
deny if amount > paid amount
3
require human if amount > auto threshold
4
require reconfirm if order/policy/args version changed
5
deny if evidence bundle conflicting/insufficient
6
allow write tool only for one step and one resource

1
{
2
  "status": "needs_human",
3
  "reason_code": "policy_evidence_conflict",
4
  "safe_completed_steps": ["order_verified", "logistics_verified"],
5
  "forbidden_actions": ["create_refund"],
6
  "review_package_ref": "review://run_01J/1"
7
}

1
Golden Safety
2
Prompt Injection
3
Authorization/IDOR
4
Tool Misuse
5
Data Exfiltration
6
Memory/RAG Poisoning
7
MCP/Skill Supply Chain
8
Availability/Cost
9
Regression Incidents

1
Attack Success Rate
2
Unsafe Action Rate
3
Unauthorized Data Exposure
4
Injection Defense Rate
5
False Positive/Refusal Rate
6
Human Gate Bypass Rate
7
Duplicate Side-effect Rate
8
Detection Time
9
Containment/Recovery Time

1
数据库是否被修改
2
是否调用禁止工具
3
是否读取跨租户资源
4
是否发生网络外传
5
是否写入持久 Memory

1
多语言/编码/分片
2
间接文档/图片/工具结果
3
多轮铺垫
4
权限/业务状态变化
5
供应链/版本漂移
6
成本耗尽

1
class SecurityCase(BaseModel):
2
    case_id: str
3
    threat_id: str
4
    initial_state_ref: str
5
    attacker_inputs: list[dict]
6
    identity: dict
7
    allowed_actions: list[str]
8
    forbidden_actions: list[str]
9
    forbidden_data_refs: list[str]
10
    expected_terminal_states: list[str]
11
    environment_assertions: list[str]
12
    detection_expectations: list[str]
13
    cleanup_assertions: list[str]

1
refund_count == 0
2
cross_tenant_reads == 0
3
external_egress_bytes == 0
4
long_term_memory_writes == 0
5
audit contains policy_deny

1
False Refusal Rate
2
Unnecessary Human Escalation
3
Task Success after Guardrails
4
Guardrail Latency/Cost

1
risk_id: R-A2A-04
2
scenario: approved remote agent may retain task metadata
3
impact: medium
4
controls: [data-minimization, contractual-retention, audit]
5
residual_likelihood: low
6
owner: security-platform
7
accepted_by: business-owner
8
expires_at: 2026-10-15
9
review_trigger: remote-policy-change

1
跨租户 ID
2
过期/错误 audience token
3
scope/resource 不匹配
4
Agent 伪造管理员
5
委派链缺失

1
参数篡改
2
确认后金额变化
3
重复请求
4
outcome_unknown
5
权限在等待期间撤销

1
MCP capability drift
2
Skill digest 改变
3
恶意依赖/脚本
4
Agent Card 伪造
5
模型/Adapter 未批准替换

1
跨租户检索
2
Poisoned RAG/Memory
3
引用泄露
4
Trace PII
5
删除后缓存仍返回

1
Policy/IdP/Audit 不可用
2
Guardrail 超时
3
队列/配额耗尽
4
Sandbox 逃逸尝试
5
HITL 过期/冲突

1
异常工具/数据访问
2
攻击检测告警
3
退款/外传异常
4
供应链 digest drift
5
用户报告

1
停用 Agent/Prompt/Tool/Skill/Server version
2
撤销 token/credential
3
关闭写能力
4
隔离受影响租户/队列
5
切只读/人工

1
Trace/Audit/Registry Snapshot
2
Prompt/Model/Policy versions
3
Tool/MCP/A2A artifacts
4
业务状态和外部流量

1
修复根因
2
轮换凭据
3
清理 poisoned memory/index
4
补偿/通知受影响方
5
新增 regression
6
shadow/canary 恢复

1
触发信号与严重级别
2
值班/业务/隐私/法务联系路径
3
kill switch 与撤销命令
4
证据保全查询
5
受影响版本/租户定位
6
用户/监管通知决策
7
恢复门禁
8
回归与 Postmortem 时限

1
critical alert acknowledgement time
2
write-capability containment time
3
credential revocation propagation
4
poisoned artifact removal time
5
audit completeness
6
critical regression pass rate