Agent 生产部署与 AgentOps：从 Demo 到可灰度、可回滚的长期系统

1
Control：允许怎样运行
2
Data/Execution：正在怎样运行
3
Evidence：刚才究竟怎样运行、是否成功

1
POST /v1/agent-runs
2
Idempotency-Key: req-user1-20260715-001
3

4
{
5
  "thread_id": "thread_01J...",
6
  "agent": "orderflow",
7
  "input": {"message": "订单三天没发货，我要退款"},
8
  "response_mode": "stream"
9
}

1
{
2
  "run_id": "run_01J...",
3
  "status": "queued",
4
  "events_url": "/v1/agent-runs/run_01J/events",
5
  "status_url": "/v1/agent-runs/run_01J"
6
}

1
run.started
2
step.started/completed
3
model/tool progress（安全投影）
4
human.required
5
run.completed/failed/cancelled

1
POST /v1/agent-runs/{run_id}/cancel

1
created/queued/running
2
awaiting_input/confirmation/human
3
retry_scheduled
4
succeeded/failed/cancelled/budget_exhausted

1
from pydantic import BaseModel
2

3

4
class WorkItem(BaseModel):
5
    work_id: str
6
    run_id: str
7
    step_id: str
8
    attempt: int
9
    tenant_id: str
10
    priority: int
11
    risk_class: str
12
    available_at: str
13
    deadline_at: str
14
    required_worker_pool: str
15
    idempotency_key: str

1
dequeue
2
-> acquire lease
3
-> heartbeat
4
-> execute/checkpoint
5
-> ack

1
critical human-resume
2
interactive user task
3
batch/report
4
maintenance/replay

1
拒绝/延迟低优先级新任务
2
返回 Retry-After
3
降低 fan-out/并发
4
暂停 batch/replay
5
降级只读/人工

1
class AdmissionDecision(BaseModel):
2
    decision: Literal["accept", "queue", "degrade", "reject"]
3
    queue_class: str | None
4
    estimated_start_at: str | None
5
    reserved_budget: dict
6
    degradation_profile: str | None
7
    retry_after_seconds: int | None
8
    reason_codes: list[str]

1
tenant/user quota
2
queue oldest age/depth
3
worker capacity
4
provider token headroom
5
tool dependency health
6
risk/priority/deadline
7
estimated steps/tokens

1
depth
2
oldest age
3
time-to-start p50/p95/p99
4
deadline miss
5
redelivery/lease expiry
6
per-tenant fairness
7
DLQ age

1
typed state/schema version
2
current node/completed steps
3
pending external operation
4
budget usage
5
pinned release manifest
6
human/confirmation state
7
last durable event sequence

1
Run 创建
2
有副作用动作前
3
工具 Observation 后
4
状态迁移后
5
暂停/HITL 前
6
最终响应前

1
DB transaction:
2
  state transition
3
  + outbox event
4
commit
5

6
publisher sends idempotently
7
consumer inbox deduplicates event_id

1
at-least-once
2
+ idempotency
3
+ CAS
4
+ outbox/inbox
5
+ external reconciliation
6
= effectively-once business outcome

1
ModelRequest
2
ModelEvent
3
Usage
4
Error Taxonomy
5
Cancellation

1
class ModelRouteRequest(BaseModel):
2
    purpose: str
3
    required_capabilities: list[str]
4
    data_classification: str
5
    region: str
6
    latency_budget_ms: int
7
    cost_budget_usd: float
8
    context_tokens: int
9
    risk_level: str

1
结构化输出/工具/上下文能力
2
数据地域/隐私
3
模型允许用途
4
当前 quota/error/latency
5
质量层级
6
成本

1
primary reasoning model
2
-> equivalent provider/model
3
-> smaller model for safe subset
4
-> deterministic rule/read-only
5
-> human

1
closed -> open -> half-open -> closed

1
request/error by error class
2
TTFT/total latency
3
tokens/sec
4
rate-limit headroom
5
schema/tool-call validity
6
safety/refusal distribution
7
cost

1
class RouteOutcome(BaseModel):
2
    route_id: str
3
    route_version: str
4
    selected_model: str
5
    alternatives_considered: list[str]
6
    hard_constraints: list[str]
7
    selection_reasons: list[str]
8
    fallback_used: bool
9
    verified_task_success: bool | None
10
    usage: dict

1
constraint violations
2
quality delta
3
cost/latency delta
4
provider concentration
5
fallback load

1
global
2
provider/model
3
tenant
4
user
5
agent/workflow
6
tool
7
risk class

1
requests/minute
2
tokens/minute
3
concurrent runs
4
tool calls
5
retrieval queries
6
cost/day

1
deterministic function cache
2
embedding cache
3
retrieval cache
4
tool read-result cache
5
model semantic/exact cache（谨慎）
6
context package cache

1
tenant
2
identity/ACL fingerprint
3
normalized input
4
model/prompt/tool/policy/index versions
5
locale/time bucket

1
高风险写结果作为“可再次执行”
2
含动态权限且 key 不完整的数据
3
高度个性化敏感响应
4
未验证的模型输出

1
multi-stage build
2
non-root
3
minimal base
4
locked dependencies/SBOM/signature
5
read-only filesystem where possible
6
no embedded secret

1
startup：慢初始化是否完成
2
liveness：进程是否需要重启
3
readiness：当前是否能接新流量

1
readiness=false
2
停止领取新 Work
3
等待/Checkpoint 当前 Step
4
释放 lease/连接
5
flush critical audit/outbox
6
在 termination grace 内退出

1
HTTP concurrency
2
queue depth/oldest age
3
active leases
4
CPU/memory
5
provider quota headroom

1
workers ≈ arrival_rate × avg_service_time / target_utilization

1
cpu/memory
2
sandbox isolation
3
network policy
4
deadline
5
checkpoint requirement
6
GPU/model locality

1
Verified Task Success
2
Correct/Unsafe Action
3
Availability
4
Run/Queue Latency
5
Checkpoint Recovery
6
Human Escalation
7
Cost per Verified Success
8
Trace/Audit Completeness

1
window: 28d
2
slos:
3
  verified_task_success: ">= 99.0% for eligible refund queries"
4
  unsafe_action_rate: "0 for critical cases"
5
  api_availability: ">= 99.9%"
6
  interactive_p95: "<= 8s to first meaningful event"
7
  queue_p95: "<= 20s"
8
  recovery_success: ">= 99.5%"

1
冻结高风险功能扩张
2
降级/回滚新版本
3
优先可靠性工作
4
增加人工门禁

1
if:
2
  - critical_safety_incident: true
3
then:
4
  - disable_high_risk_writes
5
  - rollback_latest_release
6
  - page_security_and_business_owner
7

8
if:
9
  - success_slo_burn_rate_1h: "> 14.4"
10
then:
11
  - halt_rollout
12
  - route_to_stable_release
13

14
if:
15
  - monthly_error_budget_remaining: "< 25%"
16
then:
17
  - freeze_non_reliability_changes
18
  - require_extra_canary

1
Run/Step/Attempt
2
Model/Prompt/Route
3
Tool/MCP/A2A
4
State/Checkpoint
5
Policy/HITL
6
RAG/Memory
7
Cost/Latency/Error

1
runs_total{status,agent_version}
2
run_duration_seconds
3
queue_oldest_age_seconds
4
model_calls_total{provider,model,status}
5
tool_calls_total{tool,status}
6
checkpoint_recovery_total{status}
7
cost_usd_total{tenant,agent}
8
eval_task_success{release}

1
Executive：success/safety/cost/SLO
2
Runtime：queue/worker/checkpoint/retry
3
Model：route/latency/token/error
4
Tools：success/unknown outcome/dependency
5
Quality：eval/failure slices/regressions

1
Agent/Workflow
2
Model/Adapter/Tokenizer
3
Prompt Bundle
4
Tool/MCP/Skill Registry
5
Policy/Guardrail
6
RAG Index
7
Memory Schema/Write Policy
8
Evaluator/Dataset
9
Runtime/Container

1
release: orderflow-agent@3.2.0
2
runtime_image: sha256:...
3
workflow: refund@7
4
model_route: support-default@9
5
prompt_bundle: support@12
6
tool_registry: orderflow@42
7
policy_bundle: refund-cn@18
8
rag_index: policy-2026-07-15.3
9
memory_policy: user-profile@5
10
eval_dataset: orderflow-eval@2026-07-15.3
11
eval_report: eval://orderflow/3.2.0
12
created_at: 2026-07-15T10:00:00Z

1
Compatible：旧 Run 可由新 Worker 恢复
2
Pinned：旧 Run 必须继续使用旧 Manifest
3
Migrate：运行显式 State Migration
4
Drain：等待旧 Run 完成再下线
5
Terminate/Human：无法安全迁移

1
class StateMigrationResult(BaseModel):
2
    from_schema: str
3
    to_schema: str
4
    migrated_state_ref: str
5
    preserved_effect_refs: list[str]
6
    warnings: list[str]
7
    validation_passed: bool

1
无在途 Run/Replay 保留需求
2
回滚窗口结束
3
Audit/合规保留满足
4
Artifact/Checkpoint 依赖解除

1
lint/unit/contract
2
-> scenario eval
3
-> adversarial/regression
4
-> cost/latency
5
-> manifest/sign
6
-> shadow
7
-> canary
8
-> progressive rollout
9
-> full

1
stable user/tenant bucketing
2
低风险/只读先行
3
强 HITL
4
自动 stop/rollback thresholds

1
critical safety
2
SLO burn
3
task success regression
4
cost/latency spike
5
dependency error
6
trace/audit gap

1
Model input/output/reasoning/cache
2
Embedding/Rerank
3
Tool/API
4
Vector/Graph/DB/Object storage
5
Queue/Compute
6
Observability/Eval
7
Human Review

1
Cost per Verified Success
2
Cost per Resolved Ticket
3
Cost per Safe Automated Action
4
Marginal Cost of Reflection/Multi-Agent

1
Context selection/compaction
2
Tool candidate pruning
3
Model routing/cascade
4
Cache/batch
5
stop/no-progress
6
parallelism only when useful
7
offline/nightly eval scheduling

1
class CostEntry(BaseModel):
2
    run_id: str
3
    step_id: str | None
4
    tenant_id: str
5
    category: Literal[
6
        "model", "embedding", "rerank", "tool", "compute",
7
        "storage", "observability", "human"
8
    ]
9
    provider: str
10
    units: dict
11
    amount_usd: float
12
    pricing_version: str
13
    occurred_at: str

1
critical safety unchanged
2
verified success within budget
3
human escalation not increased materially
4
latency SLO

1
class FailureEnvelope(BaseModel):
2
    failure_id: str
3
    run_id: str
4
    trace_id: str
5
    first_failure_code: str
6
    contributing_codes: list[str]
7
    release_manifest: str
8
    checkpoint_ref: str
9
    environment_refs: list[str]
10
    data_classification: str
11
    occurred_at: str

1
terminal failure
2
unsafe/policy block
3
human override
4
high cost/latency
5
fallback
6
unknown outcome
7
user correction

1
Dry Replay：不调用外部写工具
2
Fixture Replay：在模拟环境完整执行
3
Shadow Replay：新版本对历史输入运行
4
Time-travel：从 checkpoint/旧 state fork

1
api-pool
2
interactive-worker-pool
3
batch-worker-pool
4
high-risk-action-pool
5
eval/replay-pool

1
Gateway Auth
2
-> Create Run/Queue
3
-> Worker loads pinned Manifest/Checkpoint
4
-> read tools + RuleRAG
5
-> Policy/HITL
6
-> high-risk action pool executes idempotently
7
-> verify final state
8
-> event/trace/audit/eval hooks

1
退款查询 verified success
2
确认前错误写 = 0
3
重复退款 = 0
4
first meaningful event p95
5
queue age p95
6
final-state verification coverage
7
cost per resolved request

1
concurrency ≈ throughput × average time in system

1
arrival rate by scenario/tenant
2
steps/model/tool calls per run
3
service time distribution
4
token distribution
5
human wait exclusion/inclusion
6
burst factor

1
Run/Checkpoint
2
Audit
3
Business Idempotency
4
Memory/RAG Canonical
5
Index Projection
6
Artifact

1
数据驻留
2
身份/Policy 一致
3
Run affinity
4
跨区状态复制
5
重复执行风险
6
模型/工具区域能力

1
1. 用户/业务影响与严重级别
2
2. 哪个 Release/tenant/scenario
3
3. SLO/queue/provider/tool/DB 状态
4
4. first divergent Span/State
5
5. 是否有安全/副作用
6
6. 遏制/回滚/降级

1
回滚/修复
2
对账外部状态
3
补偿/通知
4
最小回归
5
shadow/canary 恢复

1
Daily：SLO/critical alerts/DLQ/unknown outcomes
2
Weekly：failure clusters/cost/fallback/human queue
3
Monthly：model/tool/policy/index drift + dependency updates
4
Quarterly：DR/security/rollback drill + retention review