START HERE
What you will learn
- Identify the definitions that influence an agent run.
- Separate versioned definitions from protected operational data.
- Create a manifest that connects a run to its inputs and evidence.
Before you start: For stakeholders, independent builders and technology leaders. Bring one workflow or software change you can describe.
“Everything in version control” needs a precise meaning
Version the definitions needed to explain intended behaviour: requirements, system instructions, tool schemas, dependency locks, evaluation cases, workflow configuration and policy rules. Otherwise a code commit can remain unchanged while a prompt or tool definition silently changes the result.
Do not commit everything the system happens to touch. Secrets, tokens, customer documents, live database contents and sensitive Terraform state belong in appropriately protected systems. Git history is durable; deleting a secret in the next commit does not remove its earlier exposure. Store safe references to approved resources instead.
A repository revision is necessary but not sufficient
Record the commit plus the model/provider identifier, runtime image digest, dependency lock digest and evaluation-set revision for each run. External services can change even when local files do not. A provider may not offer an immutable model version; record that limitation rather than claiming exact reproducibility.
Treat this record as provenance: information about where an output came from and how it was produced. Provenance is useful for investigation, but a self-written manifest is not automatically a trusted attestation. Protect the builder, the evidence store and review permissions as separate concerns.
Review behaviour changes as product changes
A prompt edit can widen tool use. A schema edit can expose a write action. A larger retry allowance can multiply external side effects. Review these changes alongside application code and rerun the relevant checks. Do not exempt a file because it is labelled configuration.
Keep execution logs and large test artifacts in access-controlled storage with retention rules. The repository can retain the run ID, artifact digest and policy decision without exposing the full record. Separate reviewable definitions, immutable evidence references and mutable operational state.
Technical examples
Definitions in Git; protected values elsewhere
An example layout. Paths do not make files trusted: review permissions and CI rules must protect the definitions and tests.
specs/report-download.yaml
agent/instructions.md
agent/tools.schema.json
policy/permissions.yaml
runtime/Dockerfile
runtime/dependency.lock
evals/cases/ownership-denied.json
tests/report-download.test.ts
ci/validation.yaml
Outside Git:
credentials -> secret store
customer records -> approved data system
run artifacts -> protected evidence storeA manifest records resolved inputs
Placeholder values must be replaced with actual resolved identifiers during a run. This is an explanatory record, not a signed attestation or an executable deployment file.
execution_id: example-run-001
source_revision: <full-commit-id>
runtime_image: <registry-image-at-sha256-digest>
model_identifier: <provider-and-model-version>
model_version_immutable: false
evaluation_revision: <evaluation-commit-id>
policy_revision: <policy-commit-id>
evidence_reference: <protected-artifact-reference>
release_decision: pending-human-reviewStep-by-step process
Inventory behaviour-changing inputs
List instructions, tools, policies, dependencies, runtime images, retrieval sources and external model identifiers. Identify the owner of each.
Classify storage boundaries
Choose what belongs in Git, a secret store, an artifact registry or a protected evidence store. Remove sensitive data before creating fixtures.
Create a run manifest
Resolve references at execution time and record the actual identities and digests used, not merely names such as latest.
Gate definition changes
Require review and relevant checks when prompts, permissions, evaluations or runtime files change. Document external inputs you cannot pin.
FOLLOW ONE EXAMPLE
Worked scenario
Investigating a regression after an apparently harmless prompt edit
A fictional agent starts making unsupported recommendations although its application code has not changed.
- The run manifest identifies a changed instruction revision and unchanged tool contract.
- The reviewer compares behaviour on the same safe evaluation cases before and after that edit.
- The team restores the prior instruction version, then proposes a revised instruction with the failing cases retained.
What this shows: Versioned definitions narrow the investigation. They do not make nondeterministic model outputs identical.
The process at a glance
Read Mermaid diagram source
flowchart TD A[Reviewed definitions in Git] --> D[Resolved run manifest] B[Protected secret references] --> D C[Runtime and model identifiers] --> D D --> E[Agent execution] E --> F[Protected evidence store] F --> G[Digest and review reference] G --> A
What you should have at the end
- A storage classification for agent inputs.
- A versioned manifest format and a review rule for behavioural configuration.
Common mistakes and how to avoid them
Committing live data for convenience
Use synthetic fixtures and protected references instead of embedding customer records or credentials.
Equating a Git SHA with reproducibility
Record external dependencies and changing provider behaviour as well as the source revision.
APPLY THE IDEA
Try it yourself
Your task: Inventory the definitions behind one research agent and classify their storage.
What to produce: A four-column register: input, owner, version/reference and permitted storage location.
Check your understanding: Can someone explain a past result without needing a secret or personal record to be copied into Git?
Key terms in plain English
- Digest
- A content-derived identifier used to check whether an artifact changed.
- Provenance
- Information connecting an output to its inputs and build process.
- Operational state
- Changing runtime information such as leases, queues and task progress.