Informal share
Models, harnesses
Theme 1
Choosing the right model Theme 2
Agent = LLM + harness
Zheng Zhang
These are my own observations from using AI, mostly large language models. I am not going to teach you anything. This is personal, and it may be unprofessional. The two themes are choosing the right model, and working with the harness around it, with a notebook that carries the work between harnesses.
Intro
A learning method in daily life
LLMs are changing everyday life.
A machine-learning method p(next token | context)
A change in daily work writing · coding · reading · deciding
Why?
Academically, an LLM is one machine-learning method among many. Yet I see its effects reaching into writing, coding, reading and everyday decisions on a scale I have not experienced with another method. [Advance.] Why can a model trained to continue sequences have that reach?
Intro
The next-token model
An LLM predicts a continuation.
Cross-entropy ↔ compression
Training → post-training → generation
T = sampling temperature · same scores zi in both panels
The basic object is a distribution for the next token given the context. A token can be a word, part of a word, or a symbol. Generation repeatedly samples a token and adds it to the sequence. [Advance.] The sampling distribution has exactly the Boltzmann form, with T a sampling parameter, not a thermodynamic temperature. The plot is the same set of scores at two values of T: at T of one half almost all the probability sits on the top token, at one and a half the alternatives get a real chance. Low T gives repeatable answers, high T more varied ones. [Advance.] Minimising next-token cross-entropy is minimising expected code length under arithmetic coding, so prediction and compression are the same objective. [Advance.] Training learns the parameters; post-training shapes how the model responds to instructions and rewards. A reasoning model can spend additional tokens before giving its answer. Nothing in this loop looks anything up or runs a calculation. A surrounding program can provide a retrieved page, run code, and return the output. That program is Part 2.
Intro
The capability debate
There may be a ceiling; LLMs will still change daily life.
Open question
Is text enough?
Other bets: world models, spatial intelligence
Even so
One of the most important moments
Three reasons
Some researchers think this loop has a ceiling. Text alone, they argue, leaves out a model of the physical world and spatial understanding, so they are building world models instead. They may be right about the ceiling. My view is that LLMs will still change almost everyone's daily life, and that this is one of the most important moments in our use of computers. I have three reasons: how much language can represent, how far recombination can take us, and what happens when the model can act through tools.
Intro · reason 1
Language connects worlds
Words connect ideas, mathematics, code and observations.
“Try the other limit.”
language
A new discovery
calculate · test · check
What starts as words can end as something new.
My first reason is that language connects worlds. It can represent objects, ideas, mathematical relationships, procedures and observations, so it lets us connect one kind of knowledge to another. Students may know this well: a supervisor can move their lips and somehow guide you towards a discovery nobody has made before. Perhaps all they say is, “Try the other limit.” [Advance.] The student still has to calculate, test and check. But the starting point was a few words, and the result can be new knowledge. That is the possibility I see in a language interface: its effects can reach far beyond conversation.
Intro · reason 2
Recombination
Recombining what is already known goes further than expected.
Literature search
Erdős problems existing solutions found in papers
Rediscovery
Kerr symmetries Lupsasca: a known result recovered
New calculation
Cosmic-string loops Scherrer: the odd-n case, checked against his even-n result
Reason 3: a model can also act on what it describes → Part 2
My second reason is that recombining what is already known goes further than expected. Start with literature search. In the Erdős episode, the useful output was finding papers that already contained solutions, connecting knowledge that had not been connected in the problem list. [Advance.] The next step is rediscovery. Alex Lupsasca asked for symmetry generators of a wave equation on Kerr. After an initial failure and a flat-space warm-up, the model recovered a known result. [Advance.] Then a small new calculation. Robert Scherrer used the model on an integral in the gravitational-wave spectrum of cosmic-string loops. It produced an analytical treatment of the odd-n case, which he could not do with his own methods. At large n it matched his earlier even-n result. Below n of about five, its asymptotic form departs from the numerics. A human guided and checked each case. [Advance.] My third reason is that a model can also act on what it describes. I come back to that in the second part. First, how I choose a model.
1 · Choosing the right model
Three kinds of cook
Speed, breadth and task quality: different models, different trade-offs.
speed
generality
task quality
Task-specific model · the trained apprentice
Fast on a familiar menu. A new task may need retraining.
LLM · the head chef
A wide menu. Quality, speed and cost vary by task.
Jev · the cook next door
New dishes without lessons, and fast.
Three kinds of model, pictured as three cooks. I borrow this analogy from a short Jev explainer. The shapes are qualitative, not a benchmark or a universal ranking. A task-specific model is the trained apprentice: fast on a familiar menu, but a new task may need retraining. We all know examples, so I will not dwell on them. [Advance.] A language model is the head chef with a wide menu. Its quality, speed and cost depend on the task and the model. [Advance.] Jev is the cook next door, designed for fast, bounded judgments. It does not write a report. The practical question is which kind of model meets the task's quality requirement at an acceptable cost.
1 · Choosing the right model
One pass or many
A language model writes its answer; Jev picks one.
17 × 24 = 408
step by step · deliberate
2 + 2 = 4
at once · immediate
LLM: generates token by token
"What colour is the sky?"
Jev: one pass, every option
"What colour is the sky?" · blue / red / yellow
blue
0.92
red
0.05
yellow
0.03
Fast / slow thinking is an analogy; sequential generation is not itself reasoning.
Jev, vendor-reported: 70–500 ms per call input $0.042 / 1M tokens · output free probabilities illustrative
Think of two familiar experiences: working out seventeen times twenty-four, and immediately knowing two plus two. This is an analogy for fast and slow responses, not a definition of reasoning. [Advance.] An autoregressive language model generates text token by token, even for an easy question. That sequential generation does not itself mean the model is reasoning. Asked what colour the sky is, it writes "The", then "sky", then "is", then "blue": a generation step for each token. [Advance.] Jev works the other way. It gets the question and the options, and in one pass returns a probability for each option. It never writes a sentence. TypeSafe reports seventy to five hundred milliseconds per call, It charges only for input: about four cents per million tokens. Output is free. TypeSafe calls this class a System One model.
1 · Choosing the right model
Jev demo: triaging bug reports
A typed answer, with a confidence that code can branch on.
state
one bug report, free text
question · Score
severity: 0 cosmetic → 4 critical
# code decides, not the model
if confidence < 0.5: a person
elif score < 1.5: backlog
elif score < 2.5: this sprint
else: on-call
Report score conf. outcome
Typo on the settings page 0.00 1.00 backlog PDF export fails over 50 pages; splitting works 2.00 1.00 this sprint Checkout returns a 500 error for every customer 3.36 0.70 on-call Notes gone after the update, or maybe deleted 3.19 0.34 a person
"Notes gone after the update, or maybe deleted"
0 cosmetic
0.01
1 minor
0.14
2 workaround
0.07
3 blocked
0.19
4 critical
0.59
score 3.19
alone, would wake on-call
confidence 0.34
split answer → a person reads it
Here is what using it looks like. These are answers recorded in TypeSafe's documentation; I did not run them live. The state is one bug report. The question is a Score called severity, five levels from cosmetic to critical. The routing is a few lines of ordinary code from the same page, not the model: if the confidence is below one half, a person reads the report; otherwise the score picks the backlog, this sprint, or the on-call engineer. The first three reports go straight through. [Advance.] The fourth is the interesting one: notes gone after the update, or maybe deleted. The score is 3.19, which on its own would wake the on-call engineer. But the probability is split: 0.59 on critical and 0.19 on blocked, in case the update lost the notes, and 0.14 on minor, in case the user deleted them. So the confidence is 0.34, and the report goes to a person. In research the same pattern fits many small questions: is this abstract relevant to the project, did this run end in an error, does this handoff claim tests that were never run.
1 · Choosing the right model
Many small judgments
Questions change daily; cheap, fast answers make small checks worth automating.
LLM · few, large jobs
a report · some code · an analysis
Jev · many small checks
every email · comment · issue · abstract
Classify 1 million abstracts, ~300 tokens each estimate · input only · accuracy not compared
Output price per 1M tokens, one provider: Haiku 5.5 $0.50 → Fable 5.2 $40 (80×)
Back to the cooks. A trained apprentice needs a fixed menu, but an agent's questions change every day: today, which emails need a reply; tomorrow, which issues are urgent. A model trained for one fixed label set may not suit the next question. This is where a model that accepts new questions is useful. A fast, cheap judgment model makes the many small checks worth automating: every email, comment, issue and abstract. A language model keeps the few large jobs: a report, some code, an analysis. [Advance.] At that scale, price matters. A million abstracts of about three hundred tokens is three hundred million input tokens: about thirteen dollars with Jev at TypeSafe's own price, which TypeSafe says may be subsidised; thirty with Haiku, twelve hundred with Opus. Input only, and accuracy is not compared. Within one provider, output prices span a factor of eighty, from Haiku to Fable. The routing rule is to use the least expensive model that reliably does the bounded task. Next, the environment around that model.
2 · Agent = LLM + harness
Painting and acting
Harness changes alter the same model's behaviour.
A full description still needs a brush.
Critic: same model, new brief, fewer permissions
Assistant: weekly market review, memory, report files
Agent = LLM + harness
after LangChain, "Agent = Model + Harness" (Mar 2026)
My third reason was that a model can act. To act, it needs a harness around it. Suppose I can describe a painting fully in words. I still need a brush to make it. That is the tool layer. [Advance.] Now suppose I want an art critic. I can use the same model with a different brief, a review procedure, and fewer permissions: a persona, a subagent or a skill for another role. [Advance.] Finally, suppose I want an assistant to review auction markets every week. It needs a schedule, memory and somewhere to leave its report. My Codex setup has five such scheduled automations, four of them active. Change the harness and the same model behaves like a different agent. My shorthand is Agent equals LLM plus harness; LangChain writes it as model plus harness. The notebook, where the assistant leaves its report, comes back later.
2 · Agent = LLM + harness
The tool-call loop
Tool results feed the next step.
Context
Model
Permission check
Tool
Observation
observation goes back into the context
stop · ask the person
continue
The harness assembles context and calls the model. A requested tool passes a permission check, runs, and returns an observation, which goes into the context for the next model call. The model then continues, stops, or asks the person for input. That small loop sits inside the larger choices about tools, instructions, state and recovery.
2 · Agent = LLM + harness
Three harnesses
Three working environments, three strengths.
Claude Code subagents
session messaging
Codex long runs
scheduled jobs
DSH everything is a plugin
access
subscription · API key · self-hosted endpoint
In practice I run that loop in three harnesses. Claude Code gives me interactive coding and writing, subagents, and sessions that can message each other; I use those to split work and gather reviews. [Advance.] Codex is where I run long jobs and scheduled automations. [Advance.] DSH, DeepSeek Harness, is the one I extend with my own domain tools: everything in it is a plugin, including parts of the agent loop and the interface. [Advance.] Access can come through a bundled subscription, an API key, or a self-hosted endpoint, depending on the tool and its configuration. Next I go inside my domain setup on DSH. After that, I show how my own skill carries a task record between these environments.
2 · Agent = LLM + harness
rheplicant-agent: the console
Projects, task documents and every execution.
This is rheplicant-agent, the digital twin agent we built ourselves on DSH, around rheplicant, our radio-telescope digital twin. The console is our own interface: a Workbench with projects, task documents and every execution they produced, next to the chat.
2 · Agent = LLM + harness
A task document sets up the physics
The signal path, drawn from the task file.
A task is a document that sets up the physics. This one, rfi-flagging, is a demo: sky components, the ionosphere, atmosphere and ground fields, radio-frequency interference, the beam, drawn from the file as a signal path.
2 · Agent = LLM + harness
Results from the run
Waterfall before and after flagging, read from saved output.
Results are read from what a run wrote to disk, not from the chat: here the waterfall before and after flagging. The console also marks which physics in this demo is placeholder, so these numbers are not predictions.
2 · Agent = LLM + harness
Trajectory: one recorded tool run
Prompt → validate → run → results · 9 seconds.
Here is the Trajectory view of a separate recorded run: I asked the agent to validate and run the forward-signal demo. The record shows the prompt, context, validate call, run call and their results. The inspector on the right shows what the validate tool received and returned. The bars above show time spent on the model and tools; the whole turn took nine seconds. This connects the conversation to an inspectable sequence of scientific tool calls.
2 · Agent = LLM + harness
Shared instructions and skills
The aw skill, task-handoff, travels with the task.
aw: task-handoff skill
one copy per project · .agents/skills
AGENTS.md
project rules · CLAUDE.md imports it
Claude Code via symlink in .claude/skills
Codex reads .agents/skills
DSH reads .agents/skills
I write my own skills to package reusable ways of working. The one I use most is aw, my workflow tool: a small command-line program plus one skill, task-handoff. Each project gets one copy of that skill. Claude Code finds it through a symlink; Codex and DSH read the folder directly. The shared project rules live in AGENTS.md, and a one-line CLAUDE.md imports them. The skill describes how to read and update the task record. The task record and the instructions around it form my notebook across the different environments.
2 · Agent = LLM + harness
Provider independence
One notebook connects providers.
I use aw, my own task-handoff skill and command-line tool, as a shared notebook. Project rules live in AGENTS.md; the skill tells each agent how to read and update the task record. My Claude subscription gives me Claude Code. My free researcher plan gives me Codex. Self-hosted Gemma gives me another model through DSH, which can also route to hosted models. These sources work on the same project through a shared notebook: AGENTS.md, skills and the task record. The task record lets the next environment pick up the work.
2 · Agent = LLM + harness
This talk's task record
Claude and Codex took turns preparing this talk.
$ aw tasks | cut -f1-3
T-001 ready r12
[Optional live step: in a prepared terminal window (prompt set with PS1='$ ', screen cleared, already in the talk folder), run aw tasks | cut -f1-3. It is local and instant. Then back to the slide.] This is the task record for this talk. [Advance.] The strip shows the preparation itself. Claude did the research, drafted the first plan and wrote the brief. Codex wrote the second plan and the storyboard. Claude reviewed that draft and returned it with findings; Codex revised it; Claude accepted the revision and built these slides. Codex then shortened the talk and revised the language example. Each box is one recorded handoff, with who recorded it and when.
2 · Agent = LLM + harness
Recorded handoffs
1,283 handoffs in 91 tasks, across three agents.
24 of 91
tasks used two or more agents
Between the twelfth of September and the twelfth of November, the records hold one thousand two hundred and eighty-three handoffs in ninety-one tasks. Codex wrote eight hundred and seventy-six, Claude three hundred and eighty-four, DSH twenty-three. Twenty-four tasks used two or more agents, and there were fifty-nine switches between agents. By my hand count, an incoming agent refused to accept the work at least fifteen times. The labels name the harness, not the model behind it; DSH can route to hosted models as well as to our server. What a different agent does at a switch is the next example.
2 · Agent = LLM + harness
Completion and acceptance
Done is a claim.
sft-wick T-001 · a public package · one block per revision
sft-wick is one of my public packages. In this task, Claude recorded the open items after a release. DSH worked through them and marked the task done at twenty thirty-one UTC. Nine minutes later Codex checked the work and found two regressions. Codex fixed what it found, and release 0.6.1 followed two days later. That is the role I want from another agent: it challenges a completion claim by checking the actual work. The test results in this trail are the agents' own reports, not tests I reran for this talk.
Closing
Summary
Task-sized models, one shared notebook.
Theme 1
Choosing the right model
task-sized models
Theme 2
Agent = LLM + harness
task record kept outside any one provider
What I do is choose a model for the task, because not every job needs an expensive language model. And I keep the task record independently of any single provider, so different models can work on the same project. The model supplies the knowledge; the harness supplies the brush, the critic's brief and the assistant's calendar; the notebook keeps the work for whoever picks it up next. This talk's own preparation went from Claude to Codex, back for review, and on to revision. Those are my two themes in practice. Thank you.
Backup ↩
System 1 terminology
Terms depend on context.
TypeSafe
"System One": typed decisions (Jev)
LeCun
autoregressive language modelling counts as System 1
Labs
non-thinking vs thinking mode
In this talk, System One refers to TypeSafe's decision-model category. LeCun uses System 1 for autoregressive models in general. Thinking and non-thinking modes are a third distinction. The labels need defining in context; they are not technical specifications.
Backup ↩
Shared interfaces
These interfaces do different jobs.
AGENTS.md project instructions
Agent Skills task procedures (SKILL.md)
Harness support varies.
AGENTS.md carries project instructions. Skills package procedures and related resources. MCP connects tools and services. A harness has to support the interface, and configuration affects what it discovers; sharing a file format does not make behaviour identical.
Backup ↩
Building a harness
A harness needs execution and recovery.
Model side loop · context · model adapter
Execution side tools · permissions · sandbox
State side session log · failures ·
A minimal loop explains the idea, but a working harness also needs model adaptation, tool registration, permission checks, persistent state and failure handling. The Claude Agent SDK, the OpenAI Agents SDK and DSH plugins put these responsibilities in different components. The rheplicant example adds domain tools around a scientific service; it does not replace checking that service.
Backup ↩
Access and payment paths
Access and task records are separate.
Subscription login check each vendor's terms
API credentials pay per token
Self-hosted endpoint OpenAI-compatible server
Check support and terms for each tool.
The access path is separate from the task record. A product login, an API credential and a self-hosted endpoint have different requirements. I use each tool's supported configuration and check its terms. The task record provides the continuity across the tools I use.
Backup ↩
A researcher-use survey
Survey results describe their respondents.
10,480
respondents · Springer Nature · Jul 2026
≈ 50%
frequent use for information gathering and editing
Self-reported use
Nearly half of the 10,480 respondents reported frequent use for information gathering and editing. This describes the respondents and what they report, not scientific quality or all researchers.
Backup ↩
Self-hosting and memory
Context is shared across slots.
Model
Gemma 4 31B · QAT Q4_0
Runtime
llama.cpp · one A100 40 GB
Capacity
8 slots · 224K context, one shared pool
Rule of thumb
weights ≈ parameters × bits ÷ 8, plus cache and runtime
The group service exposes an OpenAI-compatible API. Its 224K context is one pool shared by the eight slots, not 224K per user. For a laptop, weight memory is parameters times bits per weight divided by eight; the cache and runtime need room too, so this is a first estimate only.
Backup ↩
aw limits and adoption
A task record reports work.
Adoption (snapshot) 76 registered projects · 21 with tasks · 55 without tasks
Native-session bindings used: 0
Limits no automatic verification · local, not encrypted or redacted · revision checks do not lock code
Native-session binding was unused and many registered projects had no task, so registration overstates adoption. The state is local and not encrypted, redacted or synchronised. Revision checks protect the task metadata, not source edits. aw records test reports without running the tests. I also read cited sources against the claims attached to them: a citation can exist and still not support the sentence.
Backup ↩
Weekly handoffs
Activity came in bursts.
Weeks start on Monday, UTC; the last week is partial. All DSH use was on one day, 12 September.
Backup ↩
Harness references
Several sources shaped this framing.
Weng, 2023
planning, memory, tools around an LLM
Anthropic, 2025
harnesses for long-running work
Hashimoto, 2026
"harness engineering"
LangChain, 2026
Agent = Model + Harness
These references credit the framing. None of them invented every component. The painting analogy is my own.
Backup ↩
DSH extension layers
Plugins compose the working environment.
Profile
bundles → patch layers → user overrides
Plugins
tools · prompt sections · UI · model adapters
Domain service
rheplicant compute, over JSON-RPC
The assembled configuration still needs checking.
DSH assembles plugins through profiles, bundles and ordered patch layers; dump-config labels each row with the layer that set it. Plugins add tools, prompt sections, UI and model adapters. rheplicant-agent uses this to connect domain tools to a separate scientific service. The missing-persona defect is why the assembled configuration needs checking.
Backup ↩
Emulator runtime
A task-specific model trades generality for speed.
Runtime per call: one set of CMB and matter spectra
Bolliet and colleagues built CosmoPower emulators that return a set of spectra in about sixty milliseconds, against about a minute for high-precision CLASS or CAMB. The axis is logarithmic: the gap is a factor of about a thousand. The emulator answers only the question it was trained for.
Backup ↩
GPU energy estimate
About a quarter watt-hour per 500 tokens, GPU only.
Measured · Sep 2026 deployment test
8 concurrent image+text requests
GPU ≈ 234 W
≈ 130 tokens/s in total
Estimate · GPU only
≈ 1.8 J per token
500 tokens ≈ 0.25 Wh
≈ a 10 W LED for 90 seconds
The deployment test ran eight concurrent image-and-text requests. It recorded about two hundred and thirty-four watts of GPU power and about a hundred and thirty generated tokens per second in total. That is about 1.8 joules per token, so five hundred generated tokens cost roughly a quarter of a watt-hour. This counts the GPU only under that loaded workload, with no host, no cooling and no idle time, and the generated tokens include the model's reasoning.
Backup ↩
Routing and a self-hosted model
Name model tiers, not versions; some jobs run on the group GPU.
Rule for subagents
least expensive model that reliably does the bounded task
It named versions; they are now "older generation".
OpenAI-compatible
Group GPU server Gemma 4 31B · llama.cpp
Deployment test, 8 concurrent requests · GPU only · energy estimated
≈ 234 W at ≈ 130 tokens/s in total → 500 tokens ≈ 0.25 Wh a 10 W LED for 90 s
Price also decides the routing: which model gets each task. My rule for subagents asks for the least expensive model that can reliably do the bounded task. When I wrote it, it named specific versions; a few months later those are listed as older generation, so a rule like this should name tiers, not versions. Separately, I use Gemma, self-hosted on our group's GPU server: a quantised 31-billion-parameter model on one A100, with llama.cpp, called through DeepSeek Harness. [Advance.] Its deployment test ran eight concurrent requests: about 234 watts of GPU power for about 130 generated tokens per second in total. That works out to roughly a quarter of a watt-hour per five hundred generated tokens, GPU only, at that load.
Backup ↩
One turn in rheplicant-agent
A domain harness connects the model to scientific tools.
YAML
model route, tools, prompts,
rheplicant-agent: five model-facing tools in front of rheplicant, a radio-telescope digital twin
Follow one turn from the prompt. The harness assembles context from system-prompt sections, the project's AGENTS.md and the tool schemas, then calls the model. The model is told to call validate before run. Each call passes a policy check, goes to a Python service over JSON-RPC, and its result comes back as text, which becomes input to the next model call. Every step goes into the session log. [Advance.] Here is a real turn, recorded on 13 November in the same console. I asked it to validate and run one of the demo tasks. The Trajectory view lists every record in the turn. First come the system prompt and my message. Then come three context records the harness added: workspace instructions, the runtime context, and the list of skills. After that come the model's one-line plan, the validate call and its result, the run call and its result, and the final answer. The bars at the top show where the time went: input, model and tools. On the right is the validate call: its payload, the result text the model reads, the tool's schema, and about three seconds of runtime. The whole turn took nine seconds. [Advance.] rheplicant-agent is our own domain harness: twenty-six packages on DSH, without forking it, and rheplicant sits behind five model-facing tools, from validate to run. Model route, tools, prompt sections and UI are configuration rows. One defect taught me where rules belong: the document-authoring guidance lived in persona presets, but the console mounted the tools without a persona, so the agent never saw it. The guidance now travels with the tools.
Backup ↩
Physical-model acceptance
The physical model still needed questioning.
Codex rejects DSH ×4
implementation errors found and fixed
Continue → loop paused by hand
and question the physical model
Another task, from unpublished work, so only its shape. Codex rejected DSH's work four times and found real implementation errors. Then Codex took over and ran on its own for about a hundred and fifty more revisions over three and a half days. [Advance.] Neither agent stepped back to question the physical model itself, by which I mean the scientific assumptions behind the calculation. Eventually I paused the loop and asked that question myself. The notebook kept the attempts and reviews, but whether the calculation addressed the right physical problem stayed my decision. A longer chain of revisions can still need a different question from the researcher.
Backup ↩
Sessions and access
Sessions stay native; the task record moves.
aw moves no sessions 386
Claude Code sessions Codex imported by itself
Access is separate Anthropic consumer logins
Free · Pro · Max
only in claude.ai and Claude Code
My Codex desktop app imported three hundred and eighty-six of my Claude Code sessions on its own. aw moves no sessions; its notebook is the task record, with instructions and files that the next environment can read. [Advance.] Access to the model is a separate issue. Anthropic's consumer terms restrict where a Claude Free, Pro or Max login may be used: Claude Code and claude.ai. That concerns Anthropic's consumer login, not every provider. So in my workflow each model source uses its own supported access path, and the task record carries the continuity.
