Zach Zayac

Projects / WITAN

Local Models, the Fleet & the Mimicry Loop

Active program

A fleet of small open-weight models — benched on real prompts, kept healthy, and taught by Claude.

Updated

What problem it solves

The whole Witan bet rides on small local models being dependable colleagues rather than party tricks — which means three unglamorous problems become central: which models earn a seat, how they stay served and healthy on always-on consumer hardware, and how they get better at the specific jobs this system needs. Cloud APIs make those problems invisible by renting them. Running local means owning them.

The thesis

Three beliefs drive this segment. Models earn seats by evidence — on replayed real workload, not benchmarks. Always-on local serving is a reliability discipline — the failure modes are novel and deserve real engineering, not cron-job duct tape. And the newest and most interesting: the strongest model you have should be a teacher, not just a builder — every piece of work the frontier model does for you is potential curriculum for the small models you own.

The fleet and the EXO ring Four Apple-silicon minis run role-pinned small models behind wedge guards; a dashed ring marks EXO, which convenes them into one larger machine; a control plane sits off the ring. ALWAYS-ON, LOW-WATT, OWNED — RELIABILITY IS A PROGRAM, NOT A HOPE EXO RING — convene the minis into one larger machine for heavier jobs Front door voice → text · always-on 8B wedge guard Worker parsers · judges role-pinned 1B–9B wedge guard Worker scribes · critics role-pinned 1B–9B wedge guard Worker screeners · enrichers role-pinned 1B–9B wedge guard Control plane dashboard · queues · data · benches deliberately OFF the ring a wedged-but-alive server gets caught on-node, backstopped centrally, logged as history — and predicted before it happens heavier jobs climb to the ring before any cloud rung — the compute stays owned
The fleet: role-pinned small models on always-on Apple-silicon minis, each behind a wedge guard, with EXO able to convene them as one machine.

Key features

  • A role-pinned roster. US-origin open-weight families — Llama 3.x classes sized to the job (1B/3B/8B), with Gemma, Phi, and Granite in supporting roles — each agent pinned to a model class with fixed sampling parameters so runs reproduce. The full roster design runs to hundreds of cards: shared services, domain specialists, mixtures, cache profiles.
  • Evidence-based refresh. A bench harness replays real captured prompts — pulled from Cortex’s lossless run records — side-by-side against challenger models and produces a verdict. Nightly prompt-regression runs pin the live prompts against fixtures so a prompt edit can’t silently regress an agent.
  • The wedge program. An alive-but-wedged model server (up, health-checking, doing nothing) is this fleet’s signature failure. The answer is layered: an on-node guard, a central backstop with notification thresholds, a durable incident log, and a predictor that flags a machine likely to wedge before it does. Durable supervisors keep the serving stack itself up.
  • Fleet-aware serving. Per-endpoint routing knows which models need reasoning-style APIs, memory ceilings are enforced after real out-of-memory incidents, and scheduled windows time-share the machines between duties.
  • The EXO ring. When a task exceeds any single machine, the Mac minis convene as one larger machine via distributed inference — a middle rung on the escalation ladder between “one mini” and “the cloud,” keeping heavier work local.
  • The mimicry loop (the flagship experiment). When Claude builds a Witan feature, it now decomposes the work into atomic steps of ~50 lines or less, each written as a complete task card: the ask stated in six lines, the exact file context a model with no house knowledge would need, the constraints, and the reference diff. A 9B-class local code model then attempts the same steps; the deltas between its attempts and the reference drive prompt, harness, and serving changes. The build itself produces the training curriculum — and every evaluation across the whole system is likewise captured as a durable labeled example for eventual fine-tuning.
The mimicry loop Claude decomposes a build into atomic steps with full step records; a local 9B model replays each step; the delta against the reference diff drives prompt, harness, and serving changes — and the loop repeats. THE BUILD IS THE CURRICULUM Claude builds a feature the system's strongest coder, working at full speed Atomic steps ≤ ~50 changed lines each — one file, one concern Step record the ask in six lines · the exact context a stranger-model needs · constraints · the reference diff Local 9B replays it same ask, same context, its own diff Delta vs reference where does the small model fall short? Tune the setup prompts · harness · serving next replay runs better — measured, step by step and every evaluation, system-wide, is captured as a labeled example — the fine-tuning corpus grows as a side effect of operating
The mimicry loop: Claude builds at full speed while producing a replayable curriculum, and the local model's misses become the next round of prompt and harness improvements.

Highlights

  • The incumbent 8B kept its seat in this month’s refresh bench — challengers lost on real replayed prompts. An upgrade that doesn’t happen, decided by evidence, is the process working exactly as designed.
  • The mimicry program reframes the relationship between frontier and local models: Claude isn’t just constructing the system, it’s generating reproducible, replayable lessons in how to construct it — a path from “Claude codes everything” toward “local models code under supervision” that’s measured at every step.
  • Reliability went from reactive to predictive: the fleet now flags the machine that’s about to wedge.

Development log

July 8, 2026

Page created. This past two weeks: the wedge program (guard → backstop → log → predictor), durable serving supervisors, per-endpoint reasoning-model routing, memory-ceiling enforcement after a real OOM, the model-refresh bench and its incumbent-stands verdict, the nightly prompt-regression harness, the training-dataset layer (every evaluation becomes a labeled example), and the mimicry program’s kickoff — first step records written and a 9B code model being stood up to replay them.