Skip to content
verikun

Self-healing in CI

vk ai repairs a step whose selector stopped resolving, and it does that on CI exactly as it does on your laptop. There is no flag to turn it off. That is safe for a gate because a heal is narrow (only two failure kinds qualify) and loud (every one is recorded and named in the report).

What actually runs where

Section titled “What actually runs where”
Two zones. Left: a throwaway CI runner holding the model API key, where tests compile to a plan IR, get cached, replay, and are assembled into a report uploaded as a CI artifact; the model is woken there to compile and to repair. Right: a device box running vk server, which drives the app over adb or idb. Exactly one channel crosses between them, carrying one validated command per HTTP round-trip.

The part people get backwards: the engine is on the CI runner, not on the device box. Compiling the prose, the plan cache, replaying the plan, deciding a repair and assembling the report all happen runner-side. vk server receives one validated command per HTTP round-trip, runs it over adb/idb, and hands back pass/fail plus that step’s evidence. It holds no API key and never talks to a model, so no pull-request code and no model credential ever reach the machine with the phone. Full responsibility split in Remote devices & CI.

Should it heal, or just fail?

Section titled “Should it heal, or just fail?”

It heals, and you cannot stop it — the repair budget is a fixed three attempts per step.

What matters is how little that can hide. Exactly two failures are healable: a selector that did not resolve, and one that was ambiguous. Everything else is terminal.

  • A heal can hide a renamed or moved control. That is the point. A button that became @submit_button still gets tapped, and the report tells you it changed. Failing the build for a cosmetic rename is noise, not signal.
  • A heal cannot hide a failed assert. Assertions are never healed — healing one would mask the exact regression the test exists to catch.
  • A heal cannot hide a flow that went to the wrong screen. When the live screen has nothing serving the failed step’s intent, the model answers give_up, which is terminal.

A repair is also re-validated against the same grammar gate as the original compile, so a hallucinated command is rejected rather than run. The full matrix is in Contracts.

A heal is loud, not silent

Section titled “A heal is loud, not silent”

Every repair leaves three traces. This is what makes healing safe to leave on:

Where What you see
The step, in JUnit + HTML A model-healed: note naming the original failure and the replacement
The suite overview and index.json A modelRepairs count, per test
The report’s vk ai panel A suggested improvement — the exact replacement to paste into your prose

That last one is the useful one. verikun does not just work around the drift, it tells you what to write so the next run does not have to. A step that heals on every run is a standing tax, not a success — usually a label-only control with no stable identifier. See Troubleshooting.

If you want the strict gate anyway

Section titled “If you want the strict gate anyway”

Some teams want “a repair means the test is wrong, fail the build.” That is a policy decision, and it composes as a CI step over the index.json manifest rather than a verikun flag:

- name: Fail the build if any test needed a model repair
  if: always()
  run: |
    repairs=$(jq -s '[.[].tests[].modelRepairs] | add // 0' .verikun/suites/*/index.json)
    echo "model repairs: $repairs"
    [ "$repairs" -eq 0 ]

Gate on modelRepairs for a correctness policy, and on the per-test costUsd for a spend policy — they are not the same signal. A recompile after a verikun upgrade costs money but is not a repair, so a repairs gate fires only on genuine UI drift.

What it costs — and the cold cache

Section titled “What it costs — and the cold cache”

The cost model says a green suite costs roughly $0, because a cached plan replays with no model calls. That is a claim about a machine that keeps its cache. A CI runner usually does not.

The plan cache lives in ./.verikun/plans/, relative to the working directory. A fresh ubuntu-latest checkout has no such directory, so every test recompiles on every run and never reaches the steady state. Cost & budget covers the other levers.

Persist it:

- name: Restore the plan cache
  uses: actions/cache/restore@v6
  with:
    path: .verikun/plans
    key: verikun-plans-${{ github.run_id }}
    restore-keys: verikun-plans-


# ... install, then run the suite ...


- name: Save the plan cache
  if: always()
  uses: actions/cache/save@v6
  with:
    path: .verikun/plans
    key: verikun-plans-${{ github.run_id }}

A unique key per run so save always writes, with restore-keys prefix-matching the most recent previous entry. Use the restore + save pair rather than the combined actions/cache action: the combined action does not save when the job fails, and a red run is precisely the one whose fresh compiles you want to keep.

Cache .verikun/plans and nothing beside it. .verikun/plan-locks/ holds the short-lived per-machine locks that stop concurrent runs compiling the same chunk twice, and must not travel.

The key does not have to be right. Every entry is gated on a compiler fingerprint at read time, and the filename already folds in the prose, package, build and platform — so a wrong or stale restore produces a miss, which recompiles, never a wrong replay. A cache is portable: entries embed no absolute paths, so the same cache replays on any machine with the same verikun version, platform, package and test text, and seeding a runner from a developer’s machine is fine.

Four things worth knowing before you rely on it:

  • A verikun upgrade invalidates everything — the fingerprint folds in the version and the grammar text (Contracts).
  • --app-build $GITHUB_SHA defeats the exact key. Pass it at the granularity you want to invalidate at; a missed key still seeds from a prior plan, which beats a cold compile.
  • A restored cache is not behaviourally inert: seeding ignores the build and the fingerprint, so if a compiled plan looks wrong, clear the directory and --recompile first.
  • The path is relative to the working directory. A working-directory: on the suite step means the cache path: has to match it.

Where to go next

Section titled “Where to go next”