bs_explorer/CLAUDE.md

# Project guide for Claude Code

This file is loaded automatically when working on `bs_explorer` in any
Claude Code session, on any machine where the repo is cloned. Keep
machine-specific or user-specific facts out of here — they belong in
`~/.claude/` memory, not in the repo.

Project status, decisions, and roadmap below are durable: update them
when reality changes, not for every transient task.

## What this project is

`bs_explorer` is an application layer on top of Viveris's
[jtag-boundary-scanner](https://github.com/viveris/jtag-boundary-scanner)
library (LGPL). End goal: program FPGAs/CPLDs over JTAG from a CLI tool
on a host with an FTDI/Digilent/J-Link probe — Xilinx external SPI
configuration flash via a BSCAN proxy (started with the KU15P), and
other families (Lattice, Microsemi, …) by playing a vendor-exported SVF.

The Viveris library itself lives unchanged in `src/modules/`. Everything
new is in `src/bs/` (the REPL) and the project modules (`fpga/`, `bscan/`,
`spi_flash/`, `svf/`, `probes/`) sitting alongside the Viveris ones.

## Architecture

```
src/                       — code + libs —
├── bs/                    Application (readline REPL, no business logic)
├── libs/libftd2xx/        Vendored FTDI SDK
└── modules/
    — Viveris's library (LGPL, unchanged) —
    ├── jtag_core/         TAP state machine, IR/DR shifts
    ├── bsdl_parser/       .bsd loader
    ├── bus_over_jtag/     SPI/I²C/MDIO/parallel mem bit-bang over EXTEST
    ├── drivers/           FTDI, J-Link, Linux GPIO, LPT, Digilent (optional, dlopen)
    ├── script/            Script engine (the real UI)
    ├── config/            Built-in config.script
    ├── os_interface/      Portable fs/network wrappers
    └── natsort/           Natural pin-name sorting
    — new (this project) —
    ├── fpga/              Registry loader (parses data/fpga_registry.yaml, libyaml)
    ├── bscan/             JTAG TAP primitives (set_ir/shift_ir/shift_dr/tap_reset/
    │                      idle_cycles) + BSCAN proxy (bitstream load, SPI-over-USER1)
    ├── spi_flash/         SPI NOR chip DB + read/erase/program/verify over a callback
    ├── svf/               SVF player (svf_play): SIR/SDR/RUNTEST/STATE, masked compare
    └── probes/            Probe-config profiles loader (parses data/probes.yaml, libyaml)
data/                      — runtime resources, looked up CWD-relative —
├── fpga_registry.yaml     FPGA registry (IDCODE → BSDL, IR opcodes, proxy, caveats, prog, max_tck)
├── probes.yaml            Probe-config profiles (defaults + per-probe overrides)
├── bsdl_files/            BSDL files for target FPGAs
├── bscan_proxies/         BSCAN proxy bitstreams (MIT, from quartiq)
└── scripts/               Example scripts
doc/                       Tutorial and longer-form docs (doc/tutorial.md is the end-to-end walkthrough)
CMakeLists.txt             Top-level build (binary lands at build/bs)
```

User interacts with the project through the script engine commands
(`jtag_*`, `set`, `if`, …), exposed via the REPL or piped script files.
Adding a feature usually means adding a new script command in
`src/modules/script/script.c` and the supporting C in a new module.

## Roadmap

| Phase | Module | Status | Summary |
|-------|--------|--------|---------|
| 1 | `bs/` cleanup, REPL polish, README | **done** (commit `7cb3627`) | Fix format-strings, delete dead code, tab-completion, banner |
| 2 | `fpga/` | **done** (commit `545fe09`) | Per-target descriptor (IDCODE, BSDL, IR codes, proxy path, caveats). Now a **runtime YAML** registry (`data/fpga_registry.yaml`, libyaml), later gaining `prog` method + `max_tck_khz`. |
| 2.5 | `bscan/` | **done** (commit `dec0d14`) | Load BSCAN proxy bitstream via `CFG_IN`, expose fast `bscan_spi_xfer()` via `USER1`. Required for realistic flashing speeds. |
| 3 | `spi_flash/` | **done** (commit `c4afe87`) | Chip DB (JEDEC ID → page/sector/cmd set) + generic `read/erase/program/verify` over an `xfer` callback. detect+read validated on KCU105; erase/program implemented but not yet hardware-tested. |
| 4 | script commands | **done** (commit `d6f843e`) | `flash_detect`, `flash_read` (+file), `flash_erase`, `flash_write`, `flash_verify`. Full set validated on KCU105 (save/erase/write-random/verify/restore round-trip). ~100 KB/s write once the proxy is loaded. |
| 5 | `probes/` + JTAG-link | **done** | `data/probes.yaml` probe-config profiles (`jtag_open <idx> <profile>`, `jtag_profiles`, `jtag_close`); driver-neutral `JTAG_TCK_FREQ_KHZ`/`JTAG_RTCK`; device `max_tck_khz` clock cap resolved at `jtag_autoinit`; `prog` method tag. See the config-strategy design note. Validated on the IGLOO2 (FlashPro). |
| 6 | `svf/` | **done** (subset, commit `c77d86e`) | SVF player + `svf_play`: SIR/SDR with masked TDO compare, RUNTEST, STATE — single-device. Validated on the IGLOO2 IDCODE; a real Libero SVF and a generic `program` dispatch off the `prog` tag are still TODO. |

Move forward phase by phase: validate one with the user before starting
the next. Don't break the validated path
`jtag_open → jtag_autoinit → jtag_mode 0 EXTEST` while
refactoring.

## Key technical decisions

### Flashing path: BSCAN proxy, not EXTEST

The Viveris library has `bus_over_jtag/spi_over_jtag.c` which bit-bangs
SPI on FPGA pins placed in EXTEST. **This is not usable for actual
flashing.** Each SPI bit costs 2 full boundary-scan-register shifts
through JTAG, plus USB latency on the FTDI side. Order of magnitude:
~30 bytes/s effective, i.e. **weeks** to flash a 128 MB part. Useful
only for one-shot operations (read JEDEC ID, check wiring).

Real flashing path: load a small "BSCAN proxy" bitstream into the FPGA
fabric via standard JTAG configuration (`CFG_IN`). The proxy uses a
`BSCANE2` primitive to bridge the JTAG `USER1` instruction's DR shift
to the physical SPI pins, running at fabric speed (CCLK driven
internally — the `STARTUPE3` problem disappears). Realistic throughput
50–200 KB/s, so ~10–40 min for 128 MB. This is what Vivado, OpenOCD's
`jtagspi` driver, and `xc3sprog` all do.

### Per-FPGA descriptor

`fpga_target` struct (Phase 2) holds the per-target facts that can't be
derived from the BSDL alone:
- `idcode` + `idcode_mask` — match the device on the chain
- `bsdl_filename` — BSDL to auto-load
- `cfg_in_ir_code`, `user1_ir_code`, `jprogram_ir_code` — Xilinx-specific
  private IR opcodes (read from BSDL when available)
- `proxy_bitstream_path` — path to the BSCAN proxy `.bit` for this part
- `caveats` — flags for known hardware gotchas (e.g. CCLK via STARTUPE3)

Registry is a **runtime YAML file** (`data/fpga_registry.yaml` at the repo
root), parsed via libyaml — no longer a compile-time array. It is looked
up CWD-relative (like `data/bsdl_files/`), overridable with
`$BS_FPGA_REGISTRY`, and loaded lazily on first access. Adding a part =
one YAML entry + its `.bsd` in `data/bsdl_files/` + (optionally) its proxy
`.bit` in `data/bscan_proxies/` — no rebuild. The `family` field accepts
`xilinx_*`, `microsemi_igloo2/smartfusion2`, `lattice_machxo2/3`
(enum in `fpga.h`).

### Digilent SMT2 modules need libdjtg, not raw MPSSE

Several Xilinx dev boards (KCU105, ZCU102, …) embed a Digilent
JTAG-SMT2 / SMT2-NC for USB-JTAG. Even though it presents a stock
FTDI FT232H over USB (VID:PID 0403:6014), it runs a proprietary
firmware that **does not respond to plain MPSSE commands** — TCK
toggles but the level-shifters/buffers stay disabled, so TDO floats
high ("all ones" symptom). Standard FTDI driver path is dead on these
boards.

Workaround: `src/modules/drivers/digilent_jtag/` wraps libdjtg/libdmgr
(Digilent Adept Runtime), loaded via `dlopen` at runtime — no Digilent
binary or header in the repo. Because it's dlopen'd (degrades to "no
probe" if the libs are absent), it costs nothing to build in:
`BS_ENABLE_DIGILENT` defaults **ON** on UNIX (`<dlfcn.h>` required),
disable with `-DBS_ENABLE_DIGILENT=OFF`. Adept Runtime is only needed
at runtime to actually drive such a probe.

### Probe config profiles (data/probes.yaml)

Probe wiring/electrical settings live in `data/probes.yaml` (parsed by
`src/modules/probes/`, libyaml), layered on top of the built-in
`config.script` defaults: a `defaults:` map applied on every `jtag_open`
(so opening without a profile is deterministic) plus named `profiles:`
selected with `jtag_open <idx> <profile>` (`jtag_profiles` lists them).
Each key/value is pushed into the script envvar store the driver reads at
open time. The mechanism is driver-agnostic (any `set`-able probe var),
but today only the FTDI driver (and minimally the Linux-GPIO one) reads
config envvars, so profiles mostly tune FTDI. Motivating case: the
embedded FlashPro on Microsemi kits (FT4232H ch.A) needs ADBUS4 high-Z —
the `flashpro` profile sets `PROBE_FTDI_SET_PIN_DIR_ADBUS4: 0`. See the
config-strategy design note below for where this is headed.

### Xilinx caveats

On 7-Series / UltraScale / UltraScale+, `CCLK` is not a regular I/O
pin — it goes through the `STARTUPE3` primitive. Cannot be driven
directly in EXTEST. Non-issue once we use the BSCAN proxy (CCLK is
driven by the fabric clock inside the proxy).

The FPGA must be in a configurable state before loading the proxy.
Issue `JPROGRAM` first to reset, then `CFG_IN` + shift the bitstream,
then `JSTART` and check `DONE`.

## Probe / JTAG-link / device config strategy (design note)

Partly built, partly planned. Guiding idea: **separate three concerns
that are conflated today, and express the shared JTAG-link settings in
driver-neutral terms that get resolved per session.**

### The three concerns

| Layer | What it owns | Where it lives | Applied |
|-------|-------------|----------------|---------|
| **Probe (sonde)** | driver + interface, pin map, buffer-enable, TRST/SRST pins, level-shift, *max TCK the adapter supports* | `data/probes.yaml` (done) | at `jtag_open` |
| **JTAG link** | TCK freq, RTCK, reset behaviour, chain layout — **driver-neutral names**, resolved to effective values | *missing today* | open, then refined after detect |
| **Device** | IDCODE/BSDL/IR/proxy/caveats, programming method, *max TCK the part/board tolerates* | `data/fpga_registry.yaml` (done) | after IDCODE match |

### The smell that motivated this

Frequency has no single home. Only the FTDI driver reads
`PROBE_FTDI_TCK_FREQ_KHZ`; our Digilent driver **hardcodes 4 MHz**
(`digilent_jtag_drv.c`); J-Link/LPT read nothing. And the `PROBE_FTDI_*`
namespace mixes probe *wiring* (pin map, ADBUS4) with *link* properties
(freq, RTCK, TRST timing). Frequency isn't an FTDI fact — it's a link
fact bounded by both the probe and the board/device.

### Target model

- **Driver-neutral link vars** (`tck_khz`, `rtck`, `reset`, …) set once
  (by user / probe defaults / device). Each driver consumes them: our
  Digilent driver reads them directly; the Viveris FTDI driver is fed
  `PROBE_FTDI_TCK_FREQ_KHZ` via a thin translation shim at open (no
  Viveris edit). J-Link/LPT can stay on their defaults.
- **Resolution**: effective `tck = min(user request, probe max, device/
  board max)`. A small session step computes it at open, and re-applies
  after `jtag_autoinit` once the device (hence its cap / programming
  method) is known — using the `jtag_close`→reopen seam if a re-init is
  needed. This also dissolves the chicken-and-egg: a conservative link
  default gets you to detection, then the device refines it.
- `data/probes.yaml` gains an optional `max_tck_khz`; `data/fpga_registry.yaml`
  gains optional `max_tck_khz` + a `prog` method tag (`proxy_spi`/`svf`).

### Phasing

- **A (done)** — one canonical `JTAG_TCK_FREQ_KHZ` (kHz): mirrored to
  `PROBE_FTDI_TCK_FREQ_KHZ` at `jtag_open` for the Viveris FTDI driver,
  read directly by our Digilent driver; unset → each driver's own default
  (FTDI 1000, Digilent 4000). Set it via `set`, a `data/probes.yaml` profile,
  or `defaults:`. (FTDI path hardware-validated; Digilent path untested.)
- **B (done)** — registry `max_tck_khz` caps the clock: `jtag_autoinit`,
  after identifying the chain, clamps `JTAG_TCK_FREQ_KHZ` to the smallest
  device max and re-opens the probe once (via the stored probe id) to
  apply, then re-scans. Within-cap / unset just report the cap. Effective
  `tck = min(request, device max)`; probe-max (`min(..., probe max)`) is
  still TODO.
- **C (done)** — `prog` method tag (`proxy_spi`/`svf`/`none`) on each
  registry entry, inferred when omitted (proxy → `proxy_spi`; Microsemi/
  Lattice → `svf`); shown by `fpga_info`/`fpga_list` and available via
  `fpga_prog_method_name()` for backend dispatch. RTCK generalised as a
  neutral `JTAG_RTCK` (mirrored to `PROBE_FTDI_JTAG_ENABLE_RTCK`,
  FTDI-only). Reset abstraction deferred — it's a bundle of probe-
  specific pin/polarity/timing vars with no clean neutral form yet. The
  actual `program` dispatch command lands with the SVF player.

What exists already: the **probe layer** (`data/probes.yaml`) and the
**device layer** (`data/fpga_registry.yaml`). The new work is the **JTAG-link
layer** in the middle.

## Programming backends: beyond Xilinx external flash (design note)

Not yet implemented — captured so the design is ready. Guiding vision:
**one generic SVF player as a near-universal backend, with native
backends only where streaming / speed / control justify them.**

### Why two layers

Practically every JTAG-programmable device (Xilinx fabric config,
Lattice MachXO2/3, Microsemi IGLOO2/SmartFusion2, Altera, CPLDs, …) can
be programmed from an **SVF** file exported by its vendor tool. SVF is a
flat list of `SIR`/`SDR`/`RUNTEST` ops with the vendor's algorithm
already baked in — so a single SVF player, built on our existing
`bscan_set_ir` / `bscan_shift_dr` primitives, programs almost anything
with no per-vendor algorithm code.

Native backends are the *exception*, justified only when you want fine
control, speed, or no vendor-export step. Our Xilinx **external SPI
flash** path is the prime example and stays native: load the BSCAN proxy
once, then stream raw binary with separate read/erase/program/verify,
progress and partial ops. The equivalent SVF ("indirect flash") is huge
(tens of MB of ASCII) and inflexible.

### Where each path applies

| Target | Recommended path |
|--------|------------------|
| Xilinx external SPI config flash | **native proxy** (`bscan/`+`spi_flash/`, done). SVF works but is bloated. |
| Xilinx fabric config (volatile) | SVF, or our `bscan_load_bitstream` (equivalent) |
| Lattice MachXO2/3 (internal flash) | **SVF** (Diamond/Radiant export). Native IEEE-1532 ISC optional, only for a self-contained `.jed` flow. |
| Microsemi IGLOO2 / SmartFusion2 | **SVF/STAPL** (Libero/FlashPro export). Native algorithm too complex/proprietary. |
| Altera, CPLDs, misc JTAG | SVF |

### The SVF player (the real work)

**Status: initial implementation in `src/modules/svf/` + the `svf_play <file>`
command.** Supports the single-device subset: `SIR`/`SDR` with
`TDI`/`TDO`/`MASK`/`SMASK` and the masked TDO compare, `RUNTEST`
(TCK/SCK + SEC), `STATE` (RESET/IDLE), `ENDIR`/`ENDDR` (IDLE only),
`HIR/HDR/TIR/TDR` (length 0 only), `TRST`, `FREQUENCY`. Built on the
`bscan_*` primitives (`shift_ir`/`shift_dr`/`tap_reset`/`idle_cycles`).
Validated on the live IGLOO2 with a hand-written IDCODE-check SVF (pass
and a deliberate mismatch). `svf_play` warms up the FTDI link first (its
first data read after open returns stale FIFO content). Not yet wired
into a generic `program` dispatch off the `prog` tag; no multi-device
headers, no non-IDLE end states. Remaining design below.

A player is more than shifting bits. It must handle:
- `SIR`/`SDR` with `TDI`/`TDO`/`MASK` — the **masked TDO compare** is
  what detects prog/erase failures; that's the main addition over
  today's primitives;
- `RUNTEST` delays (erase/program waits, in TCK or time) → reuse
  `bscan_idle_cycles` + a sleep;
- `HIR/HDR/TIR/TDR` headers/trailers and `ENDIR/ENDDR/STATE` (multi-
  device chains, TAP state) — a standard subset covers ~all files.

It reuses probe drivers, `jtag_core`, `bscan_set_ir`/`bscan_shift_dr`
and IDCODE/BSDL detection unchanged. **STAPL** (`.stp`/`.jam`) is the
richer native format (a full language → needs an interpreter); more
work, defer — do SVF first.

### Registry / dispatch

`fpga_target` is Xilinx-flavoured today (`ir_cfg_in`, `ir_user1`, …).
Add a "programming method" tag (`proxy_spi` / `svf` / optional
`lattice_isc`) selected off the `family` enum (add
`FPGA_FAMILY_LATTICE_*`, `FPGA_FAMILY_MICROSEMI_*`). An SVF player is
family-agnostic and needs no per-part opcodes — it can even run without
a registry entry.

**Scope caveat.** iCE40 is usually programmed over SPI directly (or
one-time NVCM) with minimal JTAG — out of scope for this JTAG-centric
tool.

## Embedded port (design note)

Not yet implemented — captured for a possible standalone programmer.
No hard blocker: standalone JTAG programmers are all MCU-based. The
architecture has the right seam (the driver function-pointer table).

**Reference target: Arduino GIGA R1 WiFi** — STM32H747XI (Cortex-M7
@ 480 MHz + M4 @ 240 MHz, 1 MB RAM, 2 MB flash), WiFi/BT, Arduino/Mbed
OS. The comfortable high end. ESP32 is a lighter alternative (WiFi
on-die, ~520 KB RAM); avoid bare AVR (RAM too small).

**The seam — the MCU *is* the JTAG master.** Replace `src/modules/drivers/`
with a GPIO bit-bang (or SPI-assisted: MOSI=TDI, MISO=TDO, SCK=TCK,
TMS bit-banged between scans → multi-MHz) driver behind
`drv_TX_TMS` / `drv_TXRX_DATA`. Everything above that — `jtag_core`, the
`bscan_set_ir`/`bscan_shift_dr` primitives, the `fpga_target` registry,
the SVF player and the flash logic — is portable C, reused as-is.

**Dropped / replaced on the MCU:**
- readline REPL → command interface over WiFi (HTTP/telnet) or USB-CDC;
- `os_interface` (host fs/net) → Mbed OS filesystem + network (real
  implementations, little rework);
- runtime BSDL parsing → pre-baked tables (the registry already does
  this); `bsdl_parser` can be left out;
- `libftd2xx` / Digilent dlopen → gone (the MCU is the probe).

**The one real rework: stream, don't `malloc`.** `bscan_shift_dr`
currently allocates the whole shift buffer (`malloc(nbits)` — ~19 MB for
a Xilinx proxy). On an MCU the payload (proxy `.bit`, flash image, SVF)
lives on **SD or a USB stick** and must be **shifted in chunks**; WiFi
fetches the file / triggers the job. With an SD/USB source this is
natural (read a block, shift, repeat). 1 MB RAM relaxes buffer sizes but
does not remove the need (a 32 MB image never fits).

**Ideal fit: an SVF player streamed from SD** — a classic embedded
programmer design, and it matches the small-Lattice/IGLOO2-in-a-PSU use
case (small config, modest `SDR` vectors). Big Xilinx external-flash
images (huge SVF vectors) stay on the host or a large MCU.

**Hardware notes:**
- the GIGA R1 has **no onboard microSD** — add it via the GIGA Display
  Shield, an SPI SD module, or its USB-A host port (USB mass storage);
- 3.3 V GPIO → **level-shift** for 1.8 V JTAG targets (e.g. KCU105),
  same as on the host side.

## External references

- **BSCAN proxy bitstreams**: `quartiq/bscan_spi_bitstreams` (MIT).
  Pre-built `.bit` for many Xilinx parts; Migen sources to rebuild any
  part that's missing (needs Vivado). Building a proxy that isn't
  pre-built (e.g. the KU15P) is covered in `doc/tutorial.md`, Phase 2.5.
- **Reference host-side implementation**: `openocd/src/flash/nor/jtagspi.c`.
  Defines the proxy protocol (header with bit count + CS state, then
  payload). Don't reinvent — match what OpenOCD does so we share the
  same bitstreams.
- **Viveris library docs**: `src/modules/jtag_core/` headers are well
  commented. `src/modules/config/config.script` documents every runtime
  variable.

## Build & test

```sh
mkdir build && cd build && cmake .. && make
./bs/bs           # interactive REPL
```

Build needs **libyaml** (pkg-config `yaml-0.1`; Arch `libyaml`, Debian
`libyaml-dev`) — the FPGA registry is parsed from `data/fpga_registry.yaml`
at runtime. Run `bs` from the repo root so it finds that file (and
`data/bsdl_files/`, `data/bscan_proxies/`), or point `$BS_FPGA_REGISTRY` at it.

The Digilent SMT2 backend is built by default on UNIX (disable with
`-DBS_ENABLE_DIGILENT=OFF`). To actually use such a probe, install the
Adept Runtime system-wide (provides `libdjtg.so` + `libdmgr.so`).

No automated tests yet. Smoke test = banner appears, `exit` works.
After changes touching `jtag_core`, `drivers/ftdi_jtag`, or the
`autoinit` flow, manual hardware test required: probe + KU15P board
should scan and load `xcku15p_ffve1517.bsd`.

The FTDI driver compiles with int-to-pointer-cast warnings (Viveris's
code on 64-bit Linux). Don't fix in this repo — that's upstream.

## Commit conventions

- Messages in English, lowercase first word, imperative or short
  descriptive (match existing style: "phase 1: cleanup, REPL polish,
  README", "translate README to English").
- Title ≤ ~70 chars. Body wrapped ~80 chars, with bullets for what
  changed. Mention the why for non-obvious choices.
- Sign-off block:
  ```
  Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
  ```
- One logical change per commit. Don't bundle README + bugfix +
  refactor.

## What does NOT belong here

- User preferences (tone, language used in chat, "always do X first")
  → `~/.claude/`
- Machine-local facts (which probes are physically attached, paths
  outside the repo, validated-on-this-machine notes) → `~/.claude/`
- Transient task state, in-progress decisions → conversation, not file
- Generated artifacts (`build/`, downloaded proxy `.bit` if we choose
  not to vendor) → `.gitignore`