Update AGENTS.md for phases 0-2: verified state, lexer design facts, gotchas

AGENTS.md

@@ -39,18 +39,33 @@ is correct. Run tests.
 
 ## Verify with the JUnit XML, not the exit code
 
-A green `test` task does not prove tests ran. Always confirm:
+A green `test` task does not prove tests ran. Use the committed wrapper:
 
 ```bash
-cat build/test-results/test/*.xml | head -3
+./jaitest                      # whole suite, prints per-suite JUnit XML numbers
+./jaitest --tests '*Lexer*'    # one class while iterating
 ```
 
-Look for `tests="N"` with `failures="0" errors="0"` and **N > 0**.
+It fails when the total test count is 0, so a silent no-op run cannot look
+green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and
+look for `tests="N"` with `failures="0" errors="0"` and **N > 0**.
 
-Last verified state (all green):
+Last verified state (all green, `./jaigradle check` and `verifyPlugin` too):
 
 ```text
-TEST-dev.hgh.HarnessSmokeTest.xml -> tests=2 skipped=0 failures=0 errors=0
+dev.hgh.HarnessSmokeTest                    tests=2
+dev.hgh.jai.JaiFileTypeTest                 tests=3
+dev.hgh.jai.editor.JaiEditorSupportTest     tests=5
+dev.hgh.jai.highlighting.*                  tests=8
+dev.hgh.jai.lexer.JaiCorpusLexerTest        tests=2   <- the Tier 0 gate
+dev.hgh.jai.lexer.JaiLexerTest              tests=13
+-> total 33, failures+errors 0
+```
+
+The corpus gate reports what it actually did; check the line is still there:
+
+```text
+Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
 ```
 
 ---
@@ -65,20 +80,40 @@ TEST-dev.hgh.HarnessSmokeTest.xml -> tests=2 skipped=0 failures=0 errors=0
 - `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions.
 - `mise.toml` → `java = "temurin-21"` (was 17, which was broken).
 - `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified.
+- `jaitest` — runs the suite and reports the JUnit XML (see above).
 - `src/test/kotlin/dev/hgh/HarnessSmokeTest.kt` — proves the platform boots
   headlessly. **If this fails, no other test can be trusted.**
+- **Phase 0** — git repo, template stripped, `plugin.xml` rewritten.
+- **Phase 1** — `JaiLanguage`, `JaiFileType` (+ icon), `JaiTokenTypes`,
+  hand-written `JaiLexer`. Tier 0 corpus gate and Tier 1 golden tests green.
+- **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement),
+  colour settings page, commenter, brace matcher. Plugin Verifier passes
+  against IU-253/261/262.
+
+### Lexer design facts worth knowing before touching it
+
+- **State is always 0.** Nested block comments and here-strings are each
+  consumed inside a *single* token, so no context crosses a token boundary and
+  the lexer can restart anywhere. Do not add lexer states without re-checking
+  incremental re-highlighting.
+- **Deliberate divergences from the compiler's lexer**, all documented in
+  `JaiTokenTypes`' KDoc: `#ident` is one DIRECTIVE token; `#string ... ID` is one
+  HERE_STRING token; `::` and `:=` are single tokens; the backtick is its own
+  token.
+- **Built-in type names and `it`/`it_index` are IDENT**, refined into separate
+  token types only by `JaiHighlightingLexer`, which the parser never sees. Do not
+  reserve them (language reference §14.9).
+- Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build.
 
 ### Not done — pick up here
 
-1. **`git init`.** Still not a repository. Do this first; there is no rollback
+1. **Phase 3: parser.** `.bnf` grammar → Grammar-Kit generated parser + PSI in
-   point right now. Add `.kotlin` to `.gitignore` (currently untracked and
+   `src/main/gen`, plus a `ParserDefinition`. Gate: Tier 3 corpus parse with zero
-   unignored).
+   `PsiErrorElement` (allowlist committed and shrinking).
-2. **Strip the template.** `src/main/kotlin/MyToolWindowFactory.kt`,
+   Note `JaiFileTypeTest` documents the one thing blocked on this: until a
-   `MyMessageBundle.kt`, `messages/MyMessageBundle.properties`, and the
+   `ParserDefinition` exists, PSI files for `.jai` are plain text, so
-   `toolWindow` registration + `resource-bundle` in `plugin.xml` are all
+   PSI-dependent features (comment action, structure view) cannot be tested.
-   JetBrains scaffold and must go. `plugin.xml` metadata (name, vendor,
+2. **Phases 4+** — see the plan.
-   description) is still placeholder text.
-3. **Phase 1: lexer** — see the plan.
 
 ### Open questions for the user (unanswered)
 
@@ -116,6 +151,11 @@ no human review. Also assert no `BAD_CHARACTER` and that every `advance()`
 strictly increases the offset (catches infinite loops, which hang the IDE rather
 than failing visibly).
 
+This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files,
+17.1 M chars, 3.0 M tokens, in under a second. `JaiSyntaxHighlighterTest` runs
+the same sweep to assert every token type the corpus produces (105 of them) has
+a colour. Keep new lexer work under these gates rather than adding snippets.
+
 Same idea later: zero `PsiErrorElement` across the corpus for the parser;
 formatting is idempotent for the formatter.
 
@@ -125,6 +165,12 @@ formatting is idempotent for the formatter.
 
 - **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused
   `Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`.
+- **Concurrent Gradle runs corrupt the test results.** If something else (an
+  IDE, an agent's background checker) runs `test` at the same time, one of the
+  two dies with `java.io.EOFException` or
+  `NoSuchFileException: .../in-progress-results-generic.bin`, and no XML is
+  written. It is infrastructure, not a test failure — `./jaitest` retries once
+  automatically. Do not go debugging the test that "failed".
 - **`timeout` does not exist on this macOS box.** Do not use it in scripts.
 - **Noisy test stderr.** Fixture runs log Vue/JS `PluginException`s from bundled
   plugins. Tests pass regardless, but real failures can be buried — check the
@@ -139,6 +185,7 @@ formatting is idempotent for the formatter.
 
 - Verify claims by running commands; prefer empirical checks over docs. Several
   plan decisions came from testing assumptions that turned out false.
+- Work in small increments: change one thing, run `./jaitest`, commit when green.
 - Keep the working tree clean — revert throwaway probes.
 - Generated sources (once Grammar-Kit is wired up) go in `src/main/gen`, are
   committed, and are **never hand-edited**; regenerate instead.