diff --git a/AGENTS.md b/AGENTS.md index b88df3e..9e5c147 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,18 +39,33 @@ is correct. Run tests. ## Verify with the JUnit XML, not the exit code -A green `test` task does not prove tests ran. Always confirm: +A green `test` task does not prove tests ran. Use the committed wrapper: ```bash -cat build/test-results/test/*.xml | head -3 +./jaitest # whole suite, prints per-suite JUnit XML numbers +./jaitest --tests '*Lexer*' # one class while iterating ``` -Look for `tests="N"` with `failures="0" errors="0"` and **N > 0**. +It fails when the total test count is 0, so a silent no-op run cannot look +green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and +look for `tests="N"` with `failures="0" errors="0"` and **N > 0**. -Last verified state (all green): +Last verified state (all green, `./jaigradle check` and `verifyPlugin` too): ```text -TEST-dev.hgh.HarnessSmokeTest.xml -> tests=2 skipped=0 failures=0 errors=0 +dev.hgh.HarnessSmokeTest tests=2 +dev.hgh.jai.JaiFileTypeTest tests=3 +dev.hgh.jai.editor.JaiEditorSupportTest tests=5 +dev.hgh.jai.highlighting.* tests=8 +dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate +dev.hgh.jai.lexer.JaiLexerTest tests=13 +-> total 33, failures+errors 0 +``` + +The corpus gate reports what it actually did; check the line is still there: + +```text +Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly. ``` --- @@ -65,20 +80,40 @@ TEST-dev.hgh.HarnessSmokeTest.xml -> tests=2 skipped=0 failures=0 errors=0 - `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions. - `mise.toml` → `java = "temurin-21"` (was 17, which was broken). - `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified. +- `jaitest` — runs the suite and reports the JUnit XML (see above). - `src/test/kotlin/dev/hgh/HarnessSmokeTest.kt` — proves the platform boots headlessly. **If this fails, no other test can be trusted.** +- **Phase 0** — git repo, template stripped, `plugin.xml` rewritten. +- **Phase 1** — `JaiLanguage`, `JaiFileType` (+ icon), `JaiTokenTypes`, + hand-written `JaiLexer`. Tier 0 corpus gate and Tier 1 golden tests green. +- **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement), + colour settings page, commenter, brace matcher. Plugin Verifier passes + against IU-253/261/262. + +### Lexer design facts worth knowing before touching it + +- **State is always 0.** Nested block comments and here-strings are each + consumed inside a *single* token, so no context crosses a token boundary and + the lexer can restart anywhere. Do not add lexer states without re-checking + incremental re-highlighting. +- **Deliberate divergences from the compiler's lexer**, all documented in + `JaiTokenTypes`' KDoc: `#ident` is one DIRECTIVE token; `#string ... ID` is one + HERE_STRING token; `::` and `:=` are single tokens; the backtick is its own + token. +- **Built-in type names and `it`/`it_index` are IDENT**, refined into separate + token types only by `JaiHighlightingLexer`, which the parser never sees. Do not + reserve them (language reference §14.9). +- Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build. ### Not done — pick up here -1. **`git init`.** Still not a repository. Do this first; there is no rollback - point right now. Add `.kotlin` to `.gitignore` (currently untracked and - unignored). -2. **Strip the template.** `src/main/kotlin/MyToolWindowFactory.kt`, - `MyMessageBundle.kt`, `messages/MyMessageBundle.properties`, and the - `toolWindow` registration + `resource-bundle` in `plugin.xml` are all - JetBrains scaffold and must go. `plugin.xml` metadata (name, vendor, - description) is still placeholder text. -3. **Phase 1: lexer** — see the plan. +1. **Phase 3: parser.** `.bnf` grammar → Grammar-Kit generated parser + PSI in + `src/main/gen`, plus a `ParserDefinition`. Gate: Tier 3 corpus parse with zero + `PsiErrorElement` (allowlist committed and shrinking). + Note `JaiFileTypeTest` documents the one thing blocked on this: until a + `ParserDefinition` exists, PSI files for `.jai` are plain text, so + PSI-dependent features (comment action, structure view) cannot be tested. +2. **Phases 4+** — see the plan. ### Open questions for the user (unanswered) @@ -116,6 +151,11 @@ no human review. Also assert no `BAD_CHARACTER` and that every `advance()` strictly increases the offset (catches infinite loops, which hang the IDE rather than failing visibly). +This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files, +17.1 M chars, 3.0 M tokens, in under a second. `JaiSyntaxHighlighterTest` runs +the same sweep to assert every token type the corpus produces (105 of them) has +a colour. Keep new lexer work under these gates rather than adding snippets. + Same idea later: zero `PsiErrorElement` across the corpus for the parser; formatting is idempotent for the formatter. @@ -125,6 +165,12 @@ formatting is idempotent for the formatter. - **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused `Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`. +- **Concurrent Gradle runs corrupt the test results.** If something else (an + IDE, an agent's background checker) runs `test` at the same time, one of the + two dies with `java.io.EOFException` or + `NoSuchFileException: .../in-progress-results-generic.bin`, and no XML is + written. It is infrastructure, not a test failure — `./jaitest` retries once + automatically. Do not go debugging the test that "failed". - **`timeout` does not exist on this macOS box.** Do not use it in scripts. - **Noisy test stderr.** Fixture runs log Vue/JS `PluginException`s from bundled plugins. Tests pass regardless, but real failures can be buried — check the @@ -139,6 +185,7 @@ formatting is idempotent for the formatter. - Verify claims by running commands; prefer empirical checks over docs. Several plan decisions came from testing assumptions that turned out false. +- Work in small increments: change one thing, run `./jaitest`, commit when green. - Keep the working tree clean — revert throwaway probes. - Generated sources (once Grammar-Kit is wired up) go in `src/main/gen`, are committed, and are **never hand-edited**; regenerate instead.