# AGENTS.md — Jai IntelliJ Plugin Orientation for agents working on this repo. Read this first. **Goal:** an IntelliJ plugin for the **Jai** programming language. **Hard rule:** do **not** research Jai online — public material is outdated and wrong. The only source of truth is the local install at `~/.local/jai` (`how_to/`, `modules/`, `examples/`). Researching *IntelliJ plugin development* online is fine and encouraged. --- ## Run anything with `./jaigradle` ```bash ./jaigradle test # unit tests ./jaigradle check # full verification ./jaigradle tasks # discover tasks ``` **Never call `./gradlew` directly.** There is no JDK on `PATH` and `JAVA_HOME` is unset in a fresh shell. `./jaigradle` resolves the JDK via mise, then delegates. It works from an environment with no Java at all. Do not try to fix this with `org.gradle.java.home` in `gradle.properties` — already tried, it does not work, because `gradlew` is a shell script that needs a JVM to launch *itself* before it ever reads that property. **Java must be 21+.** The IntelliJ Platform test-framework jars are Java 21 bytecode (class major version 65); on JDK 17 every test fails with `UnsupportedClassVersionError`. `mise.toml` pins `java = "temurin-21"`. Note that `./jaigradle compileKotlin` **passes on JDK 17** — the template sources never touch the test framework. Compiling is not evidence the toolchain is correct. Run tests. --- ## Verify with the JUnit XML, not the exit code A green `test` task does not prove tests ran. Use the committed wrapper: ```bash ./jaitest # whole suite, prints per-suite JUnit XML numbers ./jaitest --tests '*Lexer*' # one class while iterating ``` It fails when the total test count is 0, so a silent no-op run cannot look green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and look for `tests="N"` with `failures="0" errors="0"` and **N > 0**. **A `--tests` filter sticks.** Gradle reuses the configuration cache entry from the previous run, so a later unfiltered `./jaitest` silently re-runs only that one class and still prints `OK`. `jaitest` therefore passes `--no-configuration-cache` whenever no `--tests` argument is given. Do not "optimise" that away. Last verified state (all green, `./jaigradle check` and `verifyPlugin` too): ```text dev.hgh.HarnessSmokeTest tests=2 dev.hgh.jai.JaiFileTypeTest tests=4 dev.hgh.jai.editor.JaiEditorSupportTest tests=5 dev.hgh.jai.highlighting.* tests=8 dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate dev.hgh.jai.lexer.JaiLexerTest tests=13 dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert -> total 42, failures+errors 0 ``` The corpus gates report what they actually did; check both lines are still there: ```text Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly. Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements total. ``` --- ## Current state ### Done - `docs/JAI_LANGUAGE_REFERENCE.md` — language reference derived from `~/.local/jai`, keywords/tokens transcribed from the compiler's own lexer. **Read this before writing any lexer or parser code.** - `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions. - Local IntelliJ reference projects and documentation in `~/programming/thirdparty`: - `fortran-plugin` — a simple language plugin to use as reference. - `intellij-elixir` — a more complex plugin to use as reference. - `intellij-sdk-docs` — local versions of the IntelliJ SDK documentation. - `mise.toml` → `java = "temurin-21"` (was 17, which was broken). - `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified. - `jaitest` — runs the suite and reports the JUnit XML (see above). - `src/test/kotlin/dev/hgh/HarnessSmokeTest.kt` — proves the platform boots headlessly. **If this fails, no other test can be trusted.** - **Phase 0** — git repo, template stripped, `plugin.xml` rewritten. - **Phase 1** — `JaiLanguage`, `JaiFileType` (+ icon), `JaiTokenTypes`, hand-written `JaiLexer`. Tier 0 corpus gate and Tier 1 golden tests green. - **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement), colour settings page, commenter, brace matcher. Plugin Verifier passes against IU-253/261/262. - **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in `src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and the Tier 3 corpus gate are green at **91.9% (656/714 files parse with zero `PsiErrorElement`)**. ### Lexer design facts worth knowing before touching it - **State is always 0.** Nested block comments and here-strings are each consumed inside a *single* token, so no context crosses a token boundary and the lexer can restart anywhere. Do not add lexer states without re-checking incremental re-highlighting. - **Deliberate divergences from the compiler's lexer**, all documented in `JaiTokenTypes`' KDoc: `#ident` is one DIRECTIVE token; `#string ... ID` is one HERE_STRING token; `::` and `:=` are single tokens; the backtick is its own token. - **Built-in type names and `it`/`it_index` are IDENT**, refined into separate token types only by `JaiHighlightingLexer`, which the parser never sees. Do not reserve them (language reference §14.9). - **A backslash inside an identifier is a continuation.** The compiler's lexer (`Jai_Lexer/module.jai:444`) eats the `\` and any spaces after it and keeps appending, so `left\_margin` is one identifier. The token span still covers the backslash, so the Tier 0 round-trip holds. - **`#string,\% ID` exists** in `how_to/018_print_functions.jai` even though the shipped `Jai_Lexer` module rejects anything but `,cr`. Our `scanHereString` accepts any non-space modifier run. When the module and real code disagree, real code wins. - Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build. ### Parser design facts worth knowing before touching it - **Generated sources are wiped and regenerated on every build.** `compileJava` and `compileKotlin` depend on `generateParser`, and a `Delete` task clears `src/main/gen` first. Editing `Jai.bnf` is enough; never run a bare compile and trust it. Both halves of that were real bugs: - without the dependency, a `Jai.bnf` edit compiled against the *previous* parser and the tests reported green (the `.class` was minutes older than its source); - without the wipe, Grammar-Kit's partial regeneration left `JaiDeclaration.getArgumentList()` with no override in the Impl and the build failed inside generated code. - **Grammar-Kit mints its own token instances** from the `tokens` block. Those are different objects from the ones `JaiLexer` emits, so every rule silently fails to match and a whole file becomes one error element. `tokenTypeFactory` points at `JaiTokenTypes.byName`, which resolves them reflectively. - **What BNF cannot express lives in `JaiParserUtil`**: directive-name tests (`#ident` is one token), `==` before `{` for the switch form, procedure header vs parenthesised expression, and the declaration lookahead — Jai declarations have no introducer keyword. - **A rule that creates a PSI element inside an expression can truncate it.** Using `initializer` for a named return default made the enclosing `procLiteralExpr` end early, so `-> a: int = 1 { }` silently lost its body. The private `returnDefault_` rule exists for that reason — do not "simplify" it. - **Longest-first matters for multi-token sequences only.** `operator *[]` needs `'*' '[' ']'` before `'*'`. Single-token operators cannot collide, because `+=` is one token. - **Directive flags require adjacency.** `#library,system` is a flag; `(cb: (*GUID) #c_call, ctx: *void)` is a parameter separator. Without the adjacency check the flag swallowed `, ctx`. ### Not done — pick up here 1. **Finish the Tier 3 corpus gate.** 58 of 714 files still produce a `PsiErrorElement`, all long-tail: each construct below appears in one or two files. `JaiCorpusParserTest` prints a histogram of the token at each failure point plus a spread of sample errors; that output is the work list. `MIN_CLEAN_FILES` is the ratchet — raise it, never lower it. Known gaps: - mixed declare/assign target lists: `success=, output, error := adb(…)` and `renderer:, success = make_renderer(…)`; - `using,except SKIP_THESE name: T;` (a bare identifier operand, where `using,except(x)` and `using,except .["x"]` already work); - `#asm` bodies are consumed opaquely, so nothing inside them has PSI. Decide explicitly whether the remainder becomes a committed allowlist (the plan's §3 suggestion) or gets fixed. 2. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution and go-to-definition. The PSI is in place, so these are now unblocked. `JaiFileTypeTest` used to document being blocked on the `ParserDefinition`; that no longer applies. 3. **Phases 5+** — see the plan. ### Open questions for the user (unanswered) Scope (Phases 0–4 vs 0–7), compiler integration, target IDEs, JDK policy. See `docs/BUILD_PLAN.md` §6. Scope is the one that most affects the work. --- ## Architecture decisions already made Both are justified in `docs/BUILD_PLAN.md` §2. Do not silently reverse them. - **Hand-written lexer, not JFlex.** Jai here-strings (`#string DONE … DONE`) have a *runtime-captured* terminator, which a DFA generator cannot express. Nested block comments need a depth counter. Both would end up as hand-written Java inside JFlex actions anyway. - **Grammar-Kit BNF for parser + PSI.** Verified headless: the plugin ID is `org.jetbrains.intellij.platform.grammarkit`, applied in `build.gradle.kts`. `generateLexer` / `generateParser` tasks are present in `./jaigradle tasks`. - **Ship highlighting before the parser.** Highlighting needs only the lexer. --- ## The test corpus is the main asset `~/.local/jai` holds **714 `.jai` files, 16.4 MB** of real code by the language authors. Use it as the primary correctness gate instead of hand-written samples. The highest-value assertion, per the plan: lex every file and check the concatenation of all token texts reproduces the source **byte for byte**. An IntelliJ lexer must tile its input with no gaps or overlaps, so this single invariant over 16 MB catches nearly every lexer bug — with no fixture boot and no human review. Also assert no `BAD_CHARACTER` and that every `advance()` strictly increases the offset (catches infinite loops, which hang the IDE rather than failing visibly). This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files, 17.1 M chars, 3.0 M tokens, in under a second. `JaiSyntaxHighlighterTest` runs the same sweep to assert every token type the corpus produces (105 of them) has a colour. Keep new lexer work under these gates rather than adding snippets. Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all 714 files and counts `PsiErrorElement`s — currently **656 clean (91.9%)**. It prints a histogram of the source text at each failure point, which is the fastest way to find the next construct worth supporting. The idempotence sweep for the formatter is the same idea again, later. The fast loop for grammar work: 1. `./jaitest`, then read `JaiCorpusParserTest`'s histogram and sample errors. 2. Reduce one failure to a snippet in `DebugParseTest` (inert by default). 3. Fix `Jai.bnf` or `JaiParserUtil`, re-run, watch the percentage. 4. Raise `MIN_CLEAN_FILES` and commit. --- ## Environment gotchas - **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused `Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`. - **Stale build outputs cost hours.** Symptoms: a `Jai.bnf` edit has no effect, or the build fails inside generated code complaining about a method that is not in the file you are reading. Check timestamps — `ls -l build/classes/java/main/dev/hgh/jai/parser/JaiParser.class src/main/gen/dev/hgh/jai/parser/JaiParser.java`. The build wiring in `build.gradle.kts` should prevent both, but `rm -rf build` settles it. - **Concurrent Gradle runs corrupt the test results.** If something else (an IDE, an agent's background checker) runs `test` at the same time, one of the two dies with `java.io.EOFException` or `NoSuchFileException: .../in-progress-results-generic.bin`, and no XML is written. It is infrastructure, not a test failure — `./jaitest` retries once automatically. Do not go debugging the test that "failed". - **A poisoned build cache can make tests silently vanish.** Symptom: `BUILD SUCCESSFUL`, `:compileTestKotlin FROM-CACHE`, `:test NO-SOURCE`, and `build/classes/kotlin/test` is empty. The cache stored an empty output directory from a build whose outputs were deleted underneath it, and the entry is keyed by input hash so it never gets replaced. This is the exact failure mode `./jaitest`'s "total tests = 0 is a failure" rule exists to catch. Recover with: ```bash ./jaigradle --stop rm -rf build .gradle ~/.gradle/caches/build-cache-1 ./jaitest ``` - **`Cannot create child file 'x' at /src` is a flake.** The fixture's in-memory filesystem occasionally refuses `myFixture.configureByText`. It is not a real failure — re-run. `JaiCorpusParserTest` avoids it entirely by going through `PsiFileFactory.createFileFromText` instead of the fixture, which also makes the 714-file sweep much faster. - **`timeout` does not exist on this macOS box.** Do not use it in scripts. - **Noisy test stderr.** Fixture runs log Vue/JS `PluginException`s from bundled plugins. Tests pass regardless, but real failures can be buried — check the XML. Narrowing the test fixture later would help. - **`runIde` is never required.** Everything is verifiable headlessly; that is a hard project requirement, not a preference. Do not ask the user to open an IDE to check your work. --- ## Working agreement - Verify claims by running commands; prefer empirical checks over docs. Several plan decisions came from testing assumptions that turned out false. - Work in small increments: change one thing, run `./jaitest`, commit when green. - Keep the working tree clean — revert throwaway probes. - Generated sources go in `src/main/gen`, are committed, and are **never hand-edited**. They regenerate on every build from `src/main/grammar/Jai.bnf`. - Update this file when project state changes.