# AGENTS.md — Jai IntelliJ Plugin Orientation for agents working on this repo. Read this first. **Goal:** an IntelliJ plugin for the **Jai** programming language. **Hard rule:** do **not** research Jai online — public material is outdated and wrong. The only source of truth is the local install at `~/.local/jai` (`how_to/`, `modules/`, `examples/`). Researching *IntelliJ plugin development* online is fine and encouraged. --- ## Run anything with `./jaigradle` ```bash ./jaigradle test # unit tests ./jaigradle check # full verification ./jaigradle tasks # discover tasks ``` **Never call `./gradlew` directly.** There is no JDK on `PATH` and `JAVA_HOME` is unset in a fresh shell. `./jaigradle` resolves the JDK via mise, then delegates. It works from an environment with no Java at all. Do not try to fix this with `org.gradle.java.home` in `gradle.properties` — already tried, it does not work, because `gradlew` is a shell script that needs a JVM to launch *itself* before it ever reads that property. **Java must be 21+.** The IntelliJ Platform test-framework jars are Java 21 bytecode (class major version 65); on JDK 17 every test fails with `UnsupportedClassVersionError`. `mise.toml` pins `java = "temurin-21"`. Note that `./jaigradle compileKotlin` **passes on JDK 17** — the template sources never touch the test framework. Compiling is not evidence the toolchain is correct. Run tests. --- ## Verify with the JUnit XML, not the exit code A green `test` task does not prove tests ran. Use the committed wrapper: ```bash ./jaitest # whole suite, prints per-suite JUnit XML numbers ./jaitest --tests '*Lexer*' # one class while iterating ``` It fails when the total test count is 0, so a silent no-op run cannot look green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and look for `tests="N"` with `failures="0" errors="0"` and **N > 0**. **A `--tests` filter sticks.** Gradle reuses the configuration cache entry from the previous run, so a later unfiltered `./jaitest` silently re-runs only that one class and still prints `OK`. `jaitest` therefore passes `--no-configuration-cache` whenever no `--tests` argument is given. Do not "optimise" that away. Last verified state (all green, `./jaigradle check` and `verifyPlugin` too): ```text dev.hgh.HarnessSmokeTest tests=2 dev.hgh.jai.JaiFileTypeTest tests=4 dev.hgh.jai.completion.JaiCompletionTest tests=20 dev.hgh.jai.editor.JaiEditorSupportTest tests=7 dev.hgh.jai.editor.JaiFoldingBuilderTest tests=3 dev.hgh.jai.findusages.JaiFindUsagesTest tests=2 dev.hgh.jai.highlighting.* tests=8 dev.hgh.jai.inspection.JaiUnresolvedModuleInspectionTest tests=5 dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate dev.hgh.jai.lexer.JaiLexerTest tests=13 dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees dev.hgh.jai.parser.JaiParserLongTailTest tests=1 <- focused parser regressions dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert dev.hgh.jai.refactoring.JaiRenameTest tests=3 dev.hgh.jai.reference.JaiReferenceTest tests=10 dev.hgh.jai.structure.JaiStructureViewTest tests=3 dev.hgh.jai.formatter.JaiFormatterTest tests=4 dev.hgh.jai.settings.JaiProjectSettingsTest tests=3 dev.hgh.jai.settings.JaiConfiguredRootTest tests=6 -> total 104, failures+errors 0 ``` The corpus gates report what they actually did; check both lines are still there: ```text Tier 0: lexed 714 files, 17154195 chars, 3008777 tokens cleanly. Tier 3: parsed 714 files, 714 clean (100.0%), 0 with errors, 0 PsiErrorElements total. ``` --- ## Current state ### Done - `docs/JAI_LANGUAGE_REFERENCE.md` — language reference derived from `~/.local/jai`, keywords/tokens transcribed from the compiler's own lexer. **Read this before writing any lexer or parser code.** - `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions. - Local IntelliJ reference projects and documentation in `~/programming/thirdparty`: - `fortran-plugin` — a simple language plugin to use as reference. - `intellij-elixir` — a more complex plugin to use as reference. - `intellij-sdk-docs` — local versions of the IntelliJ SDK documentation. - `mise.toml` → `java = "temurin-21"` (was 17, which was broken). - `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified. - `jaitest` — runs the suite and reports the JUnit XML (see above). - `src/test/kotlin/dev/hgh/HarnessSmokeTest.kt` — proves the platform boots headlessly. **If this fails, no other test can be trusted.** - **Phase 0** — git repo, template stripped, `plugin.xml` rewritten. - **Phase 1** — `JaiLanguage`, `JaiFileType` (+ icon), `JaiTokenTypes`, hand-written `JaiLexer`. Tier 0 corpus gate and Tier 1 golden tests green. - **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement), colour settings page, commenter, brace matcher. Plugin Verifier passes against IU-253/261/262. - **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in `src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and the Tier 3 corpus gate are green at **100.0% (714/714 files parse with zero `PsiErrorElement`)**. - **Phase 4** — structure view, folding for blocks/comments/here-strings, and `#import`/`#load` path references with standard go-to-definition resolution. Headless tests cover source-ordered declarations, nested structure members, folding ranges, local `#load`, and Jai module directories. - **Phase 5** — completion for keywords, directives, module/file paths, visible same-file or imported declarations, procedure parameters, and module aliases; same-file and imported symbol navigation; rename; and find-usages for declaration names. Struct-field completion remains future work. - **Phase 5a** — persistent project settings for Jai module/import roots, shared root-aware resolution and completion, and indexed external Jai library sources. Headless tests cover state round-tripping, the Settings panel, custom-root `#import`/`#load` navigation and completion, and cross-root find-usages. - **Phase 6** — PSI-aware formatter with operator/punctuation spacing, block indentation, opaque `#asm` preservation, and a corpus-wide idempotence gate. Headless formatter tests cover registration, representative formatting, opaque/directive token preservation, and all 714 corpus files. - **Phase 7a** — unresolved `#import`/`#load` module and file path inspection, reusing the existing reference resolver. Headless tests cover unresolved imports and loads, resolved project/local modules and files, and `#import,string` exclusions. ### Lexer design facts worth knowing before touching it - **State is always 0.** Nested block comments and here-strings are each consumed inside a *single* token, so no context crosses a token boundary and the lexer can restart anywhere. Do not add lexer states without re-checking incremental re-highlighting. - **Deliberate divergences from the compiler's lexer**, all documented in `JaiTokenTypes`' KDoc: `#ident` is one DIRECTIVE token; `#string ... ID` is one HERE_STRING token; `::` and `:=` are single tokens; the backtick is its own token. - **Built-in type names and `it`/`it_index` are IDENT**, refined into separate token types only by `JaiHighlightingLexer`, which the parser never sees. Do not reserve them (language reference §14.9). - **A backslash inside an identifier is a continuation.** The compiler's lexer (`Jai_Lexer/module.jai:444`) eats the `\` and any spaces after it and keeps appending, so `left\_margin` is one identifier. The token span still covers the backslash, so the Tier 0 round-trip holds. - **`#string,\% ID` exists** in `how_to/018_print_functions.jai` even though the shipped `Jai_Lexer` module rejects anything but `,cr`. Our `scanHereString` accepts any non-space modifier run. When the module and real code disagree, real code wins. - Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build. ### Parser design facts worth knowing before touching it - **Generated sources are wiped and regenerated on every build.** `compileJava` and `compileKotlin` depend on `generateParser`, and a `Delete` task clears `src/main/gen` first. Editing `Jai.bnf` is enough; never run a bare compile and trust it. Both halves of that were real bugs: - without the dependency, a `Jai.bnf` edit compiled against the *previous* parser and the tests reported green (the `.class` was minutes older than its source); - without the wipe, Grammar-Kit's partial regeneration left `JaiDeclaration.getArgumentList()` with no override in the Impl and the build failed inside generated code. - **Grammar-Kit mints its own token instances** from the `tokens` block. Those are different objects from the ones `JaiLexer` emits, so every rule silently fails to match and a whole file becomes one error element. `tokenTypeFactory` points at `JaiTokenTypes.byName`, which resolves them reflectively. - **What BNF cannot express lives in `JaiParserUtil`**: directive-name tests (`#ident` is one token), `==` before `{` for the switch form, procedure header vs parenthesised expression, and the declaration lookahead — Jai declarations have no introducer keyword. - **A rule that creates a PSI element inside an expression can truncate it.** Using `initializer` for a named return default made the enclosing `procLiteralExpr` end early, so `-> a: int = 1 { }` silently lost its body. The private `returnDefault_` rule exists for that reason — do not "simplify" it. - **Longest-first matters for multi-token sequences only.** `operator *[]` needs `'*' '[' ']'` before `'*'`. Single-token operators cannot collide, because `+=` is one token. - **Directive flags require adjacency.** `#library,system` is a flag; `(cb: (*GUID) #c_call, ctx: *void)` is a parameter separator. Without the adjacency check the flag swallowed `, ctx`. - **Contributed references need a PSI host.** Generated AST wrappers do not implement `ContributedReferenceHost`, so `literalExpr` uses the `JaiReferenceHostMixin` from the BNF. Do not remove the mixin and expect a `PsiReferenceContributor` to be queried through the standard reference service. ### Known code-insight gaps - Completion currently offers keywords, compiler directives, module/file paths, visible same-file or imported declarations, procedure parameters, and members of imported aliases, but not struct fields. - Struct fields do not yet have symbol resolution or completion. ### In progress — pick up here 1. **Phases 7+** — additional inspections, quick fixes, live templates, and compiler integration remain. `#asm` bodies are intentionally consumed opaquely, so nothing inside them has PSI yet. ### Next planned increment - Expand Phase 7 with additional high-confidence inspections and quick fixes; struct-field resolution and completion remain a separate known gap. ### Open questions for the user (unanswered) Scope (Phases 0–5a vs 0–7), compiler integration, target IDEs, JDK policy. See `docs/BUILD_PLAN.md` §6. Scope is the one that most affects the work. --- ## Architecture decisions already made Both are justified in `docs/BUILD_PLAN.md` §2. Do not silently reverse them. - **Hand-written lexer, not JFlex.** Jai here-strings (`#string DONE … DONE`) have a *runtime-captured* terminator, which a DFA generator cannot express. Nested block comments need a depth counter. Both would end up as hand-written Java inside JFlex actions anyway. - **Grammar-Kit BNF for parser + PSI.** Verified headless: the plugin ID is `org.jetbrains.intellij.platform.grammarkit`, applied in `build.gradle.kts`. `generateLexer` / `generateParser` tasks are present in `./jaigradle tasks`. - **Ship highlighting before the parser.** Highlighting needs only the lexer. --- ## The test corpus is the main asset `~/.local/jai` holds **714 `.jai` files, 16.4 MB** of real code by the language authors. Use it as the primary correctness gate instead of hand-written samples. The highest-value assertion, per the plan: lex every file and check the concatenation of all token texts reproduces the source **byte for byte**. An IntelliJ lexer must tile its input with no gaps or overlaps, so this single invariant over 16 MB catches nearly every lexer bug — with no fixture boot and no human review. Also assert no `BAD_CHARACTER` and that every `advance()` strictly increases the offset (catches infinite loops, which hang the IDE rather than failing visibly). This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files, 17.1 M chars, 3.0 M tokens, in under a second. `JaiSyntaxHighlighterTest` runs the same sweep to assert every token type the corpus produces (105 of them) has a colour. Keep new lexer work under these gates rather than adding snippets. Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all 714 files and counts `PsiErrorElement`s — currently **714 clean (100.0%)**. The ratchet is set to the full corpus size, so any parser regression fails the test. `#asm` bodies remain opaque by design; they are tiled as one directive body but have no inner PSI. The idempotence sweep for the formatter is the same idea again, later. The fast loop for grammar work: 1. `./jaitest`, then read `JaiCorpusParserTest`'s histogram and sample errors. 2. Reduce one failure to a snippet in `DebugParseTest` (inert by default). 3. Fix `Jai.bnf` or `JaiParserUtil`, re-run, watch the percentage. 4. Raise `MIN_CLEAN_FILES` and commit. --- ## Environment gotchas - **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused `Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`. - **Stale build outputs cost hours.** Symptoms: a `Jai.bnf` edit has no effect, or the build fails inside generated code complaining about a method that is not in the file you are reading. Check timestamps — `ls -l build/classes/java/main/dev/hgh/jai/parser/JaiParser.class src/main/gen/dev/hgh/jai/parser/JaiParser.java`. The build wiring in `build.gradle.kts` should prevent both, but `rm -rf build` settles it. - **Concurrent Gradle runs corrupt the test results.** `./jaigradle` and `./jaitest` serialize project Gradle work with the atomic `.gradle-test.lock` lock, including test-result cleanup and retries. Never run `./gradlew` directly: it bypasses both the JDK wrapper and this lock. If an interrupted process leaves a lock behind, confirm no Gradle run is active, then remove `.gradle-test.lock` and rerun. The JUnit wrapper still retries once for unexpected no-XML failures; do not debug a test until the XML exists. - **A poisoned build cache can make tests silently vanish.** Symptom: `BUILD SUCCESSFUL`, `:compileTestKotlin FROM-CACHE`, `:test NO-SOURCE`, and `build/classes/kotlin/test` is empty. The cache stored an empty output directory from a build whose outputs were deleted underneath it, and the entry is keyed by input hash so it never gets replaced. This is the exact failure mode `./jaitest`'s "total tests = 0 is a failure" rule exists to catch. Recover with: ```bash ./jaigradle --stop rm -rf .gradle-test.lock build .gradle ~/.gradle/caches/build-cache-1 ./jaitest ``` - **`Cannot create child file 'x' at /src` is a flake.** The fixture's in-memory filesystem occasionally refuses `myFixture.configureByText`. It is not a real failure — re-run. `JaiCorpusParserTest` avoids it entirely by going through `PsiFileFactory.createFileFromText` instead of the fixture, which also makes the 714-file sweep much faster. - **`timeout` does not exist on this macOS box.** Do not use it in scripts. - **Noisy test stderr.** Fixture runs log Vue/JS `PluginException`s from bundled plugins. Tests pass regardless, but real failures can be buried — check the XML. Narrowing the test fixture later would help. - **`runIde` is never required.** Everything is verifiable headlessly; that is a hard project requirement, not a preference. Do not ask the user to open an IDE to check your work. --- ## Working agreement - Verify claims by running commands; prefer empirical checks over docs. Several plan decisions came from testing assumptions that turned out false. - Work in small increments: change one thing, run `./jaitest`, commit when green. - Keep the working tree clean — revert throwaway probes. - Generated sources go in `src/main/gen`, are committed, and are **never hand-edited**. They regenerate on every build from `src/main/grammar/Jai.bnf`. - Update this file when project state changes.