# AGENTS.md — Jai IntelliJ Plugin Orientation for agents working on this repo. Read this first. **Goal:** an IntelliJ plugin for the **Jai** programming language. **Hard rule:** do **not** research Jai online — public material is outdated and wrong. The only source of truth is the local install at `~/.local/jai` (`how_to/`, `modules/`, `examples/`). Researching *IntelliJ plugin development* online is fine and encouraged. --- ## Run anything with `./jaigradle` ```bash ./jaigradle test # unit tests ./jaigradle check # full verification ./jaigradle tasks # discover tasks ``` **Never call `./gradlew` directly.** There is no JDK on `PATH` and `JAVA_HOME` is unset in a fresh shell. `./jaigradle` resolves the JDK via mise, then delegates. It works from an environment with no Java at all. Do not try to fix this with `org.gradle.java.home` in `gradle.properties` — already tried, it does not work, because `gradlew` is a shell script that needs a JVM to launch *itself* before it ever reads that property. **Java must be 21+.** The IntelliJ Platform test-framework jars are Java 21 bytecode (class major version 65); on JDK 17 every test fails with `UnsupportedClassVersionError`. `mise.toml` pins `java = "temurin-21"`. Note that `./jaigradle compileKotlin` **passes on JDK 17** — the template sources never touch the test framework. Compiling is not evidence the toolchain is correct. Run tests. --- ## Verify with the JUnit XML, not the exit code A green `test` task does not prove tests ran. Use the committed wrapper: ```bash ./jaitest # whole suite, prints per-suite JUnit XML numbers ./jaitest --tests '*Lexer*' # one class while iterating ``` It fails when the total test count is 0, so a silent no-op run cannot look green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and look for `tests="N"` with `failures="0" errors="0"` and **N > 0**. Last verified state (all green, `./jaigradle check` and `verifyPlugin` too): ```text dev.hgh.HarnessSmokeTest tests=2 dev.hgh.jai.JaiFileTypeTest tests=3 dev.hgh.jai.editor.JaiEditorSupportTest tests=5 dev.hgh.jai.highlighting.* tests=8 dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate dev.hgh.jai.lexer.JaiLexerTest tests=13 -> total 33, failures+errors 0 ``` The corpus gate reports what it actually did; check the line is still there: ```text Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly. ``` --- ## Current state ### Done - `docs/JAI_LANGUAGE_REFERENCE.md` — language reference derived from `~/.local/jai`, keywords/tokens transcribed from the compiler's own lexer. **Read this before writing any lexer or parser code.** - `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions. - Local IntelliJ reference projects and documentation in `~/programming/thirdparty`: - `fortran-plugin` — a simple language plugin to use as reference. - `intellij-elixir` — a more complex plugin to use as reference. - `intellij-sdk-docs` — local versions of the IntelliJ SDK documentation. - `mise.toml` → `java = "temurin-21"` (was 17, which was broken). - `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified. - `jaitest` — runs the suite and reports the JUnit XML (see above). - `src/test/kotlin/dev/hgh/HarnessSmokeTest.kt` — proves the platform boots headlessly. **If this fails, no other test can be trusted.** - **Phase 0** — git repo, template stripped, `plugin.xml` rewritten. - **Phase 1** — `JaiLanguage`, `JaiFileType` (+ icon), `JaiTokenTypes`, hand-written `JaiLexer`. Tier 0 corpus gate and Tier 1 golden tests green. - **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement), colour settings page, commenter, brace matcher. Plugin Verifier passes against IU-253/261/262. ### Lexer design facts worth knowing before touching it - **State is always 0.** Nested block comments and here-strings are each consumed inside a *single* token, so no context crosses a token boundary and the lexer can restart anywhere. Do not add lexer states without re-checking incremental re-highlighting. - **Deliberate divergences from the compiler's lexer**, all documented in `JaiTokenTypes`' KDoc: `#ident` is one DIRECTIVE token; `#string ... ID` is one HERE_STRING token; `::` and `:=` are single tokens; the backtick is its own token. - **Built-in type names and `it`/`it_index` are IDENT**, refined into separate token types only by `JaiHighlightingLexer`, which the parser never sees. Do not reserve them (language reference §14.9). - Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build. ### Not done — pick up here 1. **Phase 3: parser.** `.bnf` grammar → Grammar-Kit generated parser + PSI in `src/main/gen`, plus a `ParserDefinition`. Gate: Tier 3 corpus parse with zero `PsiErrorElement` (allowlist committed and shrinking). Note `JaiFileTypeTest` documents the one thing blocked on this: until a `ParserDefinition` exists, PSI files for `.jai` are plain text, so PSI-dependent features (comment action, structure view) cannot be tested. 2. **Phases 4+** — see the plan. ### Open questions for the user (unanswered) Scope (Phases 0–4 vs 0–7), compiler integration, target IDEs, JDK policy. See `docs/BUILD_PLAN.md` §6. Scope is the one that most affects the work. --- ## Architecture decisions already made Both are justified in `docs/BUILD_PLAN.md` §2. Do not silently reverse them. - **Hand-written lexer, not JFlex.** Jai here-strings (`#string DONE … DONE`) have a *runtime-captured* terminator, which a DFA generator cannot express. Nested block comments need a depth counter. Both would end up as hand-written Java inside JFlex actions anyway. - **Grammar-Kit BNF for parser + PSI.** Verified headless: the plugin ID is `org.jetbrains.intellij.platform.grammarkit`, applied in `build.gradle.kts`. `generateLexer` / `generateParser` tasks are present in `./jaigradle tasks`. - **Ship highlighting before the parser.** Highlighting needs only the lexer. --- ## The test corpus is the main asset `~/.local/jai` holds **714 `.jai` files, 16.4 MB** of real code by the language authors. Use it as the primary correctness gate instead of hand-written samples. The highest-value assertion, per the plan: lex every file and check the concatenation of all token texts reproduces the source **byte for byte**. An IntelliJ lexer must tile its input with no gaps or overlaps, so this single invariant over 16 MB catches nearly every lexer bug — with no fixture boot and no human review. Also assert no `BAD_CHARACTER` and that every `advance()` strictly increases the offset (catches infinite loops, which hang the IDE rather than failing visibly). This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files, 17.1 M chars, 3.0 M tokens, in under a second. `JaiSyntaxHighlighterTest` runs the same sweep to assert every token type the corpus produces (105 of them) has a colour. Keep new lexer work under these gates rather than adding snippets. Same idea later: zero `PsiErrorElement` across the corpus for the parser; formatting is idempotent for the formatter. --- ## Environment gotchas - **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused `Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`. - **Concurrent Gradle runs corrupt the test results.** If something else (an IDE, an agent's background checker) runs `test` at the same time, one of the two dies with `java.io.EOFException` or `NoSuchFileException: .../in-progress-results-generic.bin`, and no XML is written. It is infrastructure, not a test failure — `./jaitest` retries once automatically. Do not go debugging the test that "failed". - **A poisoned build cache can make tests silently vanish.** Symptom: `BUILD SUCCESSFUL`, `:compileTestKotlin FROM-CACHE`, `:test NO-SOURCE`, and `build/classes/kotlin/test` is empty. The cache stored an empty output directory from a build whose outputs were deleted underneath it, and the entry is keyed by input hash so it never gets replaced. This is the exact failure mode `./jaitest`'s "total tests = 0 is a failure" rule exists to catch. Recover with: ```bash ./jaigradle --stop rm -rf build .gradle ~/.gradle/caches/build-cache-1 ./jaitest ``` - **`timeout` does not exist on this macOS box.** Do not use it in scripts. - **Noisy test stderr.** Fixture runs log Vue/JS `PluginException`s from bundled plugins. Tests pass regardless, but real failures can be buried — check the XML. Narrowing the test fixture later would help. - **`runIde` is never required.** Everything is verifiable headlessly; that is a hard project requirement, not a preference. Do not ask the user to open an IDE to check your work. --- ## Working agreement - Verify claims by running commands; prefer empirical checks over docs. Several plan decisions came from testing assumptions that turned out false. - Work in small increments: change one thing, run `./jaitest`, commit when green. - Keep the working tree clean — revert throwaway probes. - Generated sources (once Grammar-Kit is wired up) go in `src/main/gen`, are committed, and are **never hand-edited**; regenerate instead. - Update this file when project state changes.