intellijai/AGENTS.md

AGENTS.md — Jai IntelliJ Plugin

Orientation for agents working on this repo. Read this first.

Goal: an IntelliJ plugin for the Jai programming language.

Hard rule: do not research Jai online — public material is outdated and wrong. The only source of truth is the local install at ~/.local/jai (how_to/, modules/, examples/). Researching IntelliJ plugin development online is fine and encouraged.

---

Run anything with ./jaigradle

./jaigradle test     # unit tests
./jaigradle check    # full verification
./jaigradle tasks    # discover tasks

Never call ./gradlew directly. There is no JDK on PATH and JAVA_HOME is unset in a fresh shell. ./jaigradle resolves the JDK via mise, then delegates. It works from an environment with no Java at all.

Do not try to fix this with org.gradle.java.home in gradle.properties — already tried, it does not work, because gradlew is a shell script that needs a JVM to launch itself before it ever reads that property.

Java must be 21+. The IntelliJ Platform test-framework jars are Java 21 bytecode (class major version 65); on JDK 17 every test fails with UnsupportedClassVersionError. mise.toml pins java = "temurin-21".

Note that ./jaigradle compileKotlin passes on JDK 17 — the template sources never touch the test framework. Compiling is not evidence the toolchain is correct. Run tests.

---

Verify with the JUnit XML, not the exit code

A green test task does not prove tests ran. Use the committed wrapper:

./jaitest                      # whole suite, prints per-suite JUnit XML numbers
./jaitest --tests '*Lexer*'    # one class while iterating

It fails when the total test count is 0, so a silent no-op run cannot look green. Doing it by hand instead: sed -n 2p build/test-results/test/*.xml and look for tests="N" with failures="0" errors="0" and N > 0.

A --tests filter sticks. Gradle reuses the configuration cache entry from the previous run, so a later unfiltered ./jaitest silently re-runs only that one class and still prints OK. jaitest therefore passes --no-configuration-cache whenever no --tests argument is given. Do not "optimise" that away.

Last verified state (all green, ./jaigradle check and verifyPlugin too):

dev.hgh.HarnessSmokeTest                       tests=2
dev.hgh.jai.JaiFileTypeTest                    tests=4
dev.hgh.jai.completion.JaiCompletionTest      tests=20
dev.hgh.jai.editor.JaiEditorSupportTest       tests=7
dev.hgh.jai.editor.JaiFoldingBuilderTest      tests=3
dev.hgh.jai.findusages.JaiFindUsagesTest      tests=2
dev.hgh.jai.highlighting.*                    tests=8
dev.hgh.jai.lexer.JaiCorpusLexerTest           tests=2   <- the Tier 0 gate
dev.hgh.jai.lexer.JaiLexerTest                 tests=13
dev.hgh.jai.parser.JaiCorpusParserTest         tests=1   <- the Tier 3 gate
dev.hgh.jai.parser.JaiParserGoldenTest         tests=5   <- Tier 2 golden trees
dev.hgh.jai.parser.JaiParserLongTailTest       tests=1   <- focused parser regressions
dev.hgh.jai.parser.DebugParseTest              tests=2   <- scratch harness, inert
dev.hgh.jai.refactoring.JaiRenameTest          tests=3
dev.hgh.jai.reference.JaiReferenceTest         tests=10
dev.hgh.jai.structure.JaiStructureViewTest     tests=3
dev.hgh.jai.formatter.JaiFormatterTest          tests=4
dev.hgh.jai.settings.JaiProjectSettingsTest    tests=3
dev.hgh.jai.settings.JaiConfiguredRootTest    tests=6
-> total 99, failures+errors 0

The corpus gates report what they actually did; check both lines are still there:

Tier 0: lexed 714 files, 17154195 chars, 3008777 tokens cleanly.
Tier 3: parsed 714 files, 714 clean (100.0%), 0 with errors, 0 PsiErrorElements total.

---

Current state

Done

• docs/JAI_LANGUAGE_REFERENCE.md — language reference derived from ~/.local/jai, keywords/tokens transcribed from the compiler's own lexer. Read this before writing any lexer or parser code.
• docs/BUILD_PLAN.md — phased plan, testing strategy, risks, open questions.
• Local IntelliJ reference projects and documentation in ~/programming/thirdparty:
  • fortran-plugin — a simple language plugin to use as reference.
  • intellij-elixir — a more complex plugin to use as reference.
  • intellij-sdk-docs — local versions of the IntelliJ SDK documentation.
• mise.toml → java = "temurin-21" (was 17, which was broken).
• jaigradle — JDK-resolving Gradle wrapper. Executable, verified.
• jaitest — runs the suite and reports the JUnit XML (see above).
• src/test/kotlin/dev/hgh/HarnessSmokeTest.kt — proves the platform boots headlessly. If this fails, no other test can be trusted.
• Phase 0 — git repo, template stripped, plugin.xml rewritten.
• Phase 1 — JaiLanguage, JaiFileType (+ icon), JaiTokenTypes, hand-written JaiLexer. Tier 0 corpus gate and Tier 1 golden tests green.
• Phase 2 — JaiSyntaxHighlighter (+ highlight-only lexer refinement), colour settings page, commenter, brace matcher. Plugin Verifier passes against IU-253/261/262.
• Phase 3 — src/main/grammar/Jai.bnf → Grammar-Kit parser and PSI in src/main/gen, JaiParserDefinition, JaiParserUtil. Tier 2 golden trees and the Tier 3 corpus gate are green at 100.0% (714/714 files parse with zero PsiErrorElement).
• Phase 4 — structure view, folding for blocks/comments/here-strings, and #import/#load path references with standard go-to-definition resolution. Headless tests cover source-ordered declarations, nested structure members, folding ranges, local #load, and Jai module directories.
• Phase 5 — completion for keywords, directives, module/file paths, visible same-file or imported declarations, procedure parameters, and module aliases; same-file and imported symbol navigation; rename; and find-usages for declaration names. Struct-field completion remains future work.
• Phase 5a — persistent project settings for Jai module/import roots, shared root-aware resolution and completion, and indexed external Jai library sources. Headless tests cover state round-tripping, the Settings panel, custom-root #import/#load navigation and completion, and cross-root find-usages.
• Phase 6 — PSI-aware formatter with operator/punctuation spacing, block indentation, opaque #asm preservation, and a corpus-wide idempotence gate. Headless formatter tests cover registration, representative formatting, opaque/directive token preservation, and all 714 corpus files.

Lexer design facts worth knowing before touching it

• State is always 0. Nested block comments and here-strings are each consumed inside a single token, so no context crosses a token boundary and the lexer can restart anywhere. Do not add lexer states without re-checking incremental re-highlighting.
• Deliberate divergences from the compiler's lexer, all documented in JaiTokenTypes' KDoc: #ident is one DIRECTIVE token; #string ... ID is one HERE_STRING token; :: and := are single tokens; the backtick is its own token.
• Built-in type names and it/it_index are IDENT, refined into separate token types only by JaiHighlightingLexer, which the parser never sees. Do not reserve them (language reference §14.9).
• A backslash inside an identifier is a continuation. The compiler's lexer (Jai_Lexer/module.jai:444) eats the \ and any spaces after it and keeps appending, so left\_margin is one identifier. The token span still covers the backslash, so the Tier 0 round-trip holds.
• #string,\% ID exists in how_to/018_print_functions.jai even though the shipped Jai_Lexer module rejects anything but ,cr. Our scanHereString accepts any non-space modifier run. When the module and real code disagree, real code wins.
• Kotlin block comments nest too: writing /* inside a KDoc breaks the build.

Parser design facts worth knowing before touching it

• Generated sources are wiped and regenerated on every build. compileJava and compileKotlin depend on generateParser, and a Delete task clears src/main/gen first. Editing Jai.bnf is enough; never run a bare compile and trust it. Both halves of that were real bugs:
  • without the dependency, a Jai.bnf edit compiled against the previous parser and the tests reported green (the .class was minutes older than its source);
  • without the wipe, Grammar-Kit's partial regeneration left JaiDeclaration.getArgumentList() with no override in the Impl and the build failed inside generated code.
• Grammar-Kit mints its own token instances from the tokens block. Those are different objects from the ones JaiLexer emits, so every rule silently fails to match and a whole file becomes one error element. tokenTypeFactory points at JaiTokenTypes.byName, which resolves them reflectively.
• What BNF cannot express lives in JaiParserUtil: directive-name tests (#ident is one token), == before { for the switch form, procedure header vs parenthesised expression, and the declaration lookahead — Jai declarations have no introducer keyword.
• A rule that creates a PSI element inside an expression can truncate it. Using initializer for a named return default made the enclosing procLiteralExpr end early, so -> a: int = 1 { } silently lost its body. The private returnDefault_ rule exists for that reason — do not "simplify" it.
• Longest-first matters for multi-token sequences only. operator *[] needs '*' '[' ']' before '*'. Single-token operators cannot collide, because += is one token.
• Directive flags require adjacency. #library,system is a flag; (cb: (*GUID) #c_call, ctx: *void) is a parameter separator. Without the adjacency check the flag swallowed , ctx.
• Contributed references need a PSI host. Generated AST wrappers do not implement ContributedReferenceHost, so literalExpr uses the JaiReferenceHostMixin from the BNF. Do not remove the mixin and expect a PsiReferenceContributor to be queried through the standard reference service.

Known code-insight gaps

• Completion currently offers keywords, compiler directives, module/file paths, visible same-file or imported declarations, procedure parameters, and members of imported aliases, but not struct fields.
• Struct fields do not yet have symbol resolution or completion.

In progress — pick up here

1. Phases 7+ — inspections, compiler integration, and other optional work remain. #asm bodies are intentionally consumed opaquely, so nothing inside them has PSI yet.

Next planned increment

• Decide whether to continue with inspections and compiler integration phases (7–8).

Open questions for the user (unanswered)

Scope (Phases 0–5a vs 0–7), compiler integration, target IDEs, JDK policy. See docs/BUILD_PLAN.md §6. Scope is the one that most affects the work.

---

Architecture decisions already made

Both are justified in docs/BUILD_PLAN.md §2. Do not silently reverse them.

• Hand-written lexer, not JFlex. Jai here-strings (#string DONE … DONE) have a runtime-captured terminator, which a DFA generator cannot express. Nested block comments need a depth counter. Both would end up as hand-written Java inside JFlex actions anyway.
• Grammar-Kit BNF for parser + PSI. Verified headless: the plugin ID is org.jetbrains.intellij.platform.grammarkit, applied in build.gradle.kts. generateLexer / generateParser tasks are present in ./jaigradle tasks.
• Ship highlighting before the parser. Highlighting needs only the lexer.

---

The test corpus is the main asset

~/.local/jai holds 714 .jai files, 16.4 MB of real code by the language authors. Use it as the primary correctness gate instead of hand-written samples.

The highest-value assertion, per the plan: lex every file and check the concatenation of all token texts reproduces the source byte for byte. An IntelliJ lexer must tile its input with no gaps or overlaps, so this single invariant over 16 MB catches nearly every lexer bug — with no fixture boot and no human review. Also assert no BAD_CHARACTER and that every advance() strictly increases the offset (catches infinite loops, which hang the IDE rather than failing visibly).

This is implemented in JaiCorpusLexerTest and it is green: 714 files, 17.1 M chars, 3.0 M tokens, in under a second. JaiSyntaxHighlighterTest runs the same sweep to assert every token type the corpus produces (105 of them) has a colour. Keep new lexer work under these gates rather than adding snippets.

Same idea for the parser, and it is now live: JaiCorpusParserTest parses all 714 files and counts PsiErrorElements — currently 714 clean (100.0%). The ratchet is set to the full corpus size, so any parser regression fails the test. #asm bodies remain opaque by design; they are tiled as one directive body but have no inner PSI. The idempotence sweep for the formatter is the same idea again, later.

The fast loop for grammar work:

1. ./jaitest, then read JaiCorpusParserTest's histogram and sample errors.
2. Reduce one failure to a snippet in DebugParseTest (inert by default).
3. Fix Jai.bnf or JaiParserUtil, re-run, watch the percentage.
4. Raise MIN_CLEAN_FILES and commit.

---

Environment gotchas

• Stale Gradle daemons. Mixing JDK 17 and 21 daemons caused Timeout waiting to lock journal cache. Fix: ./jaigradle --stop.

• Stale build outputs cost hours. Symptoms: a Jai.bnf edit has no effect, or the build fails inside generated code complaining about a method that is not in the file you are reading. Check timestamps — ls -l build/classes/java/main/dev/hgh/jai/parser/JaiParser.class src/main/gen/dev/hgh/jai/parser/JaiParser.java. The build wiring in build.gradle.kts should prevent both, but rm -rf build settles it.

• Concurrent Gradle runs corrupt the test results. ./jaigradle and ./jaitest serialize project Gradle work with the atomic .gradle-test.lock lock, including test-result cleanup and retries. Never run ./gradlew directly: it bypasses both the JDK wrapper and this lock. If an interrupted process leaves a lock behind, confirm no Gradle run is active, then remove .gradle-test.lock and rerun. The JUnit wrapper still retries once for unexpected no-XML failures; do not debug a test until the XML exists.

• A poisoned build cache can make tests silently vanish. Symptom: BUILD SUCCESSFUL, :compileTestKotlin FROM-CACHE, :test NO-SOURCE, and build/classes/kotlin/test is empty. The cache stored an empty output directory from a build whose outputs were deleted underneath it, and the entry is keyed by input hash so it never gets replaced. This is the exact failure mode ./jaitest's "total tests = 0 is a failure" rule exists to catch. Recover with:

  ./jaigradle --stop
  rm -rf .gradle-test.lock build .gradle ~/.gradle/caches/build-cache-1
  ./jaitest
  

• Cannot create child file 'x' at /src is a flake. The fixture's in-memory filesystem occasionally refuses myFixture.configureByText. It is not a real failure — re-run. JaiCorpusParserTest avoids it entirely by going through PsiFileFactory.createFileFromText instead of the fixture, which also makes the 714-file sweep much faster.

• timeout does not exist on this macOS box. Do not use it in scripts.

• Noisy test stderr. Fixture runs log Vue/JS PluginExceptions from bundled plugins. Tests pass regardless, but real failures can be buried — check the XML. Narrowing the test fixture later would help.

• runIde is never required. Everything is verifiable headlessly; that is a hard project requirement, not a preference. Do not ask the user to open an IDE to check your work.

---

Working agreement

• Verify claims by running commands; prefer empirical checks over docs. Several plan decisions came from testing assumptions that turned out false.
• Work in small increments: change one thing, run ./jaitest, commit when green.
• Keep the working tree clean — revert throwaway probes.
• Generated sources go in src/main/gen, are committed, and are never hand-edited. They regenerate on every build from src/main/grammar/Jai.bnf.
• Update this file when project state changes.