intellijai/AGENTS.md

AGENTS.md — Jai IntelliJ Plugin

Orientation for agents working on this repo. Read this first.

Goal: an IntelliJ plugin for the Jai programming language.

Hard rule: do not research Jai online — public material is outdated and wrong. The only source of truth is the local install at ~/.local/jai (how_to/, modules/, examples/). Researching IntelliJ plugin development online is fine and encouraged.

---

Run anything with ./jaigradle

./jaigradle test     # unit tests
./jaigradle check    # full verification
./jaigradle tasks    # discover tasks

Never call ./gradlew directly. There is no JDK on PATH and JAVA_HOME is unset in a fresh shell. ./jaigradle resolves the JDK via mise, then delegates. It works from an environment with no Java at all.

Do not try to fix this with org.gradle.java.home in gradle.properties — already tried, it does not work, because gradlew is a shell script that needs a JVM to launch itself before it ever reads that property.

Java must be 21+. The IntelliJ Platform test-framework jars are Java 21 bytecode (class major version 65); on JDK 17 every test fails with UnsupportedClassVersionError. mise.toml pins java = "temurin-21".

Note that ./jaigradle compileKotlin passes on JDK 17 — the template sources never touch the test framework. Compiling is not evidence the toolchain is correct. Run tests.

---

Verify with the JUnit XML, not the exit code

A green test task does not prove tests ran. Use the committed wrapper:

./jaitest                      # whole suite, prints per-suite JUnit XML numbers
./jaitest --tests '*Lexer*'    # one class while iterating

It fails when the total test count is 0, so a silent no-op run cannot look green. Doing it by hand instead: sed -n 2p build/test-results/test/*.xml and look for tests="N" with failures="0" errors="0" and N > 0.

A --tests filter sticks. Gradle reuses the configuration cache entry from the previous run, so a later unfiltered ./jaitest silently re-runs only that one class and still prints OK. jaitest therefore passes --no-configuration-cache whenever no --tests argument is given. Do not "optimise" that away.

Last verified state (all green, ./jaigradle check and verifyPlugin too):

dev.hgh.HarnessSmokeTest                    tests=2
dev.hgh.jai.JaiFileTypeTest                 tests=3
dev.hgh.jai.editor.JaiEditorSupportTest     tests=5
dev.hgh.jai.highlighting.*                  tests=8
dev.hgh.jai.lexer.JaiCorpusLexerTest        tests=2   <- the Tier 0 gate
dev.hgh.jai.lexer.JaiLexerTest              tests=13
dev.hgh.jai.parser.JaiCorpusParserTest      tests=1   <- the Tier 3 gate
dev.hgh.jai.parser.JaiParserGoldenTest      tests=5   <- Tier 2 golden trees
dev.hgh.jai.parser.DebugParseTest           tests=2   <- scratch harness, inert
-> total 41, failures+errors 0

The corpus gates report what they actually did; check both lines are still there:

Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements total.

---

Current state

Done

• docs/JAI_LANGUAGE_REFERENCE.md — language reference derived from ~/.local/jai, keywords/tokens transcribed from the compiler's own lexer. Read this before writing any lexer or parser code.
• docs/BUILD_PLAN.md — phased plan, testing strategy, risks, open questions.
• Local IntelliJ reference projects and documentation in ~/programming/thirdparty:
  • fortran-plugin — a simple language plugin to use as reference.
  • intellij-elixir — a more complex plugin to use as reference.
  • intellij-sdk-docs — local versions of the IntelliJ SDK documentation.
• mise.toml → java = "temurin-21" (was 17, which was broken).
• jaigradle — JDK-resolving Gradle wrapper. Executable, verified.
• jaitest — runs the suite and reports the JUnit XML (see above).
• src/test/kotlin/dev/hgh/HarnessSmokeTest.kt — proves the platform boots headlessly. If this fails, no other test can be trusted.
• Phase 0 — git repo, template stripped, plugin.xml rewritten.
• Phase 1 — JaiLanguage, JaiFileType (+ icon), JaiTokenTypes, hand-written JaiLexer. Tier 0 corpus gate and Tier 1 golden tests green.
• Phase 2 — JaiSyntaxHighlighter (+ highlight-only lexer refinement), colour settings page, commenter, brace matcher. Plugin Verifier passes against IU-253/261/262.
• Phase 3 — src/main/grammar/Jai.bnf → Grammar-Kit parser and PSI in src/main/gen, JaiParserDefinition, JaiParserUtil. Tier 2 golden trees and the Tier 3 corpus gate are green at 91.9% (656/714 files parse with zero PsiErrorElement).

Lexer design facts worth knowing before touching it

• State is always 0. Nested block comments and here-strings are each consumed inside a single token, so no context crosses a token boundary and the lexer can restart anywhere. Do not add lexer states without re-checking incremental re-highlighting.
• Deliberate divergences from the compiler's lexer, all documented in JaiTokenTypes' KDoc: #ident is one DIRECTIVE token; #string ... ID is one HERE_STRING token; :: and := are single tokens; the backtick is its own token.
• Built-in type names and it/it_index are IDENT, refined into separate token types only by JaiHighlightingLexer, which the parser never sees. Do not reserve them (language reference §14.9).
• A backslash inside an identifier is a continuation. The compiler's lexer (Jai_Lexer/module.jai:444) eats the \ and any spaces after it and keeps appending, so left\_margin is one identifier. The token span still covers the backslash, so the Tier 0 round-trip holds.
• #string,\% ID exists in how_to/018_print_functions.jai even though the shipped Jai_Lexer module rejects anything but ,cr. Our scanHereString accepts any non-space modifier run. When the module and real code disagree, real code wins.
• Kotlin block comments nest too: writing /* inside a KDoc breaks the build.

Parser design facts worth knowing before touching it

• Generated sources are wiped and regenerated on every build. compileJava and compileKotlin depend on generateParser, and a Delete task clears src/main/gen first. Editing Jai.bnf is enough; never run a bare compile and trust it. Both halves of that were real bugs:
  • without the dependency, a Jai.bnf edit compiled against the previous parser and the tests reported green (the .class was minutes older than its source);
  • without the wipe, Grammar-Kit's partial regeneration left JaiDeclaration.getArgumentList() with no override in the Impl and the build failed inside generated code.
• Grammar-Kit mints its own token instances from the tokens block. Those are different objects from the ones JaiLexer emits, so every rule silently fails to match and a whole file becomes one error element. tokenTypeFactory points at JaiTokenTypes.byName, which resolves them reflectively.
• What BNF cannot express lives in JaiParserUtil: directive-name tests (#ident is one token), == before { for the switch form, procedure header vs parenthesised expression, and the declaration lookahead — Jai declarations have no introducer keyword.
• A rule that creates a PSI element inside an expression can truncate it. Using initializer for a named return default made the enclosing procLiteralExpr end early, so -> a: int = 1 { } silently lost its body. The private returnDefault_ rule exists for that reason — do not "simplify" it.
• Longest-first matters for multi-token sequences only. operator *[] needs '*' '[' ']' before '*'. Single-token operators cannot collide, because += is one token.
• Directive flags require adjacency. #library,system is a flag; (cb: (*GUID) #c_call, ctx: *void) is a parameter separator. Without the adjacency check the flag swallowed , ctx.

Not done — pick up here

1. Finish the Tier 3 corpus gate. 58 of 714 files still produce a PsiErrorElement, all long-tail: each construct below appears in one or two files. JaiCorpusParserTest prints a histogram of the token at each failure point plus a spread of sample errors; that output is the work list. MIN_CLEAN_FILES is the ratchet — raise it, never lower it. Known gaps:

   • mixed declare/assign target lists: success=, output, error := adb(…) and renderer:, success = make_renderer(…);
   • using,except SKIP_THESE name: T; (a bare identifier operand, where using,except(x) and using,except .["x"] already work);
   • #asm bodies are consumed opaquely, so nothing inside them has PSI.

   Decide explicitly whether the remainder becomes a committed allowlist (the plan's §3 suggestion) or gets fixed.

2. Phase 4 — structure view, folding, #import/#load reference resolution and go-to-definition. The PSI is in place, so these are now unblocked. JaiFileTypeTest used to document being blocked on the ParserDefinition; that no longer applies.

3. Phases 5+ — see the plan.

Open questions for the user (unanswered)

Scope (Phases 0–4 vs 0–7), compiler integration, target IDEs, JDK policy. See docs/BUILD_PLAN.md §6. Scope is the one that most affects the work.

---

Architecture decisions already made

Both are justified in docs/BUILD_PLAN.md §2. Do not silently reverse them.

• Hand-written lexer, not JFlex. Jai here-strings (#string DONE … DONE) have a runtime-captured terminator, which a DFA generator cannot express. Nested block comments need a depth counter. Both would end up as hand-written Java inside JFlex actions anyway.
• Grammar-Kit BNF for parser + PSI. Verified headless: the plugin ID is org.jetbrains.intellij.platform.grammarkit, applied in build.gradle.kts. generateLexer / generateParser tasks are present in ./jaigradle tasks.
• Ship highlighting before the parser. Highlighting needs only the lexer.

---

The test corpus is the main asset

~/.local/jai holds 714 .jai files, 16.4 MB of real code by the language authors. Use it as the primary correctness gate instead of hand-written samples.

The highest-value assertion, per the plan: lex every file and check the concatenation of all token texts reproduces the source byte for byte. An IntelliJ lexer must tile its input with no gaps or overlaps, so this single invariant over 16 MB catches nearly every lexer bug — with no fixture boot and no human review. Also assert no BAD_CHARACTER and that every advance() strictly increases the offset (catches infinite loops, which hang the IDE rather than failing visibly).

This is implemented in JaiCorpusLexerTest and it is green: 714 files, 17.1 M chars, 3.0 M tokens, in under a second. JaiSyntaxHighlighterTest runs the same sweep to assert every token type the corpus produces (105 of them) has a colour. Keep new lexer work under these gates rather than adding snippets.

Same idea for the parser, and it is now live: JaiCorpusParserTest parses all 714 files and counts PsiErrorElements — currently 656 clean (91.9%). It prints a histogram of the source text at each failure point, which is the fastest way to find the next construct worth supporting. The idempotence sweep for the formatter is the same idea again, later.

The fast loop for grammar work:

1. ./jaitest, then read JaiCorpusParserTest's histogram and sample errors.
2. Reduce one failure to a snippet in DebugParseTest (inert by default).
3. Fix Jai.bnf or JaiParserUtil, re-run, watch the percentage.
4. Raise MIN_CLEAN_FILES and commit.

---

Environment gotchas

• Stale Gradle daemons. Mixing JDK 17 and 21 daemons caused Timeout waiting to lock journal cache. Fix: ./jaigradle --stop.

• Stale build outputs cost hours. Symptoms: a Jai.bnf edit has no effect, or the build fails inside generated code complaining about a method that is not in the file you are reading. Check timestamps — ls -l build/classes/java/main/dev/hgh/jai/parser/JaiParser.class src/main/gen/dev/hgh/jai/parser/JaiParser.java. The build wiring in build.gradle.kts should prevent both, but rm -rf build settles it.

• Concurrent Gradle runs corrupt the test results. If something else (an IDE, an agent's background checker) runs test at the same time, one of the two dies with java.io.EOFException or NoSuchFileException: .../in-progress-results-generic.bin, and no XML is written. It is infrastructure, not a test failure — ./jaitest retries once automatically. Do not go debugging the test that "failed".

• A poisoned build cache can make tests silently vanish. Symptom: BUILD SUCCESSFUL, :compileTestKotlin FROM-CACHE, :test NO-SOURCE, and build/classes/kotlin/test is empty. The cache stored an empty output directory from a build whose outputs were deleted underneath it, and the entry is keyed by input hash so it never gets replaced. This is the exact failure mode ./jaitest's "total tests = 0 is a failure" rule exists to catch. Recover with:

  ./jaigradle --stop
  rm -rf build .gradle ~/.gradle/caches/build-cache-1
  ./jaitest
  

• timeout does not exist on this macOS box. Do not use it in scripts.

• Noisy test stderr. Fixture runs log Vue/JS PluginExceptions from bundled plugins. Tests pass regardless, but real failures can be buried — check the XML. Narrowing the test fixture later would help.

• runIde is never required. Everything is verifiable headlessly; that is a hard project requirement, not a preference. Do not ask the user to open an IDE to check your work.

---

Working agreement

• Verify claims by running commands; prefer empirical checks over docs. Several plan decisions came from testing assumptions that turned out false.
• Work in small increments: change one thing, run ./jaitest, commit when green.
• Keep the working tree clean — revert throwaway probes.
• Generated sources go in src/main/gen, are committed, and are never hand-edited. They regenerate on every build from src/main/grammar/Jai.bnf.
• Update this file when project state changes.