intellijai/AGENTS.md

AGENTS.md — Jai IntelliJ Plugin

Orientation for agents working on this repo. Read this first.

Goal: an IntelliJ plugin for the Jai programming language.

Hard rule: do not research Jai online — public material is outdated and wrong. The only source of truth is the local install at ~/.local/jai (how_to/, modules/, examples/). Researching IntelliJ plugin development online is fine and encouraged.

---

Run anything with ./jaigradle

./jaigradle test     # unit tests
./jaigradle check    # full verification
./jaigradle tasks    # discover tasks

Never call ./gradlew directly. There is no JDK on PATH and JAVA_HOME is unset in a fresh shell. ./jaigradle resolves the JDK via mise, then delegates. It works from an environment with no Java at all.

Do not try to fix this with org.gradle.java.home in gradle.properties — already tried, it does not work, because gradlew is a shell script that needs a JVM to launch itself before it ever reads that property.

Java must be 21+. The IntelliJ Platform test-framework jars are Java 21 bytecode (class major version 65); on JDK 17 every test fails with UnsupportedClassVersionError. mise.toml pins java = "temurin-21".

Note that ./jaigradle compileKotlin passes on JDK 17 — the template sources never touch the test framework. Compiling is not evidence the toolchain is correct. Run tests.

---

Verify with the JUnit XML, not the exit code

A green test task does not prove tests ran. Use the committed wrapper:

./jaitest                      # whole suite, prints per-suite JUnit XML numbers
./jaitest --tests '*Lexer*'    # one class while iterating

It fails when the total test count is 0, so a silent no-op run cannot look green. Doing it by hand instead: sed -n 2p build/test-results/test/*.xml and look for tests="N" with failures="0" errors="0" and N > 0.

Last verified state (all green, ./jaigradle check and verifyPlugin too):

dev.hgh.HarnessSmokeTest                    tests=2
dev.hgh.jai.JaiFileTypeTest                 tests=3
dev.hgh.jai.editor.JaiEditorSupportTest     tests=5
dev.hgh.jai.highlighting.*                  tests=8
dev.hgh.jai.lexer.JaiCorpusLexerTest        tests=2   <- the Tier 0 gate
dev.hgh.jai.lexer.JaiLexerTest              tests=13
-> total 33, failures+errors 0

The corpus gate reports what it actually did; check the line is still there:

Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.

---

Current state

Done

• docs/JAI_LANGUAGE_REFERENCE.md — language reference derived from ~/.local/jai, keywords/tokens transcribed from the compiler's own lexer. Read this before writing any lexer or parser code.
• docs/BUILD_PLAN.md — phased plan, testing strategy, risks, open questions.
• mise.toml → java = "temurin-21" (was 17, which was broken).
• jaigradle — JDK-resolving Gradle wrapper. Executable, verified.
• jaitest — runs the suite and reports the JUnit XML (see above).
• src/test/kotlin/dev/hgh/HarnessSmokeTest.kt — proves the platform boots headlessly. If this fails, no other test can be trusted.
• Phase 0 — git repo, template stripped, plugin.xml rewritten.
• Phase 1 — JaiLanguage, JaiFileType (+ icon), JaiTokenTypes, hand-written JaiLexer. Tier 0 corpus gate and Tier 1 golden tests green.
• Phase 2 — JaiSyntaxHighlighter (+ highlight-only lexer refinement), colour settings page, commenter, brace matcher. Plugin Verifier passes against IU-253/261/262.

Lexer design facts worth knowing before touching it

• State is always 0. Nested block comments and here-strings are each consumed inside a single token, so no context crosses a token boundary and the lexer can restart anywhere. Do not add lexer states without re-checking incremental re-highlighting.
• Deliberate divergences from the compiler's lexer, all documented in JaiTokenTypes' KDoc: #ident is one DIRECTIVE token; #string ... ID is one HERE_STRING token; :: and := are single tokens; the backtick is its own token.
• Built-in type names and it/it_index are IDENT, refined into separate token types only by JaiHighlightingLexer, which the parser never sees. Do not reserve them (language reference §14.9).
• Kotlin block comments nest too: writing /* inside a KDoc breaks the build.

Not done — pick up here

1. Phase 3: parser. .bnf grammar → Grammar-Kit generated parser + PSI in src/main/gen, plus a ParserDefinition. Gate: Tier 3 corpus parse with zero PsiErrorElement (allowlist committed and shrinking). Note JaiFileTypeTest documents the one thing blocked on this: until a ParserDefinition exists, PSI files for .jai are plain text, so PSI-dependent features (comment action, structure view) cannot be tested.
2. Phases 4+ — see the plan.

Open questions for the user (unanswered)

Scope (Phases 0–4 vs 0–7), compiler integration, target IDEs, JDK policy. See docs/BUILD_PLAN.md §6. Scope is the one that most affects the work.

---

Architecture decisions already made

Both are justified in docs/BUILD_PLAN.md §2. Do not silently reverse them.

• Hand-written lexer, not JFlex. Jai here-strings (#string DONE … DONE) have a runtime-captured terminator, which a DFA generator cannot express. Nested block comments need a depth counter. Both would end up as hand-written Java inside JFlex actions anyway.
• Grammar-Kit BNF for parser + PSI. Verified headless: the plugin ID is org.jetbrains.intellij.platform.grammarkit, bundled in the IntelliJ Platform Gradle Plugin since 2.12.0 (we run 2.18.1). generateLexer / generateParser already appear in ./jaigradle tasks --all — no extra dependency needed.
• Ship highlighting before the parser. Highlighting needs only the lexer.

---

The test corpus is the main asset

~/.local/jai holds 714 .jai files, 16.4 MB of real code by the language authors. Use it as the primary correctness gate instead of hand-written samples.

The highest-value assertion, per the plan: lex every file and check the concatenation of all token texts reproduces the source byte for byte. An IntelliJ lexer must tile its input with no gaps or overlaps, so this single invariant over 16 MB catches nearly every lexer bug — with no fixture boot and no human review. Also assert no BAD_CHARACTER and that every advance() strictly increases the offset (catches infinite loops, which hang the IDE rather than failing visibly).

This is implemented in JaiCorpusLexerTest and it is green: 714 files, 17.1 M chars, 3.0 M tokens, in under a second. JaiSyntaxHighlighterTest runs the same sweep to assert every token type the corpus produces (105 of them) has a colour. Keep new lexer work under these gates rather than adding snippets.

Same idea later: zero PsiErrorElement across the corpus for the parser; formatting is idempotent for the formatter.

---

Environment gotchas

• Stale Gradle daemons. Mixing JDK 17 and 21 daemons caused Timeout waiting to lock journal cache. Fix: ./jaigradle --stop.

• Concurrent Gradle runs corrupt the test results. If something else (an IDE, an agent's background checker) runs test at the same time, one of the two dies with java.io.EOFException or NoSuchFileException: .../in-progress-results-generic.bin, and no XML is written. It is infrastructure, not a test failure — ./jaitest retries once automatically. Do not go debugging the test that "failed".

• A poisoned build cache can make tests silently vanish. Symptom: BUILD SUCCESSFUL, :compileTestKotlin FROM-CACHE, :test NO-SOURCE, and build/classes/kotlin/test is empty. The cache stored an empty output directory from a build whose outputs were deleted underneath it, and the entry is keyed by input hash so it never gets replaced. This is the exact failure mode ./jaitest's "total tests = 0 is a failure" rule exists to catch. Recover with:

  ./jaigradle --stop
  rm -rf build .gradle ~/.gradle/caches/build-cache-1
  ./jaitest
  

• timeout does not exist on this macOS box. Do not use it in scripts.

• Noisy test stderr. Fixture runs log Vue/JS PluginExceptions from bundled plugins. Tests pass regardless, but real failures can be buried — check the XML. Narrowing the test fixture later would help.

• runIde is never required. Everything is verifiable headlessly; that is a hard project requirement, not a preference. Do not ask the user to open an IDE to check your work.

---

Working agreement

• Verify claims by running commands; prefer empirical checks over docs. Several plan decisions came from testing assumptions that turned out false.
• Work in small increments: change one thing, run ./jaitest, commit when green.
• Keep the working tree clean — revert throwaway probes.
• Generated sources (once Grammar-Kit is wired up) go in src/main/gen, are committed, and are never hand-edited; regenerate instead.
• Update this file when project state changes.