330 lines
17 KiB
Markdown
330 lines
17 KiB
Markdown
# AGENTS.md — Jai IntelliJ Plugin
|
||
|
||
Orientation for agents working on this repo. Read this first.
|
||
|
||
**Goal:** an IntelliJ plugin for the **Jai** programming language.
|
||
|
||
**Hard rule:** do **not** research Jai online — public material is outdated and
|
||
wrong. The only source of truth is the local install at `~/.local/jai`
|
||
(`how_to/`, `modules/`, `examples/`). Researching *IntelliJ plugin development*
|
||
online is fine and encouraged.
|
||
|
||
---
|
||
|
||
## Run anything with `./jaigradle`
|
||
|
||
```bash
|
||
./jaigradle test # unit tests
|
||
./jaigradle check # full verification
|
||
./jaigradle tasks # discover tasks
|
||
```
|
||
|
||
**Never call `./gradlew` directly.** There is no JDK on `PATH` and `JAVA_HOME`
|
||
is unset in a fresh shell. `./jaigradle` resolves the JDK via mise, then
|
||
delegates. It works from an environment with no Java at all.
|
||
|
||
Do not try to fix this with `org.gradle.java.home` in `gradle.properties` —
|
||
already tried, it does not work, because `gradlew` is a shell script that needs
|
||
a JVM to launch *itself* before it ever reads that property.
|
||
|
||
**Java must be 21+.** The IntelliJ Platform test-framework jars are Java 21
|
||
bytecode (class major version 65); on JDK 17 every test fails with
|
||
`UnsupportedClassVersionError`. `mise.toml` pins `java = "temurin-21"`.
|
||
|
||
Note that `./jaigradle compileKotlin` **passes on JDK 17** — the template
|
||
sources never touch the test framework. Compiling is not evidence the toolchain
|
||
is correct. Run tests.
|
||
|
||
---
|
||
|
||
## Verify with the JUnit XML, not the exit code
|
||
|
||
A green `test` task does not prove tests ran. Use the committed wrapper:
|
||
|
||
```bash
|
||
./jaitest # whole suite, prints per-suite JUnit XML numbers
|
||
./jaitest --tests '*Lexer*' # one class while iterating
|
||
```
|
||
|
||
It fails when the total test count is 0, so a silent no-op run cannot look
|
||
green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and
|
||
look for `tests="N"` with `failures="0" errors="0"` and **N > 0**.
|
||
|
||
**A `--tests` filter sticks.** Gradle reuses the configuration cache entry from
|
||
the previous run, so a later unfiltered `./jaitest` silently re-runs only that one
|
||
class and still prints `OK`. `jaitest` therefore passes
|
||
`--no-configuration-cache` whenever no `--tests` argument is given. Do not
|
||
"optimise" that away.
|
||
|
||
Last verified state (all green, `./jaigradle check` and `verifyPlugin` too):
|
||
|
||
```text
|
||
dev.hgh.HarnessSmokeTest tests=2
|
||
dev.hgh.jai.JaiFileTypeTest tests=4
|
||
dev.hgh.jai.completion.JaiCompletionTest tests=20
|
||
dev.hgh.jai.editor.JaiEditorSupportTest tests=7
|
||
dev.hgh.jai.editor.JaiFoldingBuilderTest tests=3
|
||
dev.hgh.jai.findusages.JaiFindUsagesTest tests=2
|
||
dev.hgh.jai.highlighting.* tests=8
|
||
dev.hgh.jai.inspection.JaiUnresolvedModuleInspectionTest tests=5
|
||
dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate
|
||
dev.hgh.jai.lexer.JaiLexerTest tests=13
|
||
dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate
|
||
dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees
|
||
dev.hgh.jai.parser.JaiParserLongTailTest tests=1 <- focused parser regressions
|
||
dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert
|
||
dev.hgh.jai.refactoring.JaiRenameTest tests=3
|
||
dev.hgh.jai.reference.JaiReferenceTest tests=10
|
||
dev.hgh.jai.structure.JaiStructureViewTest tests=3
|
||
dev.hgh.jai.formatter.JaiFormatterTest tests=4
|
||
dev.hgh.jai.settings.JaiProjectSettingsTest tests=3
|
||
dev.hgh.jai.settings.JaiConfiguredRootTest tests=6
|
||
-> total 104, failures+errors 0
|
||
```
|
||
|
||
The corpus gates report what they actually did; check both lines are still there:
|
||
|
||
```text
|
||
Tier 0: lexed 714 files, 17154195 chars, 3008777 tokens cleanly.
|
||
Tier 3: parsed 714 files, 714 clean (100.0%), 0 with errors, 0 PsiErrorElements total.
|
||
```
|
||
|
||
---
|
||
|
||
## Current state
|
||
|
||
### Done
|
||
|
||
- `docs/JAI_LANGUAGE_REFERENCE.md` — language reference derived from
|
||
`~/.local/jai`, keywords/tokens transcribed from the compiler's own lexer.
|
||
**Read this before writing any lexer or parser code.**
|
||
- `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions.
|
||
- Local IntelliJ reference projects and documentation in `~/programming/thirdparty`:
|
||
- `fortran-plugin` — a simple language plugin to use as reference.
|
||
- `intellij-elixir` — a more complex plugin to use as reference.
|
||
- `intellij-sdk-docs` — local versions of the IntelliJ SDK documentation.
|
||
- `mise.toml` → `java = "temurin-21"` (was 17, which was broken).
|
||
- `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified.
|
||
- `jaitest` — runs the suite and reports the JUnit XML (see above).
|
||
- `src/test/kotlin/dev/hgh/HarnessSmokeTest.kt` — proves the platform boots
|
||
headlessly. **If this fails, no other test can be trusted.**
|
||
- **Phase 0** — git repo, template stripped, `plugin.xml` rewritten.
|
||
- **Phase 1** — `JaiLanguage`, `JaiFileType` (+ icon), `JaiTokenTypes`,
|
||
hand-written `JaiLexer`. Tier 0 corpus gate and Tier 1 golden tests green.
|
||
- **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement),
|
||
colour settings page, commenter, brace matcher. Plugin Verifier passes
|
||
against IU-253/261/262.
|
||
- **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in
|
||
`src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and
|
||
the Tier 3 corpus gate are green at **100.0% (714/714 files parse with zero
|
||
`PsiErrorElement`)**.
|
||
- **Phase 4** — structure view, folding for blocks/comments/here-strings, and
|
||
`#import`/`#load` path references with standard go-to-definition resolution.
|
||
Headless tests cover source-ordered declarations, nested structure members,
|
||
folding ranges, local `#load`, and Jai module directories.
|
||
- **Phase 5** — completion for keywords, directives, module/file paths, visible
|
||
same-file or imported declarations, procedure parameters, and module aliases;
|
||
same-file and imported symbol navigation; rename; and find-usages for
|
||
declaration names. Struct-field completion remains future work.
|
||
- **Phase 5a** — persistent project settings for Jai module/import roots,
|
||
shared root-aware resolution and completion, and indexed external Jai library
|
||
sources. Headless tests cover state round-tripping, the Settings panel,
|
||
custom-root `#import`/`#load` navigation and completion, and cross-root
|
||
find-usages.
|
||
- **Phase 6** — PSI-aware formatter with operator/punctuation spacing,
|
||
block indentation, opaque `#asm` preservation, and a corpus-wide idempotence
|
||
gate. Headless formatter tests cover registration, representative formatting,
|
||
opaque/directive token preservation, and all 714 corpus files.
|
||
- **Phase 7a** — unresolved `#import`/`#load` module and file path inspection,
|
||
reusing the existing reference resolver. Headless tests cover unresolved
|
||
imports and loads, resolved project/local modules and files, and
|
||
`#import,string` exclusions.
|
||
|
||
### Lexer design facts worth knowing before touching it
|
||
|
||
- **State is always 0.** Nested block comments and here-strings are each
|
||
consumed inside a *single* token, so no context crosses a token boundary and
|
||
the lexer can restart anywhere. Do not add lexer states without re-checking
|
||
incremental re-highlighting.
|
||
- **Deliberate divergences from the compiler's lexer**, all documented in
|
||
`JaiTokenTypes`' KDoc: `#ident` is one DIRECTIVE token; `#string ... ID` is one
|
||
HERE_STRING token; `::` and `:=` are single tokens; the backtick is its own
|
||
token.
|
||
- **Built-in type names and `it`/`it_index` are IDENT**, refined into separate
|
||
token types only by `JaiHighlightingLexer`, which the parser never sees. Do not
|
||
reserve them (language reference §14.9).
|
||
- **A backslash inside an identifier is a continuation.** The compiler's lexer
|
||
(`Jai_Lexer/module.jai:444`) eats the `\` and any spaces after it and keeps
|
||
appending, so `left\_margin` is one identifier. The token span still covers the
|
||
backslash, so the Tier 0 round-trip holds.
|
||
- **`#string,\% ID` exists** in `how_to/018_print_functions.jai` even though the
|
||
shipped `Jai_Lexer` module rejects anything but `,cr`. Our `scanHereString`
|
||
accepts any non-space modifier run. When the module and real code disagree,
|
||
real code wins.
|
||
- Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build.
|
||
|
||
### Parser design facts worth knowing before touching it
|
||
|
||
- **Generated sources are wiped and regenerated on every build.** `compileJava`
|
||
and `compileKotlin` depend on `generateParser`, and a `Delete` task clears
|
||
`src/main/gen` first. Editing `Jai.bnf` is enough; never run a bare compile and
|
||
trust it. Both halves of that were real bugs:
|
||
- without the dependency, a `Jai.bnf` edit compiled against the *previous*
|
||
parser and the tests reported green (the `.class` was minutes older than its
|
||
source);
|
||
- without the wipe, Grammar-Kit's partial regeneration left
|
||
`JaiDeclaration.getArgumentList()` with no override in the Impl and the build
|
||
failed inside generated code.
|
||
- **Grammar-Kit mints its own token instances** from the `tokens` block. Those
|
||
are different objects from the ones `JaiLexer` emits, so every rule silently
|
||
fails to match and a whole file becomes one error element. `tokenTypeFactory`
|
||
points at `JaiTokenTypes.byName`, which resolves them reflectively.
|
||
- **What BNF cannot express lives in `JaiParserUtil`**: directive-name tests
|
||
(`#ident` is one token), `==` before `{` for the switch form, procedure header
|
||
vs parenthesised expression, and the declaration lookahead — Jai declarations
|
||
have no introducer keyword.
|
||
- **A rule that creates a PSI element inside an expression can truncate it.**
|
||
Using `initializer` for a named return default made the enclosing
|
||
`procLiteralExpr` end early, so `-> a: int = 1 { }` silently lost its body. The
|
||
private `returnDefault_` rule exists for that reason — do not "simplify" it.
|
||
- **Longest-first matters for multi-token sequences only.** `operator *[]` needs
|
||
`'*' '[' ']'` before `'*'`. Single-token operators cannot collide, because `+=`
|
||
is one token.
|
||
- **Directive flags require adjacency.** `#library,system` is a flag;
|
||
`(cb: (*GUID) #c_call, ctx: *void)` is a parameter separator. Without the
|
||
adjacency check the flag swallowed `, ctx`.
|
||
- **Contributed references need a PSI host.** Generated AST wrappers do not
|
||
implement `ContributedReferenceHost`, so `literalExpr` uses the
|
||
`JaiReferenceHostMixin` from the BNF. Do not remove the mixin and expect a
|
||
`PsiReferenceContributor` to be queried through the standard reference service.
|
||
|
||
### Known code-insight gaps
|
||
|
||
- Completion currently offers keywords, compiler directives, module/file paths,
|
||
visible same-file or imported declarations, procedure parameters, and members
|
||
of imported aliases, but not struct fields.
|
||
- Struct fields do not yet have symbol resolution or completion.
|
||
|
||
### In progress — pick up here
|
||
|
||
1. **Phases 7+** — additional inspections, quick fixes, live templates, and
|
||
compiler integration remain. `#asm` bodies are intentionally consumed
|
||
opaquely, so nothing inside them has PSI yet.
|
||
|
||
### Next planned increment
|
||
|
||
- Expand Phase 7 with additional high-confidence inspections and quick fixes;
|
||
struct-field resolution and completion remain a separate known gap.
|
||
|
||
### Open questions for the user (unanswered)
|
||
|
||
Scope (Phases 0–5a vs 0–7), compiler integration, target IDEs, JDK policy. See
|
||
`docs/BUILD_PLAN.md` §6. Scope is the one that most affects the work.
|
||
|
||
---
|
||
|
||
## Architecture decisions already made
|
||
|
||
Both are justified in `docs/BUILD_PLAN.md` §2. Do not silently reverse them.
|
||
|
||
- **Hand-written lexer, not JFlex.** Jai here-strings (`#string DONE … DONE`)
|
||
have a *runtime-captured* terminator, which a DFA generator cannot express.
|
||
Nested block comments need a depth counter. Both would end up as hand-written
|
||
Java inside JFlex actions anyway.
|
||
- **Grammar-Kit BNF for parser + PSI.** Verified headless: the plugin ID is
|
||
`org.jetbrains.intellij.platform.grammarkit`, applied in `build.gradle.kts`.
|
||
`generateLexer` / `generateParser` tasks are present in `./jaigradle tasks`.
|
||
- **Ship highlighting before the parser.** Highlighting needs only the lexer.
|
||
|
||
---
|
||
|
||
## The test corpus is the main asset
|
||
|
||
`~/.local/jai` holds **714 `.jai` files, 16.4 MB** of real code by the language
|
||
authors. Use it as the primary correctness gate instead of hand-written samples.
|
||
|
||
The highest-value assertion, per the plan: lex every file and check the
|
||
concatenation of all token texts reproduces the source **byte for byte**. An
|
||
IntelliJ lexer must tile its input with no gaps or overlaps, so this single
|
||
invariant over 16 MB catches nearly every lexer bug — with no fixture boot and
|
||
no human review. Also assert no `BAD_CHARACTER` and that every `advance()`
|
||
strictly increases the offset (catches infinite loops, which hang the IDE rather
|
||
than failing visibly).
|
||
|
||
This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files,
|
||
17.1 M chars, 3.0 M tokens, in under a second. `JaiSyntaxHighlighterTest` runs
|
||
the same sweep to assert every token type the corpus produces (105 of them) has
|
||
a colour. Keep new lexer work under these gates rather than adding snippets.
|
||
|
||
Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all
|
||
714 files and counts `PsiErrorElement`s — currently **714 clean (100.0%)**. The
|
||
ratchet is set to the full corpus size, so any parser regression fails the test.
|
||
`#asm` bodies remain opaque by design; they are tiled as one directive body but
|
||
have no inner PSI. The idempotence sweep for the formatter is the same idea
|
||
again, later.
|
||
|
||
The fast loop for grammar work:
|
||
|
||
1. `./jaitest`, then read `JaiCorpusParserTest`'s histogram and sample errors.
|
||
2. Reduce one failure to a snippet in `DebugParseTest` (inert by default).
|
||
3. Fix `Jai.bnf` or `JaiParserUtil`, re-run, watch the percentage.
|
||
4. Raise `MIN_CLEAN_FILES` and commit.
|
||
|
||
---
|
||
|
||
## Environment gotchas
|
||
|
||
- **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused
|
||
`Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`.
|
||
- **Stale build outputs cost hours.** Symptoms: a `Jai.bnf` edit has no effect,
|
||
or the build fails inside generated code complaining about a method that is not
|
||
in the file you are reading. Check timestamps —
|
||
`ls -l build/classes/java/main/dev/hgh/jai/parser/JaiParser.class
|
||
src/main/gen/dev/hgh/jai/parser/JaiParser.java`. The build wiring in
|
||
`build.gradle.kts` should prevent both, but `rm -rf build` settles it.
|
||
- **Concurrent Gradle runs corrupt the test results.** `./jaigradle` and
|
||
`./jaitest` serialize project Gradle work with the atomic `.gradle-test.lock`
|
||
lock, including test-result cleanup and retries. Never run `./gradlew`
|
||
directly: it bypasses both the JDK wrapper and this lock. If an interrupted
|
||
process leaves a lock behind, confirm no Gradle run is active, then remove
|
||
`.gradle-test.lock` and rerun. The JUnit wrapper still retries once for
|
||
unexpected no-XML failures; do not debug a test until the XML exists.
|
||
- **A poisoned build cache can make tests silently vanish.** Symptom:
|
||
`BUILD SUCCESSFUL`, `:compileTestKotlin FROM-CACHE`, `:test NO-SOURCE`, and
|
||
`build/classes/kotlin/test` is empty. The cache stored an empty output
|
||
directory from a build whose outputs were deleted underneath it, and the entry
|
||
is keyed by input hash so it never gets replaced. This is the exact failure
|
||
mode `./jaitest`'s "total tests = 0 is a failure" rule exists to catch.
|
||
Recover with:
|
||
|
||
```bash
|
||
./jaigradle --stop
|
||
rm -rf .gradle-test.lock build .gradle ~/.gradle/caches/build-cache-1
|
||
./jaitest
|
||
```
|
||
|
||
- **`Cannot create child file 'x' at /src` is a flake.** The fixture's in-memory
|
||
filesystem occasionally refuses `myFixture.configureByText`. It is not a real
|
||
failure — re-run. `JaiCorpusParserTest` avoids it entirely by going through
|
||
`PsiFileFactory.createFileFromText` instead of the fixture, which also makes
|
||
the 714-file sweep much faster.
|
||
- **`timeout` does not exist on this macOS box.** Do not use it in scripts.
|
||
- **Noisy test stderr.** Fixture runs log Vue/JS `PluginException`s from bundled
|
||
plugins. Tests pass regardless, but real failures can be buried — check the
|
||
XML. Narrowing the test fixture later would help.
|
||
- **`runIde` is never required.** Everything is verifiable headlessly; that is a
|
||
hard project requirement, not a preference. Do not ask the user to open an IDE
|
||
to check your work.
|
||
|
||
---
|
||
|
||
## Working agreement
|
||
|
||
- Verify claims by running commands; prefer empirical checks over docs. Several
|
||
plan decisions came from testing assumptions that turned out false.
|
||
- Work in small increments: change one thing, run `./jaitest`, commit when green.
|
||
- Keep the working tree clean — revert throwaway probes.
|
||
- Generated sources go in `src/main/gen`, are committed, and are **never
|
||
hand-edited**. They regenerate on every build from `src/main/grammar/Jai.bnf`.
|
||
- Update this file when project state changes.
|