193 lines
8.4 KiB
Markdown
193 lines
8.4 KiB
Markdown
# AGENTS.md — Jai IntelliJ Plugin
|
||
|
||
Orientation for agents working on this repo. Read this first.
|
||
|
||
**Goal:** an IntelliJ plugin for the **Jai** programming language.
|
||
|
||
**Hard rule:** do **not** research Jai online — public material is outdated and
|
||
wrong. The only source of truth is the local install at `~/.local/jai`
|
||
(`how_to/`, `modules/`, `examples/`). Researching *IntelliJ plugin development*
|
||
online is fine and encouraged.
|
||
|
||
---
|
||
|
||
## Run anything with `./jaigradle`
|
||
|
||
```bash
|
||
./jaigradle test # unit tests
|
||
./jaigradle check # full verification
|
||
./jaigradle tasks # discover tasks
|
||
```
|
||
|
||
**Never call `./gradlew` directly.** There is no JDK on `PATH` and `JAVA_HOME`
|
||
is unset in a fresh shell. `./jaigradle` resolves the JDK via mise, then
|
||
delegates. It works from an environment with no Java at all.
|
||
|
||
Do not try to fix this with `org.gradle.java.home` in `gradle.properties` —
|
||
already tried, it does not work, because `gradlew` is a shell script that needs
|
||
a JVM to launch *itself* before it ever reads that property.
|
||
|
||
**Java must be 21+.** The IntelliJ Platform test-framework jars are Java 21
|
||
bytecode (class major version 65); on JDK 17 every test fails with
|
||
`UnsupportedClassVersionError`. `mise.toml` pins `java = "temurin-21"`.
|
||
|
||
Note that `./jaigradle compileKotlin` **passes on JDK 17** — the template
|
||
sources never touch the test framework. Compiling is not evidence the toolchain
|
||
is correct. Run tests.
|
||
|
||
---
|
||
|
||
## Verify with the JUnit XML, not the exit code
|
||
|
||
A green `test` task does not prove tests ran. Use the committed wrapper:
|
||
|
||
```bash
|
||
./jaitest # whole suite, prints per-suite JUnit XML numbers
|
||
./jaitest --tests '*Lexer*' # one class while iterating
|
||
```
|
||
|
||
It fails when the total test count is 0, so a silent no-op run cannot look
|
||
green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and
|
||
look for `tests="N"` with `failures="0" errors="0"` and **N > 0**.
|
||
|
||
Last verified state (all green, `./jaigradle check` and `verifyPlugin` too):
|
||
|
||
```text
|
||
dev.hgh.HarnessSmokeTest tests=2
|
||
dev.hgh.jai.JaiFileTypeTest tests=3
|
||
dev.hgh.jai.editor.JaiEditorSupportTest tests=5
|
||
dev.hgh.jai.highlighting.* tests=8
|
||
dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate
|
||
dev.hgh.jai.lexer.JaiLexerTest tests=13
|
||
-> total 33, failures+errors 0
|
||
```
|
||
|
||
The corpus gate reports what it actually did; check the line is still there:
|
||
|
||
```text
|
||
Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
||
```
|
||
|
||
---
|
||
|
||
## Current state
|
||
|
||
### Done
|
||
|
||
- `docs/JAI_LANGUAGE_REFERENCE.md` — language reference derived from
|
||
`~/.local/jai`, keywords/tokens transcribed from the compiler's own lexer.
|
||
**Read this before writing any lexer or parser code.**
|
||
- `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions.
|
||
- `mise.toml` → `java = "temurin-21"` (was 17, which was broken).
|
||
- `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified.
|
||
- `jaitest` — runs the suite and reports the JUnit XML (see above).
|
||
- `src/test/kotlin/dev/hgh/HarnessSmokeTest.kt` — proves the platform boots
|
||
headlessly. **If this fails, no other test can be trusted.**
|
||
- **Phase 0** — git repo, template stripped, `plugin.xml` rewritten.
|
||
- **Phase 1** — `JaiLanguage`, `JaiFileType` (+ icon), `JaiTokenTypes`,
|
||
hand-written `JaiLexer`. Tier 0 corpus gate and Tier 1 golden tests green.
|
||
- **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement),
|
||
colour settings page, commenter, brace matcher. Plugin Verifier passes
|
||
against IU-253/261/262.
|
||
|
||
### Lexer design facts worth knowing before touching it
|
||
|
||
- **State is always 0.** Nested block comments and here-strings are each
|
||
consumed inside a *single* token, so no context crosses a token boundary and
|
||
the lexer can restart anywhere. Do not add lexer states without re-checking
|
||
incremental re-highlighting.
|
||
- **Deliberate divergences from the compiler's lexer**, all documented in
|
||
`JaiTokenTypes`' KDoc: `#ident` is one DIRECTIVE token; `#string ... ID` is one
|
||
HERE_STRING token; `::` and `:=` are single tokens; the backtick is its own
|
||
token.
|
||
- **Built-in type names and `it`/`it_index` are IDENT**, refined into separate
|
||
token types only by `JaiHighlightingLexer`, which the parser never sees. Do not
|
||
reserve them (language reference §14.9).
|
||
- Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build.
|
||
|
||
### Not done — pick up here
|
||
|
||
1. **Phase 3: parser.** `.bnf` grammar → Grammar-Kit generated parser + PSI in
|
||
`src/main/gen`, plus a `ParserDefinition`. Gate: Tier 3 corpus parse with zero
|
||
`PsiErrorElement` (allowlist committed and shrinking).
|
||
Note `JaiFileTypeTest` documents the one thing blocked on this: until a
|
||
`ParserDefinition` exists, PSI files for `.jai` are plain text, so
|
||
PSI-dependent features (comment action, structure view) cannot be tested.
|
||
2. **Phases 4+** — see the plan.
|
||
|
||
### Open questions for the user (unanswered)
|
||
|
||
Scope (Phases 0–4 vs 0–7), compiler integration, target IDEs, JDK policy. See
|
||
`docs/BUILD_PLAN.md` §6. Scope is the one that most affects the work.
|
||
|
||
---
|
||
|
||
## Architecture decisions already made
|
||
|
||
Both are justified in `docs/BUILD_PLAN.md` §2. Do not silently reverse them.
|
||
|
||
- **Hand-written lexer, not JFlex.** Jai here-strings (`#string DONE … DONE`)
|
||
have a *runtime-captured* terminator, which a DFA generator cannot express.
|
||
Nested block comments need a depth counter. Both would end up as hand-written
|
||
Java inside JFlex actions anyway.
|
||
- **Grammar-Kit BNF for parser + PSI.** Verified headless: the plugin ID is
|
||
`org.jetbrains.intellij.platform.grammarkit`, bundled in the IntelliJ Platform
|
||
Gradle Plugin since 2.12.0 (we run 2.18.1). `generateLexer` / `generateParser`
|
||
already appear in `./jaigradle tasks --all` — no extra dependency needed.
|
||
- **Ship highlighting before the parser.** Highlighting needs only the lexer.
|
||
|
||
---
|
||
|
||
## The test corpus is the main asset
|
||
|
||
`~/.local/jai` holds **714 `.jai` files, 16.4 MB** of real code by the language
|
||
authors. Use it as the primary correctness gate instead of hand-written samples.
|
||
|
||
The highest-value assertion, per the plan: lex every file and check the
|
||
concatenation of all token texts reproduces the source **byte for byte**. An
|
||
IntelliJ lexer must tile its input with no gaps or overlaps, so this single
|
||
invariant over 16 MB catches nearly every lexer bug — with no fixture boot and
|
||
no human review. Also assert no `BAD_CHARACTER` and that every `advance()`
|
||
strictly increases the offset (catches infinite loops, which hang the IDE rather
|
||
than failing visibly).
|
||
|
||
This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files,
|
||
17.1 M chars, 3.0 M tokens, in under a second. `JaiSyntaxHighlighterTest` runs
|
||
the same sweep to assert every token type the corpus produces (105 of them) has
|
||
a colour. Keep new lexer work under these gates rather than adding snippets.
|
||
|
||
Same idea later: zero `PsiErrorElement` across the corpus for the parser;
|
||
formatting is idempotent for the formatter.
|
||
|
||
---
|
||
|
||
## Environment gotchas
|
||
|
||
- **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused
|
||
`Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`.
|
||
- **Concurrent Gradle runs corrupt the test results.** If something else (an
|
||
IDE, an agent's background checker) runs `test` at the same time, one of the
|
||
two dies with `java.io.EOFException` or
|
||
`NoSuchFileException: .../in-progress-results-generic.bin`, and no XML is
|
||
written. It is infrastructure, not a test failure — `./jaitest` retries once
|
||
automatically. Do not go debugging the test that "failed".
|
||
- **`timeout` does not exist on this macOS box.** Do not use it in scripts.
|
||
- **Noisy test stderr.** Fixture runs log Vue/JS `PluginException`s from bundled
|
||
plugins. Tests pass regardless, but real failures can be buried — check the
|
||
XML. Narrowing the test fixture later would help.
|
||
- **`runIde` is never required.** Everything is verifiable headlessly; that is a
|
||
hard project requirement, not a preference. Do not ask the user to open an IDE
|
||
to check your work.
|
||
|
||
---
|
||
|
||
## Working agreement
|
||
|
||
- Verify claims by running commands; prefer empirical checks over docs. Several
|
||
plan decisions came from testing assumptions that turned out false.
|
||
- Work in small increments: change one thing, run `./jaitest`, commit when green.
|
||
- Keep the working tree clean — revert throwaway probes.
|
||
- Generated sources (once Grammar-Kit is wired up) go in `src/main/gen`, are
|
||
committed, and are **never hand-edited**; regenerate instead.
|
||
- Update this file when project state changes.
|