Update AGENTS.md and the plan for phase 3
Records the parser design facts a future agent needs before touching Jai.bnf: the token-factory requirement, what lives in JaiParserUtil and why, the PSI element that truncates an enclosing expression, longest-first operator ordering, and directive-flag adjacency. Adds the stale-output symptoms and the sticky --tests filter to the environment gotchas, and writes down the grammar iteration loop. Phase 3 is marked done; the remaining 58 corpus files are listed as the pick-up point with the three known gaps named.
This commit is contained in:
hgranthorner
106
AGENTS.md
106
AGENTS.md
@@ -50,6 +50,12 @@ It fails when the total test count is 0, so a silent no-op run cannot look
|
||||
green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and
|
||||
look for `tests="N"` with `failures="0" errors="0"` and **N > 0**.
|
||||
|
||||
**A `--tests` filter sticks.** Gradle reuses the configuration cache entry from
|
||||
the previous run, so a later unfiltered `./jaitest` silently re-runs only that one
|
||||
class and still prints `OK`. `jaitest` therefore passes
|
||||
`--no-configuration-cache` whenever no `--tests` argument is given. Do not
|
||||
"optimise" that away.
|
||||
|
||||
Last verified state (all green, `./jaigradle check` and `verifyPlugin` too):
|
||||
|
||||
```text
|
||||
@@ -59,13 +65,17 @@ dev.hgh.jai.editor.JaiEditorSupportTest tests=5
|
||||
dev.hgh.jai.highlighting.* tests=8
|
||||
dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate
|
||||
dev.hgh.jai.lexer.JaiLexerTest tests=13
|
||||
-> total 33, failures+errors 0
|
||||
dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate
|
||||
dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees
|
||||
dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert
|
||||
-> total 41, failures+errors 0
|
||||
```
|
||||
|
||||
The corpus gate reports what it actually did; check the line is still there:
|
||||
The corpus gates report what they actually did; check both lines are still there:
|
||||
|
||||
```text
|
||||
Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
||||
Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements total.
|
||||
```
|
||||
|
||||
---
|
||||
@@ -93,6 +103,10 @@ Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
||||
- **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement),
|
||||
colour settings page, commenter, brace matcher. Plugin Verifier passes
|
||||
against IU-253/261/262.
|
||||
- **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in
|
||||
`src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and
|
||||
the Tier 3 corpus gate are green at **91.9% (656/714 files parse with zero
|
||||
`PsiErrorElement`)**.
|
||||
|
||||
### Lexer design facts worth knowing before touching it
|
||||
|
||||
@@ -107,17 +121,67 @@ Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
||||
- **Built-in type names and `it`/`it_index` are IDENT**, refined into separate
|
||||
token types only by `JaiHighlightingLexer`, which the parser never sees. Do not
|
||||
reserve them (language reference §14.9).
|
||||
- **A backslash inside an identifier is a continuation.** The compiler's lexer
|
||||
(`Jai_Lexer/module.jai:444`) eats the `\` and any spaces after it and keeps
|
||||
appending, so `left\_margin` is one identifier. The token span still covers the
|
||||
backslash, so the Tier 0 round-trip holds.
|
||||
- **`#string,\% ID` exists** in `how_to/018_print_functions.jai` even though the
|
||||
shipped `Jai_Lexer` module rejects anything but `,cr`. Our `scanHereString`
|
||||
accepts any non-space modifier run. When the module and real code disagree,
|
||||
real code wins.
|
||||
- Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build.
|
||||
|
||||
### Parser design facts worth knowing before touching it
|
||||
|
||||
- **Generated sources are wiped and regenerated on every build.** `compileJava`
|
||||
and `compileKotlin` depend on `generateParser`, and a `Delete` task clears
|
||||
`src/main/gen` first. Editing `Jai.bnf` is enough; never run a bare compile and
|
||||
trust it. Both halves of that were real bugs:
|
||||
- without the dependency, a `Jai.bnf` edit compiled against the *previous*
|
||||
parser and the tests reported green (the `.class` was minutes older than its
|
||||
source);
|
||||
- without the wipe, Grammar-Kit's partial regeneration left
|
||||
`JaiDeclaration.getArgumentList()` with no override in the Impl and the build
|
||||
failed inside generated code.
|
||||
- **Grammar-Kit mints its own token instances** from the `tokens` block. Those
|
||||
are different objects from the ones `JaiLexer` emits, so every rule silently
|
||||
fails to match and a whole file becomes one error element. `tokenTypeFactory`
|
||||
points at `JaiTokenTypes.byName`, which resolves them reflectively.
|
||||
- **What BNF cannot express lives in `JaiParserUtil`**: directive-name tests
|
||||
(`#ident` is one token), `==` before `{` for the switch form, procedure header
|
||||
vs parenthesised expression, and the declaration lookahead — Jai declarations
|
||||
have no introducer keyword.
|
||||
- **A rule that creates a PSI element inside an expression can truncate it.**
|
||||
Using `initializer` for a named return default made the enclosing
|
||||
`procLiteralExpr` end early, so `-> a: int = 1 { }` silently lost its body. The
|
||||
private `returnDefault_` rule exists for that reason — do not "simplify" it.
|
||||
- **Longest-first matters for multi-token sequences only.** `operator *[]` needs
|
||||
`'*' '[' ']'` before `'*'`. Single-token operators cannot collide, because `+=`
|
||||
is one token.
|
||||
- **Directive flags require adjacency.** `#library,system` is a flag;
|
||||
`(cb: (*GUID) #c_call, ctx: *void)` is a parameter separator. Without the
|
||||
adjacency check the flag swallowed `, ctx`.
|
||||
|
||||
### Not done — pick up here
|
||||
|
||||
1. **Phase 3: parser.** `.bnf` grammar → Grammar-Kit generated parser + PSI in
|
||||
`src/main/gen`, plus a `ParserDefinition`. Gate: Tier 3 corpus parse with zero
|
||||
`PsiErrorElement` (allowlist committed and shrinking).
|
||||
Note `JaiFileTypeTest` documents the one thing blocked on this: until a
|
||||
`ParserDefinition` exists, PSI files for `.jai` are plain text, so
|
||||
PSI-dependent features (comment action, structure view) cannot be tested.
|
||||
2. **Phases 4+** — see the plan.
|
||||
1. **Finish the Tier 3 corpus gate.** 58 of 714 files still produce a
|
||||
`PsiErrorElement`, all long-tail: each construct below appears in one or two
|
||||
files. `JaiCorpusParserTest` prints a histogram of the token at each failure
|
||||
point plus a spread of sample errors; that output is the work list.
|
||||
`MIN_CLEAN_FILES` is the ratchet — raise it, never lower it. Known gaps:
|
||||
- mixed declare/assign target lists: `success=, output, error := adb(…)` and
|
||||
`renderer:, success = make_renderer(…)`;
|
||||
- `using,except SKIP_THESE name: T;` (a bare identifier operand, where
|
||||
`using,except(x)` and `using,except .["x"]` already work);
|
||||
- `#asm` bodies are consumed opaquely, so nothing inside them has PSI.
|
||||
|
||||
Decide explicitly whether the remainder becomes a committed allowlist (the
|
||||
plan's §3 suggestion) or gets fixed.
|
||||
2. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution
|
||||
and go-to-definition. The PSI is in place, so these are now unblocked.
|
||||
`JaiFileTypeTest` used to document being blocked on the `ParserDefinition`;
|
||||
that no longer applies.
|
||||
3. **Phases 5+** — see the plan.
|
||||
|
||||
### Open questions for the user (unanswered)
|
||||
|
||||
@@ -159,8 +223,18 @@ This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files,
|
||||
the same sweep to assert every token type the corpus produces (105 of them) has
|
||||
a colour. Keep new lexer work under these gates rather than adding snippets.
|
||||
|
||||
Same idea later: zero `PsiErrorElement` across the corpus for the parser;
|
||||
formatting is idempotent for the formatter.
|
||||
Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all
|
||||
714 files and counts `PsiErrorElement`s — currently **656 clean (91.9%)**. It
|
||||
prints a histogram of the source text at each failure point, which is the
|
||||
fastest way to find the next construct worth supporting. The idempotence sweep
|
||||
for the formatter is the same idea again, later.
|
||||
|
||||
The fast loop for grammar work:
|
||||
|
||||
1. `./jaitest`, then read `JaiCorpusParserTest`'s histogram and sample errors.
|
||||
2. Reduce one failure to a snippet in `DebugParseTest` (inert by default).
|
||||
3. Fix `Jai.bnf` or `JaiParserUtil`, re-run, watch the percentage.
|
||||
4. Raise `MIN_CLEAN_FILES` and commit.
|
||||
|
||||
---
|
||||
|
||||
@@ -168,6 +242,12 @@ formatting is idempotent for the formatter.
|
||||
|
||||
- **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused
|
||||
`Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`.
|
||||
- **Stale build outputs cost hours.** Symptoms: a `Jai.bnf` edit has no effect,
|
||||
or the build fails inside generated code complaining about a method that is not
|
||||
in the file you are reading. Check timestamps —
|
||||
`ls -l build/classes/java/main/dev/hgh/jai/parser/JaiParser.class
|
||||
src/main/gen/dev/hgh/jai/parser/JaiParser.java`. The build wiring in
|
||||
`build.gradle.kts` should prevent both, but `rm -rf build` settles it.
|
||||
- **Concurrent Gradle runs corrupt the test results.** If something else (an
|
||||
IDE, an agent's background checker) runs `test` at the same time, one of the
|
||||
two dies with `java.io.EOFException` or
|
||||
@@ -204,6 +284,6 @@ formatting is idempotent for the formatter.
|
||||
plan decisions came from testing assumptions that turned out false.
|
||||
- Work in small increments: change one thing, run `./jaitest`, commit when green.
|
||||
- Keep the working tree clean — revert throwaway probes.
|
||||
- Generated sources (once Grammar-Kit is wired up) go in `src/main/gen`, are
|
||||
committed, and are **never hand-edited**; regenerate instead.
|
||||
- Generated sources go in `src/main/gen`, are committed, and are **never
|
||||
hand-edited**. They regenerate on every build from `src/main/grammar/Jai.bnf`.
|
||||
- Update this file when project state changes.
|
||||
|
||||
@@ -167,7 +167,7 @@ Each phase has a machine-checkable gate. Do not advance without a green gate.
|
||||
| **0** | Env fix, git, AGENTS.md, template stripped | `./jaigradle check` green with no `JAVA_HOME` |
|
||||
| **1** | File type, icon, `JaiTokenTypes`, hand-written lexer | Tier 0 corpus invariants pass on all 714 files |
|
||||
| **2** | `SyntaxHighlighter`, color settings page, commenter, brace matcher | Tier 1 golden dumps; highlighter maps every token type |
|
||||
| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target |
|
||||
| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target — **done, 91.9% (656/714)** |
|
||||
| **4** | Structure view, folding, `#import`/`#load` reference resolution + go-to-definition | Tier 4 fixture tests |
|
||||
| **5** | Completion (keywords, directives, module names), rename, find-usages | Tier 4 fixture tests |
|
||||
| **6** | Formatter, code style settings | Formatter round-trip: formatting the corpus is idempotent |
|
||||
|
||||
Reference in New Issue
Block a user