Update AGENTS.md and the plan for phase 3
Records the parser design facts a future agent needs before touching Jai.bnf: the token-factory requirement, what lives in JaiParserUtil and why, the PSI element that truncates an enclosing expression, longest-first operator ordering, and directive-flag adjacency. Adds the stale-output symptoms and the sticky --tests filter to the environment gotchas, and writes down the grammar iteration loop. Phase 3 is marked done; the remaining 58 corpus files are listed as the pick-up point with the three known gaps named.
This commit is contained in:
106
AGENTS.md
106
AGENTS.md
@@ -50,6 +50,12 @@ It fails when the total test count is 0, so a silent no-op run cannot look
|
|||||||
green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and
|
green. Doing it by hand instead: `sed -n 2p build/test-results/test/*.xml` and
|
||||||
look for `tests="N"` with `failures="0" errors="0"` and **N > 0**.
|
look for `tests="N"` with `failures="0" errors="0"` and **N > 0**.
|
||||||
|
|
||||||
|
**A `--tests` filter sticks.** Gradle reuses the configuration cache entry from
|
||||||
|
the previous run, so a later unfiltered `./jaitest` silently re-runs only that one
|
||||||
|
class and still prints `OK`. `jaitest` therefore passes
|
||||||
|
`--no-configuration-cache` whenever no `--tests` argument is given. Do not
|
||||||
|
"optimise" that away.
|
||||||
|
|
||||||
Last verified state (all green, `./jaigradle check` and `verifyPlugin` too):
|
Last verified state (all green, `./jaigradle check` and `verifyPlugin` too):
|
||||||
|
|
||||||
```text
|
```text
|
||||||
@@ -59,13 +65,17 @@ dev.hgh.jai.editor.JaiEditorSupportTest tests=5
|
|||||||
dev.hgh.jai.highlighting.* tests=8
|
dev.hgh.jai.highlighting.* tests=8
|
||||||
dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate
|
dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate
|
||||||
dev.hgh.jai.lexer.JaiLexerTest tests=13
|
dev.hgh.jai.lexer.JaiLexerTest tests=13
|
||||||
-> total 33, failures+errors 0
|
dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate
|
||||||
|
dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees
|
||||||
|
dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert
|
||||||
|
-> total 41, failures+errors 0
|
||||||
```
|
```
|
||||||
|
|
||||||
The corpus gate reports what it actually did; check the line is still there:
|
The corpus gates report what they actually did; check both lines are still there:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
||||||
|
Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements total.
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -93,6 +103,10 @@ Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
|||||||
- **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement),
|
- **Phase 2** — `JaiSyntaxHighlighter` (+ highlight-only lexer refinement),
|
||||||
colour settings page, commenter, brace matcher. Plugin Verifier passes
|
colour settings page, commenter, brace matcher. Plugin Verifier passes
|
||||||
against IU-253/261/262.
|
against IU-253/261/262.
|
||||||
|
- **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in
|
||||||
|
`src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and
|
||||||
|
the Tier 3 corpus gate are green at **91.9% (656/714 files parse with zero
|
||||||
|
`PsiErrorElement`)**.
|
||||||
|
|
||||||
### Lexer design facts worth knowing before touching it
|
### Lexer design facts worth knowing before touching it
|
||||||
|
|
||||||
@@ -107,17 +121,67 @@ Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
|||||||
- **Built-in type names and `it`/`it_index` are IDENT**, refined into separate
|
- **Built-in type names and `it`/`it_index` are IDENT**, refined into separate
|
||||||
token types only by `JaiHighlightingLexer`, which the parser never sees. Do not
|
token types only by `JaiHighlightingLexer`, which the parser never sees. Do not
|
||||||
reserve them (language reference §14.9).
|
reserve them (language reference §14.9).
|
||||||
|
- **A backslash inside an identifier is a continuation.** The compiler's lexer
|
||||||
|
(`Jai_Lexer/module.jai:444`) eats the `\` and any spaces after it and keeps
|
||||||
|
appending, so `left\_margin` is one identifier. The token span still covers the
|
||||||
|
backslash, so the Tier 0 round-trip holds.
|
||||||
|
- **`#string,\% ID` exists** in `how_to/018_print_functions.jai` even though the
|
||||||
|
shipped `Jai_Lexer` module rejects anything but `,cr`. Our `scanHereString`
|
||||||
|
accepts any non-space modifier run. When the module and real code disagree,
|
||||||
|
real code wins.
|
||||||
- Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build.
|
- Kotlin block comments nest too: writing `/*` inside a KDoc breaks the build.
|
||||||
|
|
||||||
|
### Parser design facts worth knowing before touching it
|
||||||
|
|
||||||
|
- **Generated sources are wiped and regenerated on every build.** `compileJava`
|
||||||
|
and `compileKotlin` depend on `generateParser`, and a `Delete` task clears
|
||||||
|
`src/main/gen` first. Editing `Jai.bnf` is enough; never run a bare compile and
|
||||||
|
trust it. Both halves of that were real bugs:
|
||||||
|
- without the dependency, a `Jai.bnf` edit compiled against the *previous*
|
||||||
|
parser and the tests reported green (the `.class` was minutes older than its
|
||||||
|
source);
|
||||||
|
- without the wipe, Grammar-Kit's partial regeneration left
|
||||||
|
`JaiDeclaration.getArgumentList()` with no override in the Impl and the build
|
||||||
|
failed inside generated code.
|
||||||
|
- **Grammar-Kit mints its own token instances** from the `tokens` block. Those
|
||||||
|
are different objects from the ones `JaiLexer` emits, so every rule silently
|
||||||
|
fails to match and a whole file becomes one error element. `tokenTypeFactory`
|
||||||
|
points at `JaiTokenTypes.byName`, which resolves them reflectively.
|
||||||
|
- **What BNF cannot express lives in `JaiParserUtil`**: directive-name tests
|
||||||
|
(`#ident` is one token), `==` before `{` for the switch form, procedure header
|
||||||
|
vs parenthesised expression, and the declaration lookahead — Jai declarations
|
||||||
|
have no introducer keyword.
|
||||||
|
- **A rule that creates a PSI element inside an expression can truncate it.**
|
||||||
|
Using `initializer` for a named return default made the enclosing
|
||||||
|
`procLiteralExpr` end early, so `-> a: int = 1 { }` silently lost its body. The
|
||||||
|
private `returnDefault_` rule exists for that reason — do not "simplify" it.
|
||||||
|
- **Longest-first matters for multi-token sequences only.** `operator *[]` needs
|
||||||
|
`'*' '[' ']'` before `'*'`. Single-token operators cannot collide, because `+=`
|
||||||
|
is one token.
|
||||||
|
- **Directive flags require adjacency.** `#library,system` is a flag;
|
||||||
|
`(cb: (*GUID) #c_call, ctx: *void)` is a parameter separator. Without the
|
||||||
|
adjacency check the flag swallowed `, ctx`.
|
||||||
|
|
||||||
### Not done — pick up here
|
### Not done — pick up here
|
||||||
|
|
||||||
1. **Phase 3: parser.** `.bnf` grammar → Grammar-Kit generated parser + PSI in
|
1. **Finish the Tier 3 corpus gate.** 58 of 714 files still produce a
|
||||||
`src/main/gen`, plus a `ParserDefinition`. Gate: Tier 3 corpus parse with zero
|
`PsiErrorElement`, all long-tail: each construct below appears in one or two
|
||||||
`PsiErrorElement` (allowlist committed and shrinking).
|
files. `JaiCorpusParserTest` prints a histogram of the token at each failure
|
||||||
Note `JaiFileTypeTest` documents the one thing blocked on this: until a
|
point plus a spread of sample errors; that output is the work list.
|
||||||
`ParserDefinition` exists, PSI files for `.jai` are plain text, so
|
`MIN_CLEAN_FILES` is the ratchet — raise it, never lower it. Known gaps:
|
||||||
PSI-dependent features (comment action, structure view) cannot be tested.
|
- mixed declare/assign target lists: `success=, output, error := adb(…)` and
|
||||||
2. **Phases 4+** — see the plan.
|
`renderer:, success = make_renderer(…)`;
|
||||||
|
- `using,except SKIP_THESE name: T;` (a bare identifier operand, where
|
||||||
|
`using,except(x)` and `using,except .["x"]` already work);
|
||||||
|
- `#asm` bodies are consumed opaquely, so nothing inside them has PSI.
|
||||||
|
|
||||||
|
Decide explicitly whether the remainder becomes a committed allowlist (the
|
||||||
|
plan's §3 suggestion) or gets fixed.
|
||||||
|
2. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution
|
||||||
|
and go-to-definition. The PSI is in place, so these are now unblocked.
|
||||||
|
`JaiFileTypeTest` used to document being blocked on the `ParserDefinition`;
|
||||||
|
that no longer applies.
|
||||||
|
3. **Phases 5+** — see the plan.
|
||||||
|
|
||||||
### Open questions for the user (unanswered)
|
### Open questions for the user (unanswered)
|
||||||
|
|
||||||
@@ -159,8 +223,18 @@ This is implemented in `JaiCorpusLexerTest` and it is **green**: 714 files,
|
|||||||
the same sweep to assert every token type the corpus produces (105 of them) has
|
the same sweep to assert every token type the corpus produces (105 of them) has
|
||||||
a colour. Keep new lexer work under these gates rather than adding snippets.
|
a colour. Keep new lexer work under these gates rather than adding snippets.
|
||||||
|
|
||||||
Same idea later: zero `PsiErrorElement` across the corpus for the parser;
|
Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all
|
||||||
formatting is idempotent for the formatter.
|
714 files and counts `PsiErrorElement`s — currently **656 clean (91.9%)**. It
|
||||||
|
prints a histogram of the source text at each failure point, which is the
|
||||||
|
fastest way to find the next construct worth supporting. The idempotence sweep
|
||||||
|
for the formatter is the same idea again, later.
|
||||||
|
|
||||||
|
The fast loop for grammar work:
|
||||||
|
|
||||||
|
1. `./jaitest`, then read `JaiCorpusParserTest`'s histogram and sample errors.
|
||||||
|
2. Reduce one failure to a snippet in `DebugParseTest` (inert by default).
|
||||||
|
3. Fix `Jai.bnf` or `JaiParserUtil`, re-run, watch the percentage.
|
||||||
|
4. Raise `MIN_CLEAN_FILES` and commit.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -168,6 +242,12 @@ formatting is idempotent for the formatter.
|
|||||||
|
|
||||||
- **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused
|
- **Stale Gradle daemons.** Mixing JDK 17 and 21 daemons caused
|
||||||
`Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`.
|
`Timeout waiting to lock journal cache`. Fix: `./jaigradle --stop`.
|
||||||
|
- **Stale build outputs cost hours.** Symptoms: a `Jai.bnf` edit has no effect,
|
||||||
|
or the build fails inside generated code complaining about a method that is not
|
||||||
|
in the file you are reading. Check timestamps —
|
||||||
|
`ls -l build/classes/java/main/dev/hgh/jai/parser/JaiParser.class
|
||||||
|
src/main/gen/dev/hgh/jai/parser/JaiParser.java`. The build wiring in
|
||||||
|
`build.gradle.kts` should prevent both, but `rm -rf build` settles it.
|
||||||
- **Concurrent Gradle runs corrupt the test results.** If something else (an
|
- **Concurrent Gradle runs corrupt the test results.** If something else (an
|
||||||
IDE, an agent's background checker) runs `test` at the same time, one of the
|
IDE, an agent's background checker) runs `test` at the same time, one of the
|
||||||
two dies with `java.io.EOFException` or
|
two dies with `java.io.EOFException` or
|
||||||
@@ -204,6 +284,6 @@ formatting is idempotent for the formatter.
|
|||||||
plan decisions came from testing assumptions that turned out false.
|
plan decisions came from testing assumptions that turned out false.
|
||||||
- Work in small increments: change one thing, run `./jaitest`, commit when green.
|
- Work in small increments: change one thing, run `./jaitest`, commit when green.
|
||||||
- Keep the working tree clean — revert throwaway probes.
|
- Keep the working tree clean — revert throwaway probes.
|
||||||
- Generated sources (once Grammar-Kit is wired up) go in `src/main/gen`, are
|
- Generated sources go in `src/main/gen`, are committed, and are **never
|
||||||
committed, and are **never hand-edited**; regenerate instead.
|
hand-edited**. They regenerate on every build from `src/main/grammar/Jai.bnf`.
|
||||||
- Update this file when project state changes.
|
- Update this file when project state changes.
|
||||||
|
|||||||
@@ -167,7 +167,7 @@ Each phase has a machine-checkable gate. Do not advance without a green gate.
|
|||||||
| **0** | Env fix, git, AGENTS.md, template stripped | `./jaigradle check` green with no `JAVA_HOME` |
|
| **0** | Env fix, git, AGENTS.md, template stripped | `./jaigradle check` green with no `JAVA_HOME` |
|
||||||
| **1** | File type, icon, `JaiTokenTypes`, hand-written lexer | Tier 0 corpus invariants pass on all 714 files |
|
| **1** | File type, icon, `JaiTokenTypes`, hand-written lexer | Tier 0 corpus invariants pass on all 714 files |
|
||||||
| **2** | `SyntaxHighlighter`, color settings page, commenter, brace matcher | Tier 1 golden dumps; highlighter maps every token type |
|
| **2** | `SyntaxHighlighter`, color settings page, commenter, brace matcher | Tier 1 golden dumps; highlighter maps every token type |
|
||||||
| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target |
|
| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target — **done, 91.9% (656/714)** |
|
||||||
| **4** | Structure view, folding, `#import`/`#load` reference resolution + go-to-definition | Tier 4 fixture tests |
|
| **4** | Structure view, folding, `#import`/`#load` reference resolution + go-to-definition | Tier 4 fixture tests |
|
||||||
| **5** | Completion (keywords, directives, module names), rename, find-usages | Tier 4 fixture tests |
|
| **5** | Completion (keywords, directives, module names), rename, find-usages | Tier 4 fixture tests |
|
||||||
| **6** | Formatter, code style settings | Formatter round-trip: formatting the corpus is idempotent |
|
| **6** | Formatter, code style settings | Formatter round-trip: formatting the corpus is idempotent |
|
||||||
|
|||||||
Reference in New Issue
Block a user