grant/intellijai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Parser: cover full Jai corpus

Browse Source
This commit is contained in:
hgranthorner
2026-08-04 13:44:59 -04:00
parent 8265389f64
commit bb54639ce5
21 changed files with 649 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
AGENTS.md
View File

@@ -67,15 +67,16 @@ dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate
dev.hgh.jai.lexer.JaiLexerTest tests=13
dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate
dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees
dev.hgh.jai.parser.JaiParserLongTailTest tests=1 <- focused parser regressions
dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert
-> total 42, failures+errors 0
-> total 43, failures+errors 0
```
The corpus gates report what they actually did; check both lines are still there:
```text
Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements total.
Tier 3: parsed 714 files, 714 clean (100.0%), 0 with errors, 0 PsiErrorElements total.
```
---
@@ -105,7 +106,7 @@ Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements
against IU-253/261/262.
- **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in
`src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and
the Tier 3 corpus gate are green at **91.9% (656/714 files parse with zero
the Tier 3 corpus gate are green at **100.0% (714/714 files parse with zero
`PsiErrorElement`)**.
### Lexer design facts worth knowing before touching it
@@ -164,24 +165,13 @@ Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements
### Not done — pick up here
1. **Finish the Tier 3 corpus gate.** 58 of 714 files still produce a
`PsiErrorElement`, all long-tail: each construct below appears in one or two
files. `JaiCorpusParserTest` prints a histogram of the token at each failure
point plus a spread of sample errors; that output is the work list.
`MIN_CLEAN_FILES` is the ratchet — raise it, never lower it. Known gaps:
- mixed declare/assign target lists: `success=, output, error := adb(…)` and
`renderer:, success = make_renderer(…)`;
- `using,except SKIP_THESE name: T;` (a bare identifier operand, where
`using,except(x)` and `using,except .["x"]` already work);
- `#asm` bodies are consumed opaquely, so nothing inside them has PSI.
Decide explicitly whether the remainder becomes a committed allowlist (the
plan's §3 suggestion) or gets fixed.
2. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution
and go-to-definition. The PSI is in place, so these are now unblocked.
`JaiFileTypeTest` used to document being blocked on the `ParserDefinition`;
that no longer applies.
3. **Phases 5+** — see the plan.
1. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution
and go-to-definition. The PSI is in place, and the Tier 3 corpus gate is now
fully green. `JaiFileTypeTest` used to document being blocked on the
`ParserDefinition`; that no longer applies.
`#asm` bodies are intentionally consumed opaquely, so nothing inside them has
PSI yet.
2. **Phases 5+** — see the plan.
### Open questions for the user (unanswered)
@@ -224,10 +214,11 @@ the same sweep to assert every token type the corpus produces (105 of them) has
a colour. Keep new lexer work under these gates rather than adding snippets.
Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all
714 files and counts `PsiErrorElement`s — currently **656 clean (91.9%)**. It
prints a histogram of the source text at each failure point, which is the
fastest way to find the next construct worth supporting. The idempotence sweep
for the formatter is the same idea again, later.
714 files and counts `PsiErrorElement`s — currently **714 clean (100.0%)**. The
ratchet is set to the full corpus size, so any parser regression fails the test.
`#asm` bodies remain opaque by design; they are tiled as one directive body but
have no inner PSI. The idempotence sweep for the formatter is the same idea
again, later.
The fast loop for grammar work: