Phase 3: Grammar-Kit parser, PSI and ParserDefinition
Adds src/main/grammar/Jai.bnf, the generated parser/PSI in src/main/gen, a
JaiParserDefinition, and the Tier 3 corpus parse gate (406/714 files clean).
Two things were not obvious:
- Grammar-Kit mints its own token instances from the .bnf 'tokens' block, which
are different objects from the ones JaiLexer emits, so every rule silently
failed to match. Fixed with tokenTypeFactory -> JaiTokenTypes.byName.
- A backslash inside an identifier is a continuation in the compiler's lexer
(Jai_Lexer/module.jai:444): 'left\_margin' is one identifier. JaiLexer now
does the same; the Tier 0 round-trip still holds because the token span
covers the backslash and the skipped spaces.
Constructs that BNF alone cannot express live in JaiParserUtil: directive-name
tests (#ident is one token), '==' before '{' for the switch form, procedure
header vs parenthesised expression, and the declaration lookahead.
This commit is contained in:
@@ -78,6 +78,10 @@ Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
|
||||
`~/.local/jai`, keywords/tokens transcribed from the compiler's own lexer.
|
||||
**Read this before writing any lexer or parser code.**
|
||||
- `docs/BUILD_PLAN.md` — phased plan, testing strategy, risks, open questions.
|
||||
- Local IntelliJ reference projects and documentation in `~/programming/thirdparty`:
|
||||
- `fortran-plugin` — a simple language plugin to use as reference.
|
||||
- `intellij-elixir` — a more complex plugin to use as reference.
|
||||
- `intellij-sdk-docs` — local versions of the IntelliJ SDK documentation.
|
||||
- `mise.toml` → `java = "temurin-21"` (was 17, which was broken).
|
||||
- `jaigradle` — JDK-resolving Gradle wrapper. Executable, verified.
|
||||
- `jaitest` — runs the suite and reports the JUnit XML (see above).
|
||||
@@ -131,9 +135,8 @@ Both are justified in `docs/BUILD_PLAN.md` §2. Do not silently reverse them.
|
||||
Nested block comments need a depth counter. Both would end up as hand-written
|
||||
Java inside JFlex actions anyway.
|
||||
- **Grammar-Kit BNF for parser + PSI.** Verified headless: the plugin ID is
|
||||
`org.jetbrains.intellij.platform.grammarkit`, bundled in the IntelliJ Platform
|
||||
Gradle Plugin since 2.12.0 (we run 2.18.1). `generateLexer` / `generateParser`
|
||||
already appear in `./jaigradle tasks --all` — no extra dependency needed.
|
||||
`org.jetbrains.intellij.platform.grammarkit`, applied in `build.gradle.kts`.
|
||||
`generateLexer` / `generateParser` tasks are present in `./jaigradle tasks`.
|
||||
- **Ship highlighting before the parser.** Highlighting needs only the lexer.
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user