Parser: cover full Jai corpus

This commit is contained in:
hgranthorner
2026-08-04 13:44:59 -04:00
parent 8265389f64
commit bb54639ce5
21 changed files with 649 additions and 113 deletions

View File

@@ -67,15 +67,16 @@ dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate
dev.hgh.jai.lexer.JaiLexerTest tests=13
dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate
dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees
dev.hgh.jai.parser.JaiParserLongTailTest tests=1 <- focused parser regressions
dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert
-> total 42, failures+errors 0
-> total 43, failures+errors 0
```
The corpus gates report what they actually did; check both lines are still there:
```text
Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly.
Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements total.
Tier 3: parsed 714 files, 714 clean (100.0%), 0 with errors, 0 PsiErrorElements total.
```
---
@@ -105,7 +106,7 @@ Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements
against IU-253/261/262.
- **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in
`src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and
the Tier 3 corpus gate are green at **91.9% (656/714 files parse with zero
the Tier 3 corpus gate are green at **100.0% (714/714 files parse with zero
`PsiErrorElement`)**.
### Lexer design facts worth knowing before touching it
@@ -164,24 +165,13 @@ Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements
### Not done — pick up here
1. **Finish the Tier 3 corpus gate.** 58 of 714 files still produce a
`PsiErrorElement`, all long-tail: each construct below appears in one or two
files. `JaiCorpusParserTest` prints a histogram of the token at each failure
point plus a spread of sample errors; that output is the work list.
`MIN_CLEAN_FILES` is the ratchet — raise it, never lower it. Known gaps:
- mixed declare/assign target lists: `success=, output, error := adb(…)` and
`renderer:, success = make_renderer(…)`;
- `using,except SKIP_THESE name: T;` (a bare identifier operand, where
`using,except(x)` and `using,except .["x"]` already work);
- `#asm` bodies are consumed opaquely, so nothing inside them has PSI.
Decide explicitly whether the remainder becomes a committed allowlist (the
plan's §3 suggestion) or gets fixed.
2. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution
and go-to-definition. The PSI is in place, so these are now unblocked.
`JaiFileTypeTest` used to document being blocked on the `ParserDefinition`;
that no longer applies.
3. **Phases 5+** — see the plan.
1. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution
and go-to-definition. The PSI is in place, and the Tier 3 corpus gate is now
fully green. `JaiFileTypeTest` used to document being blocked on the
`ParserDefinition`; that no longer applies.
`#asm` bodies are intentionally consumed opaquely, so nothing inside them has
PSI yet.
2. **Phases 5+** — see the plan.
### Open questions for the user (unanswered)
@@ -224,10 +214,11 @@ the same sweep to assert every token type the corpus produces (105 of them) has
a colour. Keep new lexer work under these gates rather than adding snippets.
Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all
714 files and counts `PsiErrorElement`s — currently **656 clean (91.9%)**. It
prints a histogram of the source text at each failure point, which is the
fastest way to find the next construct worth supporting. The idempotence sweep
for the formatter is the same idea again, later.
714 files and counts `PsiErrorElement`s — currently **714 clean (100.0%)**. The
ratchet is set to the full corpus size, so any parser regression fails the test.
`#asm` bodies remain opaque by design; they are tiled as one directive body but
have no inner PSI. The idempotence sweep for the formatter is the same idea
again, later.
The fast loop for grammar work:

View File

@@ -13,12 +13,11 @@
- Comment/uncomment support and brace matching, including `.{` and `.[`
- Grammar-Kit parser and PSI for the whole language: declarations, procedures,
structs, enums, unions, control flow including the `if x == { case }` switch
form, and compiler directives. 91.9% of the 714-file Jai distribution parses
with no syntax errors.
form, and compiler directives. The full 714-file Jai distribution parses with
no syntax errors; `JaiCorpusParserTest` is ratcheted at 100%.
### Known gaps
- 58 files in the Jai distribution still report a syntax error; each involves a
construct used in only one or two files. `#asm` bodies are parsed as opaque
blocks.
- `#asm` bodies are parsed as opaque blocks, so their assembly instructions do
not have inner PSI.
- Structure view, folding, and go-to-definition are not implemented yet.

View File

@@ -167,7 +167,7 @@ Each phase has a machine-checkable gate. Do not advance without a green gate.
| **0** | Env fix, git, AGENTS.md, template stripped | `./jaigradle check` green with no `JAVA_HOME` |
| **1** | File type, icon, `JaiTokenTypes`, hand-written lexer | Tier 0 corpus invariants pass on all 714 files |
| **2** | `SyntaxHighlighter`, color settings page, commenter, brace matcher | Tier 1 golden dumps; highlighter maps every token type |
| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target — **done, 91.9% (656/714)** |
| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target — **done, 100.0% (714/714)** |
| **4** | Structure view, folding, `#import`/`#load` reference resolution + go-to-definition | Tier 4 fixture tests |
| **5** | Completion (keywords, directives, module names), rename, find-usages | Tier 4 fixture tests |
| **6** | Formatter, code style settings | Formatter round-trip: formatting the corpus is idempotent |

View File

@@ -26,7 +26,14 @@ fi
run_gradle() {
rm -rf build/reports/tests/test
./jaigradle cleanTest test "${extra[@]}" "$@" 2>&1 | grep -vE '^\[[0-9.]+s\]\[warning\]\[cds\]'
# Bash 3.2 (the macOS system shell) treats an empty array expansion as an
# unset variable under `set -u`. Keep the no-extra-args invocation separate
# so the wrapper works both with and without a test filter.
if ((${#extra[@]})); then
./jaigradle cleanTest test "${extra[@]}" "$@" 2>&1 | grep -vE '^\[[0-9.]+s\]\[warning\]\[cds\]'
else
./jaigradle cleanTest test "$@" 2>&1 | grep -vE '^\[[0-9.]+s\]\[warning\]\[cds\]'
fi
return "${PIPESTATUS[0]}"
}

View File

@@ -47,7 +47,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
PROC_LITERAL_EXPR, RANGE_EXPR, REF_EXPR, RELATIONAL_EXPR,
SHIFT_EXPR, STRUCT_EXPR, STRUCT_LITERAL_EXPR, TYPED_ARRAY_LITERAL_EXPR,
TYPED_STRUCT_LITERAL_EXPR, UNARY_DOT_EXPR, UNINITIALIZED_EXPR, UNION_EXPR,
ZERO_LITERAL_EXPR),
VALUE_LITERAL_EXPR, ZERO_LITERAL_EXPR),
};
/* ********************************************************** */
@@ -215,18 +215,37 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// <<exprNoAssign>> (',' <<exprNoAssign>>)*
// <<exprNoAssign>> [':']
static boolean assignTarget_(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "assignTarget_")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = exprNoAssign(builder_, level_ + 1);
result_ = result_ && assignTarget__1(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// [':']
private static boolean assignTarget__1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "assignTarget__1")) return false;
consumeToken(builder_, COLON);
return true;
}
/* ********************************************************** */
// assignTarget_ (',' assignTarget_)*
static boolean assignTargets_(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "assignTargets_")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = exprNoAssign(builder_, level_ + 1);
result_ = assignTarget_(builder_, level_ + 1);
result_ = result_ && assignTargets__1(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// (',' <<exprNoAssign>>)*
// (',' assignTarget_)*
private static boolean assignTargets__1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "assignTargets__1")) return false;
while (true) {
@@ -237,13 +256,13 @@ public class JaiParser implements PsiParser, LightPsiParser {
return true;
}
// ',' <<exprNoAssign>>
// ',' assignTarget_
private static boolean assignTargets__1_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "assignTargets__1_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = consumeToken(builder_, COMMA);
result_ = result_ && exprNoAssign(builder_, level_ + 1);
result_ = result_ && assignTarget_(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
@@ -494,14 +513,13 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// ['`'] IDENT
// ['`'] (IDENT | keywordAsName_)
public static boolean declName(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declName")) return false;
if (!nextTokenIs(builder_, "<decl name>", BACKTICK, IDENT)) return false;
boolean result_;
Marker marker_ = enter_section_(builder_, level_, _NONE_, DECL_NAME, "<decl name>");
result_ = declName_0(builder_, level_ + 1);
result_ = result_ && consumeToken(builder_, IDENT);
result_ = result_ && declName_1(builder_, level_ + 1);
exit_section_(builder_, level_, marker_, result_, false, null);
return result_;
}
@@ -513,20 +531,29 @@ public class JaiParser implements PsiParser, LightPsiParser {
return true;
}
// IDENT | keywordAsName_
private static boolean declName_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declName_1")) return false;
boolean result_;
result_ = consumeToken(builder_, IDENT);
if (!result_) result_ = keywordAsName_(builder_, level_ + 1);
return result_;
}
/* ********************************************************** */
// declName (',' declName)*
// declName (['='] ',' declName ['='])* ['=']
public static boolean declNames(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declNames")) return false;
if (!nextTokenIs(builder_, "<decl names>", BACKTICK, IDENT)) return false;
boolean result_;
Marker marker_ = enter_section_(builder_, level_, _NONE_, DECL_NAMES, "<decl names>");
result_ = declName(builder_, level_ + 1);
result_ = result_ && declNames_1(builder_, level_ + 1);
result_ = result_ && declNames_2(builder_, level_ + 1);
exit_section_(builder_, level_, marker_, result_, false, null);
return result_;
}
// (',' declName)*
// (['='] ',' declName ['='])*
private static boolean declNames_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declNames_1")) return false;
while (true) {
@@ -537,17 +564,40 @@ public class JaiParser implements PsiParser, LightPsiParser {
return true;
}
// ',' declName
// ['='] ',' declName ['=']
private static boolean declNames_1_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declNames_1_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = consumeToken(builder_, COMMA);
result_ = declNames_1_0_0(builder_, level_ + 1);
result_ = result_ && consumeToken(builder_, COMMA);
result_ = result_ && declName(builder_, level_ + 1);
result_ = result_ && declNames_1_0_3(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// ['=']
private static boolean declNames_1_0_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declNames_1_0_0")) return false;
consumeToken(builder_, EQ);
return true;
}
// ['=']
private static boolean declNames_1_0_3(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declNames_1_0_3")) return false;
consumeToken(builder_, EQ);
return true;
}
// ['=']
private static boolean declNames_2(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "declNames_2")) return false;
consumeToken(builder_, EQ);
return true;
}
/* ********************************************************** */
// ':' [<<exprNoAssign>>] procModifier* [',' assignTargets_] [('=' | ':') initializerList_]
// | ':=' initializerList_
@@ -1274,7 +1324,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// '<=' expr | '*=' expr | '<' | '*'
// '<=' expr | '*=' expr | '<' | '*' | ','
static boolean forModifier_(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "forModifier_")) return false;
boolean result_;
@@ -1283,6 +1333,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
if (!result_) result_ = forModifier__1(builder_, level_ + 1);
if (!result_) result_ = consumeToken(builder_, LT);
if (!result_) result_ = consumeToken(builder_, STAR);
if (!result_) result_ = consumeToken(builder_, COMMA);
exit_section_(builder_, marker_, null, result_);
return result_;
}
@@ -1442,59 +1493,73 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// ifKeyword_ [<<dir "#complete">>] expr (switchBody | ['then'] statement [elseClause])
// nestedInlineIfStatement
// | inlineIfDirectiveStatement
// | ifKeyword_ [<<dir "#complete">>] expr (switchBody | ['then'] statement [elseClause])
public static boolean ifStatement(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_, level_, _NONE_, IF_STATEMENT, "<if statement>");
result_ = ifKeyword_(builder_, level_ + 1);
result_ = result_ && ifStatement_1(builder_, level_ + 1);
result_ = result_ && expr(builder_, level_ + 1, -1);
result_ = result_ && ifStatement_3(builder_, level_ + 1);
result_ = nestedInlineIfStatement(builder_, level_ + 1);
if (!result_) result_ = inlineIfDirectiveStatement(builder_, level_ + 1);
if (!result_) result_ = ifStatement_2(builder_, level_ + 1);
exit_section_(builder_, level_, marker_, result_, false, null);
return result_;
}
// ifKeyword_ [<<dir "#complete">>] expr (switchBody | ['then'] statement [elseClause])
private static boolean ifStatement_2(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_2")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = ifKeyword_(builder_, level_ + 1);
result_ = result_ && ifStatement_2_1(builder_, level_ + 1);
result_ = result_ && expr(builder_, level_ + 1, -1);
result_ = result_ && ifStatement_2_3(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// [<<dir "#complete">>]
private static boolean ifStatement_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_1")) return false;
private static boolean ifStatement_2_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_2_1")) return false;
dir(builder_, level_ + 1, "#complete");
return true;
}
// switchBody | ['then'] statement [elseClause]
private static boolean ifStatement_3(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_3")) return false;
private static boolean ifStatement_2_3(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_2_3")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = switchBody(builder_, level_ + 1);
if (!result_) result_ = ifStatement_3_1(builder_, level_ + 1);
if (!result_) result_ = ifStatement_2_3_1(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// ['then'] statement [elseClause]
private static boolean ifStatement_3_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_3_1")) return false;
private static boolean ifStatement_2_3_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_2_3_1")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = ifStatement_3_1_0(builder_, level_ + 1);
result_ = ifStatement_2_3_1_0(builder_, level_ + 1);
result_ = result_ && statement(builder_, level_ + 1);
result_ = result_ && ifStatement_3_1_2(builder_, level_ + 1);
result_ = result_ && ifStatement_2_3_1_2(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// ['then']
private static boolean ifStatement_3_1_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_3_1_0")) return false;
private static boolean ifStatement_2_3_1_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_2_3_1_0")) return false;
consumeToken(builder_, KW_THEN);
return true;
}
// [elseClause]
private static boolean ifStatement_3_1_2(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_3_1_2")) return false;
private static boolean ifStatement_2_3_1_2(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "ifStatement_2_3_1_2")) return false;
elseClause(builder_, level_ + 1);
return true;
}
@@ -1567,6 +1632,27 @@ public class JaiParser implements PsiParser, LightPsiParser {
return result_;
}
/* ********************************************************** */
// <<dir "#if">> IDENT statement [elseClause]
static boolean inlineIfDirectiveStatement(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "inlineIfDirectiveStatement")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = dir(builder_, level_ + 1, "#if");
result_ = result_ && consumeToken(builder_, IDENT);
result_ = result_ && statement(builder_, level_ + 1);
result_ = result_ && inlineIfDirectiveStatement_3(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// [elseClause]
private static boolean inlineIfDirectiveStatement_3(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "inlineIfDirectiveStatement_3")) return false;
elseClause(builder_, level_ + 1);
return true;
}
/* ********************************************************** */
// 'interface' [expr]
public static boolean interfaceExpr(PsiBuilder builder_, int level_) {
@@ -1681,18 +1767,30 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// IDENT | NUMBER | keywordAsName_
// 'operator' operatorSymbol | IDENT | NUMBER | keywordAsName_
public static boolean memberName(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "memberName")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_, level_, _NONE_, MEMBER_NAME, "<member name>");
result_ = consumeToken(builder_, IDENT);
result_ = memberName_0(builder_, level_ + 1);
if (!result_) result_ = consumeToken(builder_, IDENT);
if (!result_) result_ = consumeToken(builder_, NUMBER);
if (!result_) result_ = keywordAsName_(builder_, level_ + 1);
exit_section_(builder_, level_, marker_, result_, false, null);
return result_;
}
// 'operator' operatorSymbol
private static boolean memberName_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "memberName_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = consumeToken(builder_, KW_OPERATOR);
result_ = result_ && operatorSymbol(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
/* ********************************************************** */
// <<isNamedArgumentAhead>> memberName '=' (block | jumpArgument_ | expr)
public static boolean namedArgument(PsiBuilder builder_, int level_) {
@@ -1717,6 +1815,12 @@ public class JaiParser implements PsiParser, LightPsiParser {
return result_;
}
/* ********************************************************** */
// <<parseNestedInlineIfStatement>>
static boolean nestedInlineIfStatement(PsiBuilder builder_, int level_) {
return parseNestedInlineIfStatement(builder_, level_ + 1);
}
/* ********************************************************** */
// NOTE+
public static boolean noteStatement(PsiBuilder builder_, int level_) {
@@ -1752,7 +1856,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
/* ********************************************************** */
// '*' '[' ']' | '[' ']' '=' | '[' ']'
// | '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '<' | '>' | '<=' | '>='
// | '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '!' | '<' | '>' | '<=' | '>='
// | '&' | '|' | '^' | '~' | '<<' | '>>' | '<<<' | '>>>'
// | '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^='
// | '<<=' | '>>=' | '<<<=' | '>>>='
@@ -1770,6 +1874,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
if (!result_) result_ = consumeToken(builder_, PERCENT);
if (!result_) result_ = consumeToken(builder_, EQ_EQ);
if (!result_) result_ = consumeToken(builder_, NOT_EQ);
if (!result_) result_ = consumeToken(builder_, NOT);
if (!result_) result_ = consumeToken(builder_, LT);
if (!result_) result_ = consumeToken(builder_, GT);
if (!result_) result_ = consumeToken(builder_, LT_EQ);
@@ -1841,17 +1946,53 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// <<dirPlain>> | 'using'
// <<dirPlain>> | 'using' [directiveFlag_ [<<exprNoAssign>>]]
static boolean paramModifier_(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "paramModifier_")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = dirPlain(builder_, level_ + 1);
if (!result_) result_ = consumeToken(builder_, KW_USING);
if (!result_) result_ = paramModifier__1(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// 'using' [directiveFlag_ [<<exprNoAssign>>]]
private static boolean paramModifier__1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "paramModifier__1")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = consumeToken(builder_, KW_USING);
result_ = result_ && paramModifier__1_1(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// [directiveFlag_ [<<exprNoAssign>>]]
private static boolean paramModifier__1_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "paramModifier__1_1")) return false;
paramModifier__1_1_0(builder_, level_ + 1);
return true;
}
// directiveFlag_ [<<exprNoAssign>>]
private static boolean paramModifier__1_1_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "paramModifier__1_1_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = directiveFlag_(builder_, level_ + 1);
result_ = result_ && paramModifier__1_1_0_1(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// [<<exprNoAssign>>]
private static boolean paramModifier__1_1_0_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "paramModifier__1_1_0_1")) return false;
exprNoAssign(builder_, level_ + 1);
return true;
}
/* ********************************************************** */
// ['$' | '$$'] IDENT
public static boolean paramName(PsiBuilder builder_, int level_) {
@@ -2212,18 +2353,29 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// directiveExpr
// !<<nonProcDirectiveAhead>> directiveExpr
public static boolean procModifier(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "procModifier")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_, level_, _NONE_, PROC_MODIFIER, "<proc modifier>");
result_ = directiveExpr(builder_, level_ + 1);
result_ = procModifier_0(builder_, level_ + 1);
result_ = result_ && directiveExpr(builder_, level_ + 1);
exit_section_(builder_, level_, marker_, result_, false, null);
return result_;
}
// !<<nonProcDirectiveAhead>>
private static boolean procModifier_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "procModifier_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_, level_, _NOT_);
result_ = !nonProcDirectiveAhead(builder_, level_ + 1);
exit_section_(builder_, level_, marker_, result_, false, null);
return result_;
}
/* ********************************************************** */
// ['`'] 'push_context' directiveFlag_* [expr] block
// ['`'] 'push_context' directiveFlag_* [expr] (block | semi)
public static boolean pushContextStatement(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "pushContextStatement")) return false;
if (!nextTokenIs(builder_, "<push context statement>", BACKTICK, KW_PUSH_CONTEXT)) return false;
@@ -2233,7 +2385,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
result_ = result_ && consumeToken(builder_, KW_PUSH_CONTEXT);
result_ = result_ && pushContextStatement_2(builder_, level_ + 1);
result_ = result_ && pushContextStatement_3(builder_, level_ + 1);
result_ = result_ && block(builder_, level_ + 1);
result_ = result_ && pushContextStatement_4(builder_, level_ + 1);
exit_section_(builder_, level_, marker_, result_, false, null);
return result_;
}
@@ -2263,6 +2415,15 @@ public class JaiParser implements PsiParser, LightPsiParser {
return true;
}
// block | semi
private static boolean pushContextStatement_4(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "pushContextStatement_4")) return false;
boolean result_;
result_ = block(builder_, level_ + 1);
if (!result_) result_ = semi(builder_, level_ + 1);
return result_;
}
/* ********************************************************** */
// IDENT
public static boolean refExpr(PsiBuilder builder_, int level_) {
@@ -2447,7 +2608,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// ['`'] 'return' [expr (',' expr)*] semiWithNotes_
// ['`'] 'return' [expr (',' expr)* [',']] semiWithNotes_
public static boolean returnStatement(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "returnStatement")) return false;
if (!nextTokenIs(builder_, "<return statement>", BACKTICK, KW_RETURN)) return false;
@@ -2468,20 +2629,21 @@ public class JaiParser implements PsiParser, LightPsiParser {
return true;
}
// [expr (',' expr)*]
// [expr (',' expr)* [',']]
private static boolean returnStatement_2(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "returnStatement_2")) return false;
returnStatement_2_0(builder_, level_ + 1);
return true;
}
// expr (',' expr)*
// expr (',' expr)* [',']
private static boolean returnStatement_2_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "returnStatement_2_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = expr(builder_, level_ + 1, -1);
result_ = result_ && returnStatement_2_0_1(builder_, level_ + 1);
result_ = result_ && returnStatement_2_0_2(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
@@ -2508,6 +2670,13 @@ public class JaiParser implements PsiParser, LightPsiParser {
return result_;
}
// [',']
private static boolean returnStatement_2_0_2(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "returnStatement_2_0_2")) return false;
consumeToken(builder_, COMMA);
return true;
}
/* ********************************************************** */
// <<isReturnGroupAhead>> returnGroup
// | returnItem (',' <<notPolyParamAhead>> returnItem)*
@@ -2931,7 +3100,28 @@ public class JaiParser implements PsiParser, LightPsiParser {
}
/* ********************************************************** */
// 'while' [<<isLabelAhead>> IDENT ':'] expr statement
// '{' [argList] '}'
public static boolean valueLiteralExpr(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "valueLiteralExpr")) return false;
if (!nextTokenIs(builder_, LBRACE)) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = consumeToken(builder_, LBRACE);
result_ = result_ && valueLiteralExpr_1(builder_, level_ + 1);
result_ = result_ && consumeToken(builder_, RBRACE);
exit_section_(builder_, marker_, VALUE_LITERAL_EXPR, result_);
return result_;
}
// [argList]
private static boolean valueLiteralExpr_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "valueLiteralExpr_1")) return false;
argList(builder_, level_ + 1);
return true;
}
/* ********************************************************** */
// 'while' [<<isLabelAhead>> ['`'] IDENT (':' | ':=')] expr statement
public static boolean whileStatement(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "whileStatement")) return false;
if (!nextTokenIs(builder_, KW_WHILE)) return false;
@@ -2945,24 +3135,42 @@ public class JaiParser implements PsiParser, LightPsiParser {
return result_;
}
// [<<isLabelAhead>> IDENT ':']
// [<<isLabelAhead>> ['`'] IDENT (':' | ':=')]
private static boolean whileStatement_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "whileStatement_1")) return false;
whileStatement_1_0(builder_, level_ + 1);
return true;
}
// <<isLabelAhead>> IDENT ':'
// <<isLabelAhead>> ['`'] IDENT (':' | ':=')
private static boolean whileStatement_1_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "whileStatement_1_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = isLabelAhead(builder_, level_ + 1);
result_ = result_ && consumeTokens(builder_, 0, IDENT, COLON);
result_ = result_ && whileStatement_1_0_1(builder_, level_ + 1);
result_ = result_ && consumeToken(builder_, IDENT);
result_ = result_ && whileStatement_1_0_3(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// ['`']
private static boolean whileStatement_1_0_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "whileStatement_1_0_1")) return false;
consumeToken(builder_, BACKTICK);
return true;
}
// ':' | ':='
private static boolean whileStatement_1_0_3(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "whileStatement_1_0_3")) return false;
boolean result_;
result_ = consumeToken(builder_, COLON);
if (!result_) result_ = consumeToken(builder_, COLON_EQ);
return result_;
}
/* ********************************************************** */
// '{' '}'
public static boolean zeroLiteralExpr(PsiBuilder builder_, int level_) {
@@ -2979,7 +3187,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
// Expression root: expr
// Operator priority table:
// 0: BINARY(assignExpr)
// 1: BINARY(lambdaExpr)
// 1: POSTFIX(lambdaExpr)
// 2: BINARY(orExpr)
// 3: BINARY(andExpr)
// 4: BINARY(equalityExpr)
@@ -3034,8 +3242,8 @@ public class JaiParser implements PsiParser, LightPsiParser {
result_ = expr(builder_, level_, -1);
exit_section_(builder_, level_, marker_, ASSIGN_EXPR, result_, true, null);
}
else if (priority_ < 1 && consumeTokenSmart(builder_, QUICK_LAMBDA)) {
result_ = expr(builder_, level_, 0);
else if (priority_ < 1 && lambdaExpr_0(builder_, level_ + 1)) {
result_ = true;
exit_section_(builder_, level_, marker_, LAMBDA_EXPR, result_, true, null);
}
else if (priority_ < 2 && consumeTokenSmart(builder_, OR_OR)) {
@@ -3118,6 +3326,26 @@ public class JaiParser implements PsiParser, LightPsiParser {
return result_;
}
// '=>' (block | expr)
private static boolean lambdaExpr_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "lambdaExpr_0")) return false;
boolean result_;
Marker marker_ = enter_section_(builder_);
result_ = consumeTokenSmart(builder_, QUICK_LAMBDA);
result_ = result_ && lambdaExpr_0_1(builder_, level_ + 1);
exit_section_(builder_, marker_, null, result_);
return result_;
}
// block | expr
private static boolean lambdaExpr_0_1(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "lambdaExpr_0_1")) return false;
boolean result_;
result_ = block(builder_, level_ + 1);
if (!result_) result_ = expr(builder_, level_ + 1, -1);
return result_;
}
// <<eqEqNotSwitch>> | '!=' | '==='
private static boolean equalityExpr_0(PsiBuilder builder_, int level_) {
if (!recursion_guard_(builder_, level_, "equalityExpr_0")) return false;
@@ -3531,6 +3759,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
// | parenExpr
// | structLiteralExpr
// | arrayLiteralExpr
// | valueLiteralExpr
// | unaryDotExpr
// | backtickExpr
// | builtinCallExpr
@@ -3551,6 +3780,7 @@ public class JaiParser implements PsiParser, LightPsiParser {
if (!result_) result_ = parenExpr(builder_, level_ + 1);
if (!result_) result_ = structLiteralExpr(builder_, level_ + 1);
if (!result_) result_ = arrayLiteralExpr(builder_, level_ + 1);
if (!result_) result_ = valueLiteralExpr(builder_, level_ + 1);
if (!result_) result_ = unaryDotExpr(builder_, level_ + 1);
if (!result_) result_ = backtickExpr(builder_, level_ + 1);
if (!result_) result_ = builtinCallExpr(builder_, level_ + 1);

View File

@@ -10,7 +10,7 @@ public interface JaiIfStatement extends PsiElement {
@Nullable
JaiElseClause getElseClause();
@NotNull
@Nullable
JaiExpr getExpr();
@Nullable

View File

@@ -7,6 +7,9 @@ import com.intellij.psi.PsiElement;
public interface JaiLambdaExpr extends JaiExpr {
@Nullable
JaiBlock getBlock();
@NotNull
List<JaiExpr> getExprList();

View File

@@ -7,4 +7,7 @@ import com.intellij.psi.PsiElement;
public interface JaiMemberName extends PsiElement {
@Nullable
JaiOperatorSymbol getOperatorSymbol();
}

View File

@@ -7,7 +7,7 @@ import com.intellij.psi.PsiElement;
public interface JaiPushContextStatement extends PsiElement {
@NotNull
@Nullable
JaiBlock getBlock();
@Nullable

View File

@@ -87,6 +87,7 @@ public interface JaiTypes {
IElementType UNINITIALIZED_EXPR = new JaiElementType("UNINITIALIZED_EXPR");
IElementType UNION_EXPR = new JaiElementType("UNION_EXPR");
IElementType USING_STATEMENT = new JaiElementType("USING_STATEMENT");
IElementType VALUE_LITERAL_EXPR = new JaiElementType("VALUE_LITERAL_EXPR");
IElementType WHILE_STATEMENT = new JaiElementType("WHILE_STATEMENT");
IElementType ZERO_LITERAL_EXPR = new JaiElementType("ZERO_LITERAL_EXPR");
@@ -430,6 +431,9 @@ public interface JaiTypes {
else if (type == USING_STATEMENT) {
return new JaiUsingStatementImpl(node);
}
else if (type == VALUE_LITERAL_EXPR) {
return new JaiValueLiteralExprImpl(node);
}
else if (type == WHILE_STATEMENT) {
return new JaiWhileStatementImpl(node);
}

View File

@@ -0,0 +1,13 @@
// This is a generated file. Not intended for manual editing.
package dev.hgh.jai.psi;
import java.util.List;
import org.jetbrains.annotations.*;
import com.intellij.psi.PsiElement;
public interface JaiValueLiteralExpr extends JaiExpr {
@NotNull
List<JaiArgument> getArgumentList();
}

View File

@@ -319,6 +319,10 @@ public class JaiVisitor extends PsiElementVisitor {
visitPsiElement(o);
}
public void visitValueLiteralExpr(@NotNull JaiValueLiteralExpr o) {
visitExpr(o);
}
public void visitWhileStatement(@NotNull JaiWhileStatement o) {
visitPsiElement(o);
}

View File

@@ -34,9 +34,9 @@ public class JaiIfStatementImpl extends ASTWrapperPsiElement implements JaiIfSta
}
@Override
@NotNull
@Nullable
public JaiExpr getExpr() {
return findNotNullChildByClass(JaiExpr.class);
return findChildByClass(JaiExpr.class);
}
@Override

View File

@@ -27,6 +27,12 @@ public class JaiLambdaExprImpl extends JaiExprImpl implements JaiLambdaExpr {
else super.accept(visitor);
}
@Override
@Nullable
public JaiBlock getBlock() {
return findChildByClass(JaiBlock.class);
}
@Override
@NotNull
public List<JaiExpr> getExprList() {

View File

@@ -27,4 +27,10 @@ public class JaiMemberNameImpl extends ASTWrapperPsiElement implements JaiMember
else super.accept(visitor);
}
@Override
@Nullable
public JaiOperatorSymbol getOperatorSymbol() {
return findChildByClass(JaiOperatorSymbol.class);
}
}

View File

@@ -28,9 +28,9 @@ public class JaiPushContextStatementImpl extends ASTWrapperPsiElement implements
}
@Override
@NotNull
@Nullable
public JaiBlock getBlock() {
return findNotNullChildByClass(JaiBlock.class);
return findChildByClass(JaiBlock.class);
}
@Override

View File

@@ -0,0 +1,36 @@
// This is a generated file. Not intended for manual editing.
package dev.hgh.jai.psi.impl;
import java.util.List;
import org.jetbrains.annotations.*;
import com.intellij.lang.ASTNode;
import com.intellij.psi.PsiElement;
import com.intellij.psi.PsiElementVisitor;
import com.intellij.psi.util.PsiTreeUtil;
import static dev.hgh.jai.psi.JaiTypes.*;
import dev.hgh.jai.psi.*;
public class JaiValueLiteralExprImpl extends JaiExprImpl implements JaiValueLiteralExpr {
public JaiValueLiteralExprImpl(@NotNull ASTNode node) {
super(node);
}
@Override
public void accept(@NotNull JaiVisitor visitor) {
visitor.visitValueLiteralExpr(this);
}
@Override
public void accept(@NotNull PsiElementVisitor visitor) {
if (visitor instanceof JaiVisitor) accept((JaiVisitor)visitor);
else super.accept(visitor);
}
@Override
@NotNull
public List<JaiArgument> getArgumentList() {
return PsiTreeUtil.getChildrenOfTypeAsList(this, JaiArgument.class);
}
}

View File

@@ -181,7 +181,11 @@ directiveStatement ::= <<dirBare>> directiveFlag_* [';']
// `#if` is spelled as a directive but is structurally an `if`, and it pairs with the
// ordinary `else` keyword (see modules/File_Watcher/module.jai).
ifStatement ::= ifKeyword_ [<<dir "#complete">>] expr (switchBody | ['then'] statement [elseClause])
ifStatement ::= nestedInlineIfStatement
| inlineIfDirectiveStatement
| ifKeyword_ [<<dir "#complete">>] expr (switchBody | ['then'] statement [elseClause])
private nestedInlineIfStatement ::= <<parseNestedInlineIfStatement>>
private inlineIfDirectiveStatement ::= <<dir "#if">> IDENT statement [elseClause]
private ifKeyword_ ::= 'if' | <<dir "#if">>
elseClause ::= 'else' statement
@@ -190,21 +194,21 @@ elseClause ::= 'else' statement
switchBody ::= '==' '{' caseClause* '}'
caseClause ::= 'case' [expr] [';'] statement*
whileStatement ::= 'while' [<<isLabelAhead>> IDENT ':'] expr statement
whileStatement ::= 'while' [<<isLabelAhead>> ['`'] IDENT (':' | ':=')] expr statement
// `for < * name, index : iterable` — `<` reverses, `*` iterates by pointer, and
// `for :expansion` selects a named for_expansion.
forStatement ::= 'for' [':' expr] forModifier_* [<<isForNamesAhead>> forNames ':'] expr statement
private forModifier_ ::= '<=' expr | '*=' expr | '<' | '*'
private forModifier_ ::= '<=' expr | '*=' expr | '<' | '*' | ','
forNames ::= forName (',' forName)*
forName ::= ['`'] IDENT
returnStatement ::= ['`'] 'return' [expr (',' expr)*] semiWithNotes_
returnStatement ::= ['`'] 'return' [expr (',' expr)* [',']] semiWithNotes_
breakStatement ::= 'break' [IDENT] semi
continueStatement ::= 'continue' [IDENT] semi
removeStatement ::= 'remove' [expr] semi
deferStatement ::= ['`'] 'defer' statement
pushContextStatement ::= ['`'] 'push_context' directiveFlag_* [expr] block
pushContextStatement ::= ['`'] 'push_context' directiveFlag_* [expr] (block | semi)
usingStatement ::= 'using' [',' expr] expr semi
expressionStatement ::= expr (',' expr)* semiWithNotes_
@@ -221,8 +225,8 @@ private declModifier_ ::= <<dirWithBlock>> directiveFlag_* block
| <<dirPlain>> directiveFlag_* [parenExpr | STRING]
| usingModifier_
private usingModifier_ ::= 'using' [directiveFlag_ [<<exprNoAssign>>]]
declNames ::= declName (',' declName)*
declName ::= ['`'] IDENT
declNames ::= declName (['='] ',' declName ['='])* ['=']
declName ::= ['`'] (IDENT | keywordAsName_)
private declTail_ ::= ':' [<<exprNoAssign>>] procModifier* [',' assignTargets_] [('=' | ':') initializerList_]
| ':=' initializerList_
| '::' initializerList_
@@ -230,7 +234,8 @@ private declTail_ ::= ':' [<<exprNoAssign>>] procModifier* [',' assignTargets_]
private initializerList_ ::= initializer (',' initializer)*
// `success:, apk.zip_dir = load_zip_directory(path);` declares the first name and
// assigns to the remaining lvalues.
private assignTargets_ ::= <<exprNoAssign>> (',' <<exprNoAssign>>)*
private assignTargets_ ::= assignTarget_ (',' assignTarget_)*
private assignTarget_ ::= <<exprNoAssign>> [':']
initializer ::= '---' | expr
// `operator + :: (a: T, b: T) -> T { }`
@@ -238,7 +243,7 @@ operatorDeclaration ::= 'operator' operatorSymbol '::' expr semiWithNotes_
// Longest first: `*` would otherwise match the start of `*[]`, and `[]` the start
// of `[]=`. Single-token operators cannot collide, since `+=` is one token.
operatorSymbol ::= '*' '[' ']' | '[' ']' '=' | '[' ']'
| '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '<' | '>' | '<=' | '>='
| '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '!' | '<' | '>' | '<=' | '>='
| '&' | '|' | '^' | '~' | '<<' | '>>' | '<<<' | '>>>'
| '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^='
| '<<=' | '>>=' | '<<<=' | '>>>='
@@ -278,7 +283,7 @@ assignExpr ::= expr assignOp_ expr { rightAssociative=true }
private assignOp_ ::= '=' | '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^='
| '<<=' | '>>=' | '<<<=' | '>>>=' | '&&=' | '||='
lambdaExpr ::= expr '=>' expr { rightAssociative=true }
lambdaExpr ::= expr '=>' (block | expr) { rightAssociative=true }
orExpr ::= expr '||' expr
andExpr ::= expr '&&' expr
@@ -321,7 +326,7 @@ derefExpr ::= expr '.*'
typedStructLiteralExpr ::= expr '.{' [argList] '}'
typedArrayLiteralExpr ::= expr '.[' [argList] ']'
memberName ::= IDENT | NUMBER | keywordAsName_
memberName ::= 'operator' operatorSymbol | IDENT | NUMBER | keywordAsName_
// A few keywords double as field names in the distribution (`context.print_style`,
// `.type`, `it.operator`), so member access accepts them.
private keywordAsName_ ::= 'if' | 'for' | 'case' | 'enum' | 'cast' | 'union' | 'struct'
@@ -345,6 +350,7 @@ primaryExpr ::= procLiteralExpr
| parenExpr
| structLiteralExpr
| arrayLiteralExpr
| valueLiteralExpr
| unaryDotExpr
| backtickExpr
| builtinCallExpr
@@ -360,6 +366,7 @@ uninitializedExpr ::= '---' | '--'
// `table.* = {};` zero-initialises. Only the empty form is an expression; a
// non-empty `{ … }` in expression position would collide with a block statement.
zeroLiteralExpr ::= '{' '}'
valueLiteralExpr ::= '{' [argList] '}'
backtickExpr ::= '`' (IDENT | 'defer' | 'return' | 'push_context' | 'operator')
unaryDotExpr ::= '.' memberName
structLiteralExpr ::= '.{' [argList] '}'
@@ -378,7 +385,7 @@ procLiteralExpr ::= <<isProcHeaderAhead>> '(' [paramList] ')' ['->' returnTypes_
private paramList ::= parameter (',' parameter)* [',']
parameter ::= paramModifier_* parameterTail_
private paramModifier_ ::= <<dirPlain>> | 'using'
private paramModifier_ ::= <<dirPlain>> | 'using' [directiveFlag_ [<<exprNoAssign>>]]
private parameterTail_ ::= <<isParamNameAhead>> paramNames paramTypeTail_ | '..' | expr
paramNames ::= paramName (',' paramName)*
paramName ::= ['$' | '$$'] IDENT
@@ -399,7 +406,7 @@ private returnDefault_ ::= '---' | <<exprNoAssign>>
// `#foreign lib`, `#modify { ... }`, `#c_call`, `#expand`, …
// `#foreign lib "symbol"`, `#modify { ... }`, `#c_call`, `#expand`, `#align 8` — the
// same shape as any other directive, so there is one rule for both positions.
procModifier ::= directiveExpr
procModifier ::= !<<nonProcDirectiveAhead>> directiveExpr
structExpr ::= 'struct' ['(' [paramList] ')'] structModifier_* block
enumExpr ::= ('enum' | 'enum_flags') [!'{' expr] structModifier_* block

View File

@@ -37,7 +37,6 @@ object JaiParserUtil : GeneratedParserUtilBase() {
"insert",
"code",
"run_and_insert",
"compile_time",
"bytes",
"no_reset",
)
@@ -79,6 +78,9 @@ object JaiParserUtil : GeneratedParserUtilBase() {
"poke_name",
"dump",
"foreign",
"compiler",
"intrinsic",
"program_export",
"define",
"include",
"undef",
@@ -100,7 +102,40 @@ object JaiParserUtil : GeneratedParserUtilBase() {
* that declaration's scope. Taken from the corpus, where only these four appear
* directly in front of a `name :` / `name ::` / `name :=`.
*/
private val DIRECTIVES_ON_DECLARATIONS = setOf("as", "overlay", "add_context", "no_reset")
private val DIRECTIVES_ON_DECLARATIONS =
setOf("as", "overlay", "add_context", "no_reset", "program_export")
/** Scope switches are statements, never procedure modifiers. */
private val NON_PROC_DIRECTIVES = setOf("scope_file", "scope_module", "scope_export")
/** Keyword tokens accepted by the grammar's `keywordAsName_` name rule. */
private val DECL_NAME_KEYWORDS =
setOf(
T.KW_IF,
T.KW_FOR,
T.KW_CASE,
T.KW_ENUM,
T.KW_CAST,
T.KW_UNION,
T.KW_STRUCT,
T.KW_REMOVE,
T.KW_INLINE,
T.KW_CONTEXT,
T.KW_OPERATOR,
T.KW_INTERFACE,
T.KW_TYPE_INFO,
T.KW_USING,
T.KW_DEFER,
T.KW_RETURN,
T.KW_BREAK,
T.KW_CONTINUE,
T.KW_THEN,
T.KW_ELSE,
T.KW_WHILE,
T.KW_NULL,
T.KW_TRUE,
T.KW_FALSE,
)
// ---------------------------------------------------------------- directives
@@ -154,6 +189,22 @@ object JaiParserUtil : GeneratedParserUtilBase() {
return true
}
/**
* True when the current directive is a scope switch, which is a statement
* rather than a procedure modifier. Keeping this as a lookahead predicate lets
* the grammar's normal `directiveExpr` consume the modifier's operand, such
* as the library name in `#foreign kernel32`.
*/
@JvmStatic
fun nonProcDirectiveAhead(
b: PsiBuilder,
level: Int,
): Boolean {
if (b.tokenType !== T.DIRECTIVE) return false
val text = b.tokenText ?: return false
return text.removePrefix("#") in NON_PROC_DIRECTIVES
}
/**
* A directive that takes neither an operand nor a block, so it stands alone as a
* statement without a `;` — `#scope_file`, `#no_padding`, `#compiler`.
@@ -166,7 +217,12 @@ object JaiParserUtil : GeneratedParserUtilBase() {
if (b.tokenType !== T.DIRECTIVE) return false
val text = b.tokenText ?: return false
val name = text.removePrefix("#")
if (name in DIRECTIVES_WITH_OPERAND || name in DIRECTIVES_WITH_BLOCK) return false
if (name in DIRECTIVES_WITH_BLOCK) return false
if (name in DIRECTIVES_WITH_OPERAND &&
!(name == "program_export" && b.lookAhead(1) !== T.STRING)
) {
return false
}
b.advanceLexer()
return true
}
@@ -345,6 +401,8 @@ object JaiParserUtil : GeneratedParserUtilBase() {
* `#as using,except(vtable) iunknown: IUnknown;` and the macro-exported
* `` `it := entry.value; ``.
*/
private fun isDeclarationNameToken(token: IElementType?): Boolean = token === T.IDENT || token in DECL_NAME_KEYWORDS
@JvmStatic
fun isDeclarationAhead(
b: PsiBuilder,
@@ -352,17 +410,24 @@ object JaiParserUtil : GeneratedParserUtilBase() {
): Boolean {
var i = 0
var inPrefix = false
var sawUsingPrefix = false
loop@ while (true) {
when (b.lookAhead(i)) {
T.DIRECTIVE -> {
// After `using,except`, a directive is the operand of the
// using modifier (`using,except #run ... name: T`), not a
// second declaration modifier.
if (sawUsingPrefix && !inPrefix) break@loop
if (!isDeclarationDirective(b, i)) break@loop
i++
inPrefix = true
if (b.lookAhead(i) === T.STRING) i++
}
T.KW_USING -> {
i++
inPrefix = true
sawUsingPrefix = true
}
T.LPAREN, T.LBRACKET, T.BEGIN_ARRAY_LITERAL, T.BEGIN_STRUCT_LITERAL -> {
@@ -376,6 +441,17 @@ object JaiParserUtil : GeneratedParserUtilBase() {
if (!inPrefix || b.lookAhead(i + 1) !== T.IDENT) break@loop
i += 2
inPrefix = false
// `using,except SKIP_THESE orientation: T` has a bare
// identifier as the flag's operand. The declaration grammar
// consumes it through `exprNoAssign`; skip it here so the
// actual field name is what declNames sees.
if (sawUsingPrefix &&
b.lookAhead(i) === T.IDENT &&
b.lookAhead(i + 1) === T.IDENT &&
b.lookAhead(i + 2) === T.COLON
) {
i++
}
}
else -> {
@@ -383,14 +459,66 @@ object JaiParserUtil : GeneratedParserUtilBase() {
}
}
}
// A using-except operand can be a directive expression, for example:
// `using,except #run names_to_skip(Type) business: Business;`. The
// prefix loop intentionally stops at that directive, so find the first
// declaration name after its (possibly grouped) operand.
if (sawUsingPrefix && !isDeclarationNameToken(b.lookAhead(i))) {
var candidate = i
while (candidate <= MAX_LOOKAHEAD) {
when (b.lookAhead(candidate)) {
T.LPAREN, T.LBRACKET, T.BEGIN_ARRAY_LITERAL, T.BEGIN_STRUCT_LITERAL -> {
candidate = skipBalancedGroup(b, candidate)
if (candidate < 0) return false
continue
}
T.SEMICOLON, T.RBRACE -> {
return false
}
else -> {}
}
if (isDeclarationNameToken(b.lookAhead(candidate)) &&
b.lookAhead(candidate + 1) in setOf(T.COLON, T.COLON_COLON, T.COLON_EQ)
) {
return true
}
candidate++
}
}
while (true) {
if (b.lookAhead(i) === T.BACKTICK) i++
if (b.lookAhead(i) !== T.IDENT) return false
if (!isDeclarationNameToken(b.lookAhead(i))) return false
i++
when (b.lookAhead(i)) {
T.COLON, T.COLON_COLON, T.COLON_EQ -> return true
T.COMMA -> i++
else -> return false
T.COLON, T.COLON_COLON, T.COLON_EQ -> {
return true
}
// In a mixed declaration/assignment list, an existing lvalue is
// marked with `=` before the next declared name:
// `success, output=, error := adb(...)`.
T.EQ -> {
when (b.lookAhead(i + 1)) {
T.COMMA -> i += 2
// The final assigned target may be followed directly by
// the declaration's `:=` tail:
// `target, existing= := make_target()`.
T.COLON_EQ -> return true
else -> return false
}
}
T.COMMA -> {
i++
}
else -> {
return false
}
}
}
}
@@ -526,12 +654,43 @@ object JaiParserUtil : GeneratedParserUtilBase() {
return t !== T.DOLLAR && t !== T.DOUBLE_DOLLAR
}
/** `IDENT ':'` — a loop label. */
/** Consumes `if (condition) #if FLAG statement` as one nested statement. */
@JvmStatic
fun parseNestedInlineIfStatement(
b: PsiBuilder,
level: Int,
): Boolean {
val marker = b.mark()
fun fail(): Boolean {
marker.rollbackTo()
return false
}
if (b.tokenType !== T.KW_IF || b.lookAhead(1) !== T.LPAREN) return fail()
b.advanceLexer()
if (!JaiParser.parenExpr(b, level + 1)) return fail()
if (b.tokenType !== T.DIRECTIVE || b.tokenText != "#if") return fail()
b.advanceLexer()
if (b.tokenType !== T.IDENT) return fail()
b.advanceLexer()
if (!JaiParser.statement(b, level + 1)) return fail()
marker.drop()
return true
}
/** `IDENT ':'` or `IDENT ':='` — a loop label. */
@JvmStatic
fun isLabelAhead(
b: PsiBuilder,
level: Int,
): Boolean = b.tokenType === T.IDENT && b.lookAhead(1) === T.COLON
): Boolean {
val nameOffset = if (b.tokenType === T.BACKTICK) 1 else 0
if (b.lookAhead(nameOffset) !== T.IDENT) return false
val separator = b.lookAhead(nameOffset + 1)
return separator === T.COLON || separator === T.COLON_EQ
}
private const val MAX_LOOKAHEAD = 4096

View File

@@ -106,7 +106,7 @@ class JaiCorpusParserTest : BasePlatformTestCase() {
private companion object {
/** Ratchet. Raise it as the grammar improves; never lower it. */
const val MIN_CLEAN_FILES = 650
const val MIN_CLEAN_FILES = 714
const val REPORTED_ERRORS = 25
const val REPORTED_KINDS = 20
const val GOT_WIDTH = 14

View File

@@ -0,0 +1,68 @@
package dev.hgh.jai.parser
import com.intellij.psi.PsiErrorElement
import com.intellij.psi.PsiFileFactory
import com.intellij.psi.PsiRecursiveElementWalkingVisitor
import com.intellij.testFramework.fixtures.BasePlatformTestCase
import dev.hgh.jai.JaiLanguage
/**
* Focused regressions for syntax recovered while closing the corpus gate.
*
* [JaiCorpusParserTest] proves acceptance across the full local distribution;
* these small examples make the individual grammar contracts easy to exercise
* while iterating on generated PSI.
*/
class JaiParserLongTailTest : BasePlatformTestCase() {
fun testLongTailConstructsParseWithoutErrors() {
val snippets =
listOf(
"result=, output, error := call();",
"""
Thing :: struct {
using,except SKIP_THESE field: Field;
}
proc :: (using,except(x,y,z) value: Value) {}
""".trimIndent(),
"""
factory :: () {
callback = p => { release(p); };
result := { callback, p };
}
""".trimIndent(),
"""
while `loop := ready { break loop; }
if #compile_time { report(); } else { fallback(); }
if (release) #if ENABLE_STATISTICS report();
""".trimIndent(),
"""
operator ! :: inline (value: Flag) -> bool { return value; }
push_context,defer_pop ctx;
return first, second,;
""".trimIndent(),
)
val factory = PsiFileFactory.getInstance(project)
for ((index, text) in snippets.withIndex()) {
val file = factory.createFileFromText("long-tail-$index.jai", JaiLanguage, text)
val errors = errorsIn(file)
assertTrue(
"Long-tail snippet $index produced parser errors: ${errors.joinToString { it.errorDescription }}\n$text",
errors.isEmpty(),
)
}
}
private fun errorsIn(file: com.intellij.psi.PsiFile): List<PsiErrorElement> {
val errors = mutableListOf<PsiErrorElement>()
file.accept(
object : PsiRecursiveElementWalkingVisitor() {
override fun visitErrorElement(element: PsiErrorElement) {
errors.add(element)
super.visitErrorElement(element)
}
},
)
return errors
}
}