diff --git a/AGENTS.md b/AGENTS.md index 3e55050..4ae5d22 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -67,15 +67,16 @@ dev.hgh.jai.lexer.JaiCorpusLexerTest tests=2 <- the Tier 0 gate dev.hgh.jai.lexer.JaiLexerTest tests=13 dev.hgh.jai.parser.JaiCorpusParserTest tests=1 <- the Tier 3 gate dev.hgh.jai.parser.JaiParserGoldenTest tests=5 <- Tier 2 golden trees +dev.hgh.jai.parser.JaiParserLongTailTest tests=1 <- focused parser regressions dev.hgh.jai.parser.DebugParseTest tests=2 <- scratch harness, inert --> total 42, failures+errors 0 +-> total 43, failures+errors 0 ``` The corpus gates report what they actually did; check both lines are still there: ```text Tier 0: lexed 714 files, 17154195 chars, 3009381 tokens cleanly. -Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements total. +Tier 3: parsed 714 files, 714 clean (100.0%), 0 with errors, 0 PsiErrorElements total. ``` --- @@ -105,7 +106,7 @@ Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements against IU-253/261/262. - **Phase 3** — `src/main/grammar/Jai.bnf` → Grammar-Kit parser and PSI in `src/main/gen`, `JaiParserDefinition`, `JaiParserUtil`. Tier 2 golden trees and - the Tier 3 corpus gate are green at **91.9% (656/714 files parse with zero + the Tier 3 corpus gate are green at **100.0% (714/714 files parse with zero `PsiErrorElement`)**. ### Lexer design facts worth knowing before touching it @@ -164,24 +165,13 @@ Tier 3: parsed 714 files, 656 clean (91.9%), 58 with errors, 58 PsiErrorElements ### Not done — pick up here -1. **Finish the Tier 3 corpus gate.** 58 of 714 files still produce a - `PsiErrorElement`, all long-tail: each construct below appears in one or two - files. `JaiCorpusParserTest` prints a histogram of the token at each failure - point plus a spread of sample errors; that output is the work list. - `MIN_CLEAN_FILES` is the ratchet — raise it, never lower it. Known gaps: - - mixed declare/assign target lists: `success=, output, error := adb(…)` and - `renderer:, success = make_renderer(…)`; - - `using,except SKIP_THESE name: T;` (a bare identifier operand, where - `using,except(x)` and `using,except .["x"]` already work); - - `#asm` bodies are consumed opaquely, so nothing inside them has PSI. - - Decide explicitly whether the remainder becomes a committed allowlist (the - plan's §3 suggestion) or gets fixed. -2. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution - and go-to-definition. The PSI is in place, so these are now unblocked. - `JaiFileTypeTest` used to document being blocked on the `ParserDefinition`; - that no longer applies. -3. **Phases 5+** — see the plan. +1. **Phase 4** — structure view, folding, `#import`/`#load` reference resolution + and go-to-definition. The PSI is in place, and the Tier 3 corpus gate is now + fully green. `JaiFileTypeTest` used to document being blocked on the + `ParserDefinition`; that no longer applies. + `#asm` bodies are intentionally consumed opaquely, so nothing inside them has + PSI yet. +2. **Phases 5+** — see the plan. ### Open questions for the user (unanswered) @@ -224,10 +214,11 @@ the same sweep to assert every token type the corpus produces (105 of them) has a colour. Keep new lexer work under these gates rather than adding snippets. Same idea for the parser, and it is now live: `JaiCorpusParserTest` parses all -714 files and counts `PsiErrorElement`s — currently **656 clean (91.9%)**. It -prints a histogram of the source text at each failure point, which is the -fastest way to find the next construct worth supporting. The idempotence sweep -for the formatter is the same idea again, later. +714 files and counts `PsiErrorElement`s — currently **714 clean (100.0%)**. The +ratchet is set to the full corpus size, so any parser regression fails the test. +`#asm` bodies remain opaque by design; they are tiled as one directive body but +have no inner PSI. The idempotence sweep for the formatter is the same idea +again, later. The fast loop for grammar work: diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b0ac3f..239d289 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,12 +13,11 @@ - Comment/uncomment support and brace matching, including `.{` and `.[` - Grammar-Kit parser and PSI for the whole language: declarations, procedures, structs, enums, unions, control flow including the `if x == { case }` switch - form, and compiler directives. 91.9% of the 714-file Jai distribution parses - with no syntax errors. + form, and compiler directives. The full 714-file Jai distribution parses with + no syntax errors; `JaiCorpusParserTest` is ratcheted at 100%. ### Known gaps -- 58 files in the Jai distribution still report a syntax error; each involves a - construct used in only one or two files. `#asm` bodies are parsed as opaque - blocks. +- `#asm` bodies are parsed as opaque blocks, so their assembly instructions do + not have inner PSI. - Structure view, folding, and go-to-definition are not implemented yet. diff --git a/docs/BUILD_PLAN.md b/docs/BUILD_PLAN.md index 2097fd2..2a08f1a 100644 --- a/docs/BUILD_PLAN.md +++ b/docs/BUILD_PLAN.md @@ -167,7 +167,7 @@ Each phase has a machine-checkable gate. Do not advance without a green gate. | **0** | Env fix, git, AGENTS.md, template stripped | `./jaigradle check` green with no `JAVA_HOME` | | **1** | File type, icon, `JaiTokenTypes`, hand-written lexer | Tier 0 corpus invariants pass on all 714 files | | **2** | `SyntaxHighlighter`, color settings page, commenter, brace matcher | Tier 1 golden dumps; highlighter maps every token type | -| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target — **done, 91.9% (656/714)** | +| **3** | `.bnf` grammar, generated parser + PSI, `ParserDefinition` | Tier 2 golden trees; Tier 3 corpus parse ≥ target — **done, 100.0% (714/714)** | | **4** | Structure view, folding, `#import`/`#load` reference resolution + go-to-definition | Tier 4 fixture tests | | **5** | Completion (keywords, directives, module names), rename, find-usages | Tier 4 fixture tests | | **6** | Formatter, code style settings | Formatter round-trip: formatting the corpus is idempotent | diff --git a/jaitest b/jaitest index 73e7e86..aaa50bf 100755 --- a/jaitest +++ b/jaitest @@ -26,7 +26,14 @@ fi run_gradle() { rm -rf build/reports/tests/test - ./jaigradle cleanTest test "${extra[@]}" "$@" 2>&1 | grep -vE '^\[[0-9.]+s\]\[warning\]\[cds\]' + # Bash 3.2 (the macOS system shell) treats an empty array expansion as an + # unset variable under `set -u`. Keep the no-extra-args invocation separate + # so the wrapper works both with and without a test filter. + if ((${#extra[@]})); then + ./jaigradle cleanTest test "${extra[@]}" "$@" 2>&1 | grep -vE '^\[[0-9.]+s\]\[warning\]\[cds\]' + else + ./jaigradle cleanTest test "$@" 2>&1 | grep -vE '^\[[0-9.]+s\]\[warning\]\[cds\]' + fi return "${PIPESTATUS[0]}" } diff --git a/src/main/gen/dev/hgh/jai/parser/JaiParser.java b/src/main/gen/dev/hgh/jai/parser/JaiParser.java index 9047f4f..f26caa8 100644 --- a/src/main/gen/dev/hgh/jai/parser/JaiParser.java +++ b/src/main/gen/dev/hgh/jai/parser/JaiParser.java @@ -47,7 +47,7 @@ public class JaiParser implements PsiParser, LightPsiParser { PROC_LITERAL_EXPR, RANGE_EXPR, REF_EXPR, RELATIONAL_EXPR, SHIFT_EXPR, STRUCT_EXPR, STRUCT_LITERAL_EXPR, TYPED_ARRAY_LITERAL_EXPR, TYPED_STRUCT_LITERAL_EXPR, UNARY_DOT_EXPR, UNINITIALIZED_EXPR, UNION_EXPR, - ZERO_LITERAL_EXPR), + VALUE_LITERAL_EXPR, ZERO_LITERAL_EXPR), }; /* ********************************************************** */ @@ -215,18 +215,37 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // <> (',' <>)* + // <> [':'] + static boolean assignTarget_(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "assignTarget_")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = exprNoAssign(builder_, level_ + 1); + result_ = result_ && assignTarget__1(builder_, level_ + 1); + exit_section_(builder_, marker_, null, result_); + return result_; + } + + // [':'] + private static boolean assignTarget__1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "assignTarget__1")) return false; + consumeToken(builder_, COLON); + return true; + } + + /* ********************************************************** */ + // assignTarget_ (',' assignTarget_)* static boolean assignTargets_(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "assignTargets_")) return false; boolean result_; Marker marker_ = enter_section_(builder_); - result_ = exprNoAssign(builder_, level_ + 1); + result_ = assignTarget_(builder_, level_ + 1); result_ = result_ && assignTargets__1(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } - // (',' <>)* + // (',' assignTarget_)* private static boolean assignTargets__1(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "assignTargets__1")) return false; while (true) { @@ -237,13 +256,13 @@ public class JaiParser implements PsiParser, LightPsiParser { return true; } - // ',' <> + // ',' assignTarget_ private static boolean assignTargets__1_0(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "assignTargets__1_0")) return false; boolean result_; Marker marker_ = enter_section_(builder_); result_ = consumeToken(builder_, COMMA); - result_ = result_ && exprNoAssign(builder_, level_ + 1); + result_ = result_ && assignTarget_(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } @@ -494,14 +513,13 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // ['`'] IDENT + // ['`'] (IDENT | keywordAsName_) public static boolean declName(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "declName")) return false; - if (!nextTokenIs(builder_, "", BACKTICK, IDENT)) return false; boolean result_; Marker marker_ = enter_section_(builder_, level_, _NONE_, DECL_NAME, ""); result_ = declName_0(builder_, level_ + 1); - result_ = result_ && consumeToken(builder_, IDENT); + result_ = result_ && declName_1(builder_, level_ + 1); exit_section_(builder_, level_, marker_, result_, false, null); return result_; } @@ -513,20 +531,29 @@ public class JaiParser implements PsiParser, LightPsiParser { return true; } + // IDENT | keywordAsName_ + private static boolean declName_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "declName_1")) return false; + boolean result_; + result_ = consumeToken(builder_, IDENT); + if (!result_) result_ = keywordAsName_(builder_, level_ + 1); + return result_; + } + /* ********************************************************** */ - // declName (',' declName)* + // declName (['='] ',' declName ['='])* ['='] public static boolean declNames(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "declNames")) return false; - if (!nextTokenIs(builder_, "", BACKTICK, IDENT)) return false; boolean result_; Marker marker_ = enter_section_(builder_, level_, _NONE_, DECL_NAMES, ""); result_ = declName(builder_, level_ + 1); result_ = result_ && declNames_1(builder_, level_ + 1); + result_ = result_ && declNames_2(builder_, level_ + 1); exit_section_(builder_, level_, marker_, result_, false, null); return result_; } - // (',' declName)* + // (['='] ',' declName ['='])* private static boolean declNames_1(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "declNames_1")) return false; while (true) { @@ -537,17 +564,40 @@ public class JaiParser implements PsiParser, LightPsiParser { return true; } - // ',' declName + // ['='] ',' declName ['='] private static boolean declNames_1_0(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "declNames_1_0")) return false; boolean result_; Marker marker_ = enter_section_(builder_); - result_ = consumeToken(builder_, COMMA); + result_ = declNames_1_0_0(builder_, level_ + 1); + result_ = result_ && consumeToken(builder_, COMMA); result_ = result_ && declName(builder_, level_ + 1); + result_ = result_ && declNames_1_0_3(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } + // ['='] + private static boolean declNames_1_0_0(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "declNames_1_0_0")) return false; + consumeToken(builder_, EQ); + return true; + } + + // ['='] + private static boolean declNames_1_0_3(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "declNames_1_0_3")) return false; + consumeToken(builder_, EQ); + return true; + } + + // ['='] + private static boolean declNames_2(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "declNames_2")) return false; + consumeToken(builder_, EQ); + return true; + } + /* ********************************************************** */ // ':' [<>] procModifier* [',' assignTargets_] [('=' | ':') initializerList_] // | ':=' initializerList_ @@ -1274,7 +1324,7 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // '<=' expr | '*=' expr | '<' | '*' + // '<=' expr | '*=' expr | '<' | '*' | ',' static boolean forModifier_(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "forModifier_")) return false; boolean result_; @@ -1283,6 +1333,7 @@ public class JaiParser implements PsiParser, LightPsiParser { if (!result_) result_ = forModifier__1(builder_, level_ + 1); if (!result_) result_ = consumeToken(builder_, LT); if (!result_) result_ = consumeToken(builder_, STAR); + if (!result_) result_ = consumeToken(builder_, COMMA); exit_section_(builder_, marker_, null, result_); return result_; } @@ -1442,59 +1493,73 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // ifKeyword_ [<>] expr (switchBody | ['then'] statement [elseClause]) + // nestedInlineIfStatement + // | inlineIfDirectiveStatement + // | ifKeyword_ [<>] expr (switchBody | ['then'] statement [elseClause]) public static boolean ifStatement(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "ifStatement")) return false; boolean result_; Marker marker_ = enter_section_(builder_, level_, _NONE_, IF_STATEMENT, ""); - result_ = ifKeyword_(builder_, level_ + 1); - result_ = result_ && ifStatement_1(builder_, level_ + 1); - result_ = result_ && expr(builder_, level_ + 1, -1); - result_ = result_ && ifStatement_3(builder_, level_ + 1); + result_ = nestedInlineIfStatement(builder_, level_ + 1); + if (!result_) result_ = inlineIfDirectiveStatement(builder_, level_ + 1); + if (!result_) result_ = ifStatement_2(builder_, level_ + 1); exit_section_(builder_, level_, marker_, result_, false, null); return result_; } + // ifKeyword_ [<>] expr (switchBody | ['then'] statement [elseClause]) + private static boolean ifStatement_2(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "ifStatement_2")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = ifKeyword_(builder_, level_ + 1); + result_ = result_ && ifStatement_2_1(builder_, level_ + 1); + result_ = result_ && expr(builder_, level_ + 1, -1); + result_ = result_ && ifStatement_2_3(builder_, level_ + 1); + exit_section_(builder_, marker_, null, result_); + return result_; + } + // [<>] - private static boolean ifStatement_1(PsiBuilder builder_, int level_) { - if (!recursion_guard_(builder_, level_, "ifStatement_1")) return false; + private static boolean ifStatement_2_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "ifStatement_2_1")) return false; dir(builder_, level_ + 1, "#complete"); return true; } // switchBody | ['then'] statement [elseClause] - private static boolean ifStatement_3(PsiBuilder builder_, int level_) { - if (!recursion_guard_(builder_, level_, "ifStatement_3")) return false; + private static boolean ifStatement_2_3(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "ifStatement_2_3")) return false; boolean result_; Marker marker_ = enter_section_(builder_); result_ = switchBody(builder_, level_ + 1); - if (!result_) result_ = ifStatement_3_1(builder_, level_ + 1); + if (!result_) result_ = ifStatement_2_3_1(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } // ['then'] statement [elseClause] - private static boolean ifStatement_3_1(PsiBuilder builder_, int level_) { - if (!recursion_guard_(builder_, level_, "ifStatement_3_1")) return false; + private static boolean ifStatement_2_3_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "ifStatement_2_3_1")) return false; boolean result_; Marker marker_ = enter_section_(builder_); - result_ = ifStatement_3_1_0(builder_, level_ + 1); + result_ = ifStatement_2_3_1_0(builder_, level_ + 1); result_ = result_ && statement(builder_, level_ + 1); - result_ = result_ && ifStatement_3_1_2(builder_, level_ + 1); + result_ = result_ && ifStatement_2_3_1_2(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } // ['then'] - private static boolean ifStatement_3_1_0(PsiBuilder builder_, int level_) { - if (!recursion_guard_(builder_, level_, "ifStatement_3_1_0")) return false; + private static boolean ifStatement_2_3_1_0(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "ifStatement_2_3_1_0")) return false; consumeToken(builder_, KW_THEN); return true; } // [elseClause] - private static boolean ifStatement_3_1_2(PsiBuilder builder_, int level_) { - if (!recursion_guard_(builder_, level_, "ifStatement_3_1_2")) return false; + private static boolean ifStatement_2_3_1_2(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "ifStatement_2_3_1_2")) return false; elseClause(builder_, level_ + 1); return true; } @@ -1567,6 +1632,27 @@ public class JaiParser implements PsiParser, LightPsiParser { return result_; } + /* ********************************************************** */ + // <> IDENT statement [elseClause] + static boolean inlineIfDirectiveStatement(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "inlineIfDirectiveStatement")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = dir(builder_, level_ + 1, "#if"); + result_ = result_ && consumeToken(builder_, IDENT); + result_ = result_ && statement(builder_, level_ + 1); + result_ = result_ && inlineIfDirectiveStatement_3(builder_, level_ + 1); + exit_section_(builder_, marker_, null, result_); + return result_; + } + + // [elseClause] + private static boolean inlineIfDirectiveStatement_3(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "inlineIfDirectiveStatement_3")) return false; + elseClause(builder_, level_ + 1); + return true; + } + /* ********************************************************** */ // 'interface' [expr] public static boolean interfaceExpr(PsiBuilder builder_, int level_) { @@ -1681,18 +1767,30 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // IDENT | NUMBER | keywordAsName_ + // 'operator' operatorSymbol | IDENT | NUMBER | keywordAsName_ public static boolean memberName(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "memberName")) return false; boolean result_; Marker marker_ = enter_section_(builder_, level_, _NONE_, MEMBER_NAME, ""); - result_ = consumeToken(builder_, IDENT); + result_ = memberName_0(builder_, level_ + 1); + if (!result_) result_ = consumeToken(builder_, IDENT); if (!result_) result_ = consumeToken(builder_, NUMBER); if (!result_) result_ = keywordAsName_(builder_, level_ + 1); exit_section_(builder_, level_, marker_, result_, false, null); return result_; } + // 'operator' operatorSymbol + private static boolean memberName_0(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "memberName_0")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = consumeToken(builder_, KW_OPERATOR); + result_ = result_ && operatorSymbol(builder_, level_ + 1); + exit_section_(builder_, marker_, null, result_); + return result_; + } + /* ********************************************************** */ // <> memberName '=' (block | jumpArgument_ | expr) public static boolean namedArgument(PsiBuilder builder_, int level_) { @@ -1717,6 +1815,12 @@ public class JaiParser implements PsiParser, LightPsiParser { return result_; } + /* ********************************************************** */ + // <> + static boolean nestedInlineIfStatement(PsiBuilder builder_, int level_) { + return parseNestedInlineIfStatement(builder_, level_ + 1); + } + /* ********************************************************** */ // NOTE+ public static boolean noteStatement(PsiBuilder builder_, int level_) { @@ -1752,7 +1856,7 @@ public class JaiParser implements PsiParser, LightPsiParser { /* ********************************************************** */ // '*' '[' ']' | '[' ']' '=' | '[' ']' - // | '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '<' | '>' | '<=' | '>=' + // | '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '!' | '<' | '>' | '<=' | '>=' // | '&' | '|' | '^' | '~' | '<<' | '>>' | '<<<' | '>>>' // | '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' // | '<<=' | '>>=' | '<<<=' | '>>>=' @@ -1770,6 +1874,7 @@ public class JaiParser implements PsiParser, LightPsiParser { if (!result_) result_ = consumeToken(builder_, PERCENT); if (!result_) result_ = consumeToken(builder_, EQ_EQ); if (!result_) result_ = consumeToken(builder_, NOT_EQ); + if (!result_) result_ = consumeToken(builder_, NOT); if (!result_) result_ = consumeToken(builder_, LT); if (!result_) result_ = consumeToken(builder_, GT); if (!result_) result_ = consumeToken(builder_, LT_EQ); @@ -1841,17 +1946,53 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // <> | 'using' + // <> | 'using' [directiveFlag_ [<>]] static boolean paramModifier_(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "paramModifier_")) return false; boolean result_; Marker marker_ = enter_section_(builder_); result_ = dirPlain(builder_, level_ + 1); - if (!result_) result_ = consumeToken(builder_, KW_USING); + if (!result_) result_ = paramModifier__1(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } + // 'using' [directiveFlag_ [<>]] + private static boolean paramModifier__1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "paramModifier__1")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = consumeToken(builder_, KW_USING); + result_ = result_ && paramModifier__1_1(builder_, level_ + 1); + exit_section_(builder_, marker_, null, result_); + return result_; + } + + // [directiveFlag_ [<>]] + private static boolean paramModifier__1_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "paramModifier__1_1")) return false; + paramModifier__1_1_0(builder_, level_ + 1); + return true; + } + + // directiveFlag_ [<>] + private static boolean paramModifier__1_1_0(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "paramModifier__1_1_0")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = directiveFlag_(builder_, level_ + 1); + result_ = result_ && paramModifier__1_1_0_1(builder_, level_ + 1); + exit_section_(builder_, marker_, null, result_); + return result_; + } + + // [<>] + private static boolean paramModifier__1_1_0_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "paramModifier__1_1_0_1")) return false; + exprNoAssign(builder_, level_ + 1); + return true; + } + /* ********************************************************** */ // ['$' | '$$'] IDENT public static boolean paramName(PsiBuilder builder_, int level_) { @@ -2212,18 +2353,29 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // directiveExpr + // !<> directiveExpr public static boolean procModifier(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "procModifier")) return false; boolean result_; Marker marker_ = enter_section_(builder_, level_, _NONE_, PROC_MODIFIER, ""); - result_ = directiveExpr(builder_, level_ + 1); + result_ = procModifier_0(builder_, level_ + 1); + result_ = result_ && directiveExpr(builder_, level_ + 1); + exit_section_(builder_, level_, marker_, result_, false, null); + return result_; + } + + // !<> + private static boolean procModifier_0(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "procModifier_0")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_, level_, _NOT_); + result_ = !nonProcDirectiveAhead(builder_, level_ + 1); exit_section_(builder_, level_, marker_, result_, false, null); return result_; } /* ********************************************************** */ - // ['`'] 'push_context' directiveFlag_* [expr] block + // ['`'] 'push_context' directiveFlag_* [expr] (block | semi) public static boolean pushContextStatement(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "pushContextStatement")) return false; if (!nextTokenIs(builder_, "", BACKTICK, KW_PUSH_CONTEXT)) return false; @@ -2233,7 +2385,7 @@ public class JaiParser implements PsiParser, LightPsiParser { result_ = result_ && consumeToken(builder_, KW_PUSH_CONTEXT); result_ = result_ && pushContextStatement_2(builder_, level_ + 1); result_ = result_ && pushContextStatement_3(builder_, level_ + 1); - result_ = result_ && block(builder_, level_ + 1); + result_ = result_ && pushContextStatement_4(builder_, level_ + 1); exit_section_(builder_, level_, marker_, result_, false, null); return result_; } @@ -2263,6 +2415,15 @@ public class JaiParser implements PsiParser, LightPsiParser { return true; } + // block | semi + private static boolean pushContextStatement_4(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "pushContextStatement_4")) return false; + boolean result_; + result_ = block(builder_, level_ + 1); + if (!result_) result_ = semi(builder_, level_ + 1); + return result_; + } + /* ********************************************************** */ // IDENT public static boolean refExpr(PsiBuilder builder_, int level_) { @@ -2447,7 +2608,7 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // ['`'] 'return' [expr (',' expr)*] semiWithNotes_ + // ['`'] 'return' [expr (',' expr)* [',']] semiWithNotes_ public static boolean returnStatement(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "returnStatement")) return false; if (!nextTokenIs(builder_, "", BACKTICK, KW_RETURN)) return false; @@ -2468,20 +2629,21 @@ public class JaiParser implements PsiParser, LightPsiParser { return true; } - // [expr (',' expr)*] + // [expr (',' expr)* [',']] private static boolean returnStatement_2(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "returnStatement_2")) return false; returnStatement_2_0(builder_, level_ + 1); return true; } - // expr (',' expr)* + // expr (',' expr)* [','] private static boolean returnStatement_2_0(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "returnStatement_2_0")) return false; boolean result_; Marker marker_ = enter_section_(builder_); result_ = expr(builder_, level_ + 1, -1); result_ = result_ && returnStatement_2_0_1(builder_, level_ + 1); + result_ = result_ && returnStatement_2_0_2(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } @@ -2508,6 +2670,13 @@ public class JaiParser implements PsiParser, LightPsiParser { return result_; } + // [','] + private static boolean returnStatement_2_0_2(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "returnStatement_2_0_2")) return false; + consumeToken(builder_, COMMA); + return true; + } + /* ********************************************************** */ // <> returnGroup // | returnItem (',' <> returnItem)* @@ -2931,7 +3100,28 @@ public class JaiParser implements PsiParser, LightPsiParser { } /* ********************************************************** */ - // 'while' [<> IDENT ':'] expr statement + // '{' [argList] '}' + public static boolean valueLiteralExpr(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "valueLiteralExpr")) return false; + if (!nextTokenIs(builder_, LBRACE)) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = consumeToken(builder_, LBRACE); + result_ = result_ && valueLiteralExpr_1(builder_, level_ + 1); + result_ = result_ && consumeToken(builder_, RBRACE); + exit_section_(builder_, marker_, VALUE_LITERAL_EXPR, result_); + return result_; + } + + // [argList] + private static boolean valueLiteralExpr_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "valueLiteralExpr_1")) return false; + argList(builder_, level_ + 1); + return true; + } + + /* ********************************************************** */ + // 'while' [<> ['`'] IDENT (':' | ':=')] expr statement public static boolean whileStatement(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "whileStatement")) return false; if (!nextTokenIs(builder_, KW_WHILE)) return false; @@ -2945,24 +3135,42 @@ public class JaiParser implements PsiParser, LightPsiParser { return result_; } - // [<> IDENT ':'] + // [<> ['`'] IDENT (':' | ':=')] private static boolean whileStatement_1(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "whileStatement_1")) return false; whileStatement_1_0(builder_, level_ + 1); return true; } - // <> IDENT ':' + // <> ['`'] IDENT (':' | ':=') private static boolean whileStatement_1_0(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "whileStatement_1_0")) return false; boolean result_; Marker marker_ = enter_section_(builder_); result_ = isLabelAhead(builder_, level_ + 1); - result_ = result_ && consumeTokens(builder_, 0, IDENT, COLON); + result_ = result_ && whileStatement_1_0_1(builder_, level_ + 1); + result_ = result_ && consumeToken(builder_, IDENT); + result_ = result_ && whileStatement_1_0_3(builder_, level_ + 1); exit_section_(builder_, marker_, null, result_); return result_; } + // ['`'] + private static boolean whileStatement_1_0_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "whileStatement_1_0_1")) return false; + consumeToken(builder_, BACKTICK); + return true; + } + + // ':' | ':=' + private static boolean whileStatement_1_0_3(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "whileStatement_1_0_3")) return false; + boolean result_; + result_ = consumeToken(builder_, COLON); + if (!result_) result_ = consumeToken(builder_, COLON_EQ); + return result_; + } + /* ********************************************************** */ // '{' '}' public static boolean zeroLiteralExpr(PsiBuilder builder_, int level_) { @@ -2979,7 +3187,7 @@ public class JaiParser implements PsiParser, LightPsiParser { // Expression root: expr // Operator priority table: // 0: BINARY(assignExpr) - // 1: BINARY(lambdaExpr) + // 1: POSTFIX(lambdaExpr) // 2: BINARY(orExpr) // 3: BINARY(andExpr) // 4: BINARY(equalityExpr) @@ -3034,8 +3242,8 @@ public class JaiParser implements PsiParser, LightPsiParser { result_ = expr(builder_, level_, -1); exit_section_(builder_, level_, marker_, ASSIGN_EXPR, result_, true, null); } - else if (priority_ < 1 && consumeTokenSmart(builder_, QUICK_LAMBDA)) { - result_ = expr(builder_, level_, 0); + else if (priority_ < 1 && lambdaExpr_0(builder_, level_ + 1)) { + result_ = true; exit_section_(builder_, level_, marker_, LAMBDA_EXPR, result_, true, null); } else if (priority_ < 2 && consumeTokenSmart(builder_, OR_OR)) { @@ -3118,6 +3326,26 @@ public class JaiParser implements PsiParser, LightPsiParser { return result_; } + // '=>' (block | expr) + private static boolean lambdaExpr_0(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "lambdaExpr_0")) return false; + boolean result_; + Marker marker_ = enter_section_(builder_); + result_ = consumeTokenSmart(builder_, QUICK_LAMBDA); + result_ = result_ && lambdaExpr_0_1(builder_, level_ + 1); + exit_section_(builder_, marker_, null, result_); + return result_; + } + + // block | expr + private static boolean lambdaExpr_0_1(PsiBuilder builder_, int level_) { + if (!recursion_guard_(builder_, level_, "lambdaExpr_0_1")) return false; + boolean result_; + result_ = block(builder_, level_ + 1); + if (!result_) result_ = expr(builder_, level_ + 1, -1); + return result_; + } + // <> | '!=' | '===' private static boolean equalityExpr_0(PsiBuilder builder_, int level_) { if (!recursion_guard_(builder_, level_, "equalityExpr_0")) return false; @@ -3531,6 +3759,7 @@ public class JaiParser implements PsiParser, LightPsiParser { // | parenExpr // | structLiteralExpr // | arrayLiteralExpr + // | valueLiteralExpr // | unaryDotExpr // | backtickExpr // | builtinCallExpr @@ -3551,6 +3780,7 @@ public class JaiParser implements PsiParser, LightPsiParser { if (!result_) result_ = parenExpr(builder_, level_ + 1); if (!result_) result_ = structLiteralExpr(builder_, level_ + 1); if (!result_) result_ = arrayLiteralExpr(builder_, level_ + 1); + if (!result_) result_ = valueLiteralExpr(builder_, level_ + 1); if (!result_) result_ = unaryDotExpr(builder_, level_ + 1); if (!result_) result_ = backtickExpr(builder_, level_ + 1); if (!result_) result_ = builtinCallExpr(builder_, level_ + 1); diff --git a/src/main/gen/dev/hgh/jai/psi/JaiIfStatement.java b/src/main/gen/dev/hgh/jai/psi/JaiIfStatement.java index d0742c0..642a495 100644 --- a/src/main/gen/dev/hgh/jai/psi/JaiIfStatement.java +++ b/src/main/gen/dev/hgh/jai/psi/JaiIfStatement.java @@ -10,7 +10,7 @@ public interface JaiIfStatement extends PsiElement { @Nullable JaiElseClause getElseClause(); - @NotNull + @Nullable JaiExpr getExpr(); @Nullable diff --git a/src/main/gen/dev/hgh/jai/psi/JaiLambdaExpr.java b/src/main/gen/dev/hgh/jai/psi/JaiLambdaExpr.java index 68943cd..228d54f 100644 --- a/src/main/gen/dev/hgh/jai/psi/JaiLambdaExpr.java +++ b/src/main/gen/dev/hgh/jai/psi/JaiLambdaExpr.java @@ -7,6 +7,9 @@ import com.intellij.psi.PsiElement; public interface JaiLambdaExpr extends JaiExpr { + @Nullable + JaiBlock getBlock(); + @NotNull List getExprList(); diff --git a/src/main/gen/dev/hgh/jai/psi/JaiMemberName.java b/src/main/gen/dev/hgh/jai/psi/JaiMemberName.java index c116181..ae8ba9a 100644 --- a/src/main/gen/dev/hgh/jai/psi/JaiMemberName.java +++ b/src/main/gen/dev/hgh/jai/psi/JaiMemberName.java @@ -7,4 +7,7 @@ import com.intellij.psi.PsiElement; public interface JaiMemberName extends PsiElement { + @Nullable + JaiOperatorSymbol getOperatorSymbol(); + } diff --git a/src/main/gen/dev/hgh/jai/psi/JaiPushContextStatement.java b/src/main/gen/dev/hgh/jai/psi/JaiPushContextStatement.java index fec6614..2cad5ea 100644 --- a/src/main/gen/dev/hgh/jai/psi/JaiPushContextStatement.java +++ b/src/main/gen/dev/hgh/jai/psi/JaiPushContextStatement.java @@ -7,7 +7,7 @@ import com.intellij.psi.PsiElement; public interface JaiPushContextStatement extends PsiElement { - @NotNull + @Nullable JaiBlock getBlock(); @Nullable diff --git a/src/main/gen/dev/hgh/jai/psi/JaiTypes.java b/src/main/gen/dev/hgh/jai/psi/JaiTypes.java index 2500459..657d647 100644 --- a/src/main/gen/dev/hgh/jai/psi/JaiTypes.java +++ b/src/main/gen/dev/hgh/jai/psi/JaiTypes.java @@ -87,6 +87,7 @@ public interface JaiTypes { IElementType UNINITIALIZED_EXPR = new JaiElementType("UNINITIALIZED_EXPR"); IElementType UNION_EXPR = new JaiElementType("UNION_EXPR"); IElementType USING_STATEMENT = new JaiElementType("USING_STATEMENT"); + IElementType VALUE_LITERAL_EXPR = new JaiElementType("VALUE_LITERAL_EXPR"); IElementType WHILE_STATEMENT = new JaiElementType("WHILE_STATEMENT"); IElementType ZERO_LITERAL_EXPR = new JaiElementType("ZERO_LITERAL_EXPR"); @@ -430,6 +431,9 @@ public interface JaiTypes { else if (type == USING_STATEMENT) { return new JaiUsingStatementImpl(node); } + else if (type == VALUE_LITERAL_EXPR) { + return new JaiValueLiteralExprImpl(node); + } else if (type == WHILE_STATEMENT) { return new JaiWhileStatementImpl(node); } diff --git a/src/main/gen/dev/hgh/jai/psi/JaiValueLiteralExpr.java b/src/main/gen/dev/hgh/jai/psi/JaiValueLiteralExpr.java new file mode 100644 index 0000000..0cb3718 --- /dev/null +++ b/src/main/gen/dev/hgh/jai/psi/JaiValueLiteralExpr.java @@ -0,0 +1,13 @@ +// This is a generated file. Not intended for manual editing. +package dev.hgh.jai.psi; + +import java.util.List; +import org.jetbrains.annotations.*; +import com.intellij.psi.PsiElement; + +public interface JaiValueLiteralExpr extends JaiExpr { + + @NotNull + List getArgumentList(); + +} diff --git a/src/main/gen/dev/hgh/jai/psi/JaiVisitor.java b/src/main/gen/dev/hgh/jai/psi/JaiVisitor.java index b9d2c27..7f4481e 100644 --- a/src/main/gen/dev/hgh/jai/psi/JaiVisitor.java +++ b/src/main/gen/dev/hgh/jai/psi/JaiVisitor.java @@ -319,6 +319,10 @@ public class JaiVisitor extends PsiElementVisitor { visitPsiElement(o); } + public void visitValueLiteralExpr(@NotNull JaiValueLiteralExpr o) { + visitExpr(o); + } + public void visitWhileStatement(@NotNull JaiWhileStatement o) { visitPsiElement(o); } diff --git a/src/main/gen/dev/hgh/jai/psi/impl/JaiIfStatementImpl.java b/src/main/gen/dev/hgh/jai/psi/impl/JaiIfStatementImpl.java index b7a90a1..672bca9 100644 --- a/src/main/gen/dev/hgh/jai/psi/impl/JaiIfStatementImpl.java +++ b/src/main/gen/dev/hgh/jai/psi/impl/JaiIfStatementImpl.java @@ -34,9 +34,9 @@ public class JaiIfStatementImpl extends ASTWrapperPsiElement implements JaiIfSta } @Override - @NotNull + @Nullable public JaiExpr getExpr() { - return findNotNullChildByClass(JaiExpr.class); + return findChildByClass(JaiExpr.class); } @Override diff --git a/src/main/gen/dev/hgh/jai/psi/impl/JaiLambdaExprImpl.java b/src/main/gen/dev/hgh/jai/psi/impl/JaiLambdaExprImpl.java index 8cffe20..913ff2c 100644 --- a/src/main/gen/dev/hgh/jai/psi/impl/JaiLambdaExprImpl.java +++ b/src/main/gen/dev/hgh/jai/psi/impl/JaiLambdaExprImpl.java @@ -27,6 +27,12 @@ public class JaiLambdaExprImpl extends JaiExprImpl implements JaiLambdaExpr { else super.accept(visitor); } + @Override + @Nullable + public JaiBlock getBlock() { + return findChildByClass(JaiBlock.class); + } + @Override @NotNull public List getExprList() { diff --git a/src/main/gen/dev/hgh/jai/psi/impl/JaiMemberNameImpl.java b/src/main/gen/dev/hgh/jai/psi/impl/JaiMemberNameImpl.java index 2b71f2f..1ea4b59 100644 --- a/src/main/gen/dev/hgh/jai/psi/impl/JaiMemberNameImpl.java +++ b/src/main/gen/dev/hgh/jai/psi/impl/JaiMemberNameImpl.java @@ -27,4 +27,10 @@ public class JaiMemberNameImpl extends ASTWrapperPsiElement implements JaiMember else super.accept(visitor); } + @Override + @Nullable + public JaiOperatorSymbol getOperatorSymbol() { + return findChildByClass(JaiOperatorSymbol.class); + } + } diff --git a/src/main/gen/dev/hgh/jai/psi/impl/JaiPushContextStatementImpl.java b/src/main/gen/dev/hgh/jai/psi/impl/JaiPushContextStatementImpl.java index 975d587..7b493fb 100644 --- a/src/main/gen/dev/hgh/jai/psi/impl/JaiPushContextStatementImpl.java +++ b/src/main/gen/dev/hgh/jai/psi/impl/JaiPushContextStatementImpl.java @@ -28,9 +28,9 @@ public class JaiPushContextStatementImpl extends ASTWrapperPsiElement implements } @Override - @NotNull + @Nullable public JaiBlock getBlock() { - return findNotNullChildByClass(JaiBlock.class); + return findChildByClass(JaiBlock.class); } @Override diff --git a/src/main/gen/dev/hgh/jai/psi/impl/JaiValueLiteralExprImpl.java b/src/main/gen/dev/hgh/jai/psi/impl/JaiValueLiteralExprImpl.java new file mode 100644 index 0000000..2804fe7 --- /dev/null +++ b/src/main/gen/dev/hgh/jai/psi/impl/JaiValueLiteralExprImpl.java @@ -0,0 +1,36 @@ +// This is a generated file. Not intended for manual editing. +package dev.hgh.jai.psi.impl; + +import java.util.List; +import org.jetbrains.annotations.*; +import com.intellij.lang.ASTNode; +import com.intellij.psi.PsiElement; +import com.intellij.psi.PsiElementVisitor; +import com.intellij.psi.util.PsiTreeUtil; +import static dev.hgh.jai.psi.JaiTypes.*; +import dev.hgh.jai.psi.*; + +public class JaiValueLiteralExprImpl extends JaiExprImpl implements JaiValueLiteralExpr { + + public JaiValueLiteralExprImpl(@NotNull ASTNode node) { + super(node); + } + + @Override + public void accept(@NotNull JaiVisitor visitor) { + visitor.visitValueLiteralExpr(this); + } + + @Override + public void accept(@NotNull PsiElementVisitor visitor) { + if (visitor instanceof JaiVisitor) accept((JaiVisitor)visitor); + else super.accept(visitor); + } + + @Override + @NotNull + public List getArgumentList() { + return PsiTreeUtil.getChildrenOfTypeAsList(this, JaiArgument.class); + } + +} diff --git a/src/main/grammar/Jai.bnf b/src/main/grammar/Jai.bnf index f672247..c5dcb6d 100644 --- a/src/main/grammar/Jai.bnf +++ b/src/main/grammar/Jai.bnf @@ -181,7 +181,11 @@ directiveStatement ::= <> directiveFlag_* [';'] // `#if` is spelled as a directive but is structurally an `if`, and it pairs with the // ordinary `else` keyword (see modules/File_Watcher/module.jai). -ifStatement ::= ifKeyword_ [<>] expr (switchBody | ['then'] statement [elseClause]) +ifStatement ::= nestedInlineIfStatement + | inlineIfDirectiveStatement + | ifKeyword_ [<>] expr (switchBody | ['then'] statement [elseClause]) +private nestedInlineIfStatement ::= <> +private inlineIfDirectiveStatement ::= <> IDENT statement [elseClause] private ifKeyword_ ::= 'if' | <> elseClause ::= 'else' statement @@ -190,21 +194,21 @@ elseClause ::= 'else' statement switchBody ::= '==' '{' caseClause* '}' caseClause ::= 'case' [expr] [';'] statement* -whileStatement ::= 'while' [<> IDENT ':'] expr statement +whileStatement ::= 'while' [<> ['`'] IDENT (':' | ':=')] expr statement // `for < * name, index : iterable` — `<` reverses, `*` iterates by pointer, and // `for :expansion` selects a named for_expansion. forStatement ::= 'for' [':' expr] forModifier_* [<> forNames ':'] expr statement -private forModifier_ ::= '<=' expr | '*=' expr | '<' | '*' +private forModifier_ ::= '<=' expr | '*=' expr | '<' | '*' | ',' forNames ::= forName (',' forName)* forName ::= ['`'] IDENT -returnStatement ::= ['`'] 'return' [expr (',' expr)*] semiWithNotes_ +returnStatement ::= ['`'] 'return' [expr (',' expr)* [',']] semiWithNotes_ breakStatement ::= 'break' [IDENT] semi continueStatement ::= 'continue' [IDENT] semi removeStatement ::= 'remove' [expr] semi deferStatement ::= ['`'] 'defer' statement -pushContextStatement ::= ['`'] 'push_context' directiveFlag_* [expr] block +pushContextStatement ::= ['`'] 'push_context' directiveFlag_* [expr] (block | semi) usingStatement ::= 'using' [',' expr] expr semi expressionStatement ::= expr (',' expr)* semiWithNotes_ @@ -221,8 +225,8 @@ private declModifier_ ::= <> directiveFlag_* block | <> directiveFlag_* [parenExpr | STRING] | usingModifier_ private usingModifier_ ::= 'using' [directiveFlag_ [<>]] -declNames ::= declName (',' declName)* -declName ::= ['`'] IDENT +declNames ::= declName (['='] ',' declName ['='])* ['='] +declName ::= ['`'] (IDENT | keywordAsName_) private declTail_ ::= ':' [<>] procModifier* [',' assignTargets_] [('=' | ':') initializerList_] | ':=' initializerList_ | '::' initializerList_ @@ -230,7 +234,8 @@ private declTail_ ::= ':' [<>] procModifier* [',' assignTargets_] private initializerList_ ::= initializer (',' initializer)* // `success:, apk.zip_dir = load_zip_directory(path);` declares the first name and // assigns to the remaining lvalues. -private assignTargets_ ::= <> (',' <>)* +private assignTargets_ ::= assignTarget_ (',' assignTarget_)* +private assignTarget_ ::= <> [':'] initializer ::= '---' | expr // `operator + :: (a: T, b: T) -> T { }` @@ -238,7 +243,7 @@ operatorDeclaration ::= 'operator' operatorSymbol '::' expr semiWithNotes_ // Longest first: `*` would otherwise match the start of `*[]`, and `[]` the start // of `[]=`. Single-token operators cannot collide, since `+=` is one token. operatorSymbol ::= '*' '[' ']' | '[' ']' '=' | '[' ']' - | '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '<' | '>' | '<=' | '>=' + | '+' | '-' | '*' | '/' | '%' | '==' | '!=' | '!' | '<' | '>' | '<=' | '>=' | '&' | '|' | '^' | '~' | '<<' | '>>' | '<<<' | '>>>' | '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' | '<<=' | '>>=' | '<<<=' | '>>>=' @@ -278,7 +283,7 @@ assignExpr ::= expr assignOp_ expr { rightAssociative=true } private assignOp_ ::= '=' | '+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' | '<<=' | '>>=' | '<<<=' | '>>>=' | '&&=' | '||=' -lambdaExpr ::= expr '=>' expr { rightAssociative=true } +lambdaExpr ::= expr '=>' (block | expr) { rightAssociative=true } orExpr ::= expr '||' expr andExpr ::= expr '&&' expr @@ -321,7 +326,7 @@ derefExpr ::= expr '.*' typedStructLiteralExpr ::= expr '.{' [argList] '}' typedArrayLiteralExpr ::= expr '.[' [argList] ']' -memberName ::= IDENT | NUMBER | keywordAsName_ +memberName ::= 'operator' operatorSymbol | IDENT | NUMBER | keywordAsName_ // A few keywords double as field names in the distribution (`context.print_style`, // `.type`, `it.operator`), so member access accepts them. private keywordAsName_ ::= 'if' | 'for' | 'case' | 'enum' | 'cast' | 'union' | 'struct' @@ -345,6 +350,7 @@ primaryExpr ::= procLiteralExpr | parenExpr | structLiteralExpr | arrayLiteralExpr + | valueLiteralExpr | unaryDotExpr | backtickExpr | builtinCallExpr @@ -360,6 +366,7 @@ uninitializedExpr ::= '---' | '--' // `table.* = {};` zero-initialises. Only the empty form is an expression; a // non-empty `{ … }` in expression position would collide with a block statement. zeroLiteralExpr ::= '{' '}' +valueLiteralExpr ::= '{' [argList] '}' backtickExpr ::= '`' (IDENT | 'defer' | 'return' | 'push_context' | 'operator') unaryDotExpr ::= '.' memberName structLiteralExpr ::= '.{' [argList] '}' @@ -378,7 +385,7 @@ procLiteralExpr ::= <> '(' [paramList] ')' ['->' returnTypes_ private paramList ::= parameter (',' parameter)* [','] parameter ::= paramModifier_* parameterTail_ -private paramModifier_ ::= <> | 'using' +private paramModifier_ ::= <> | 'using' [directiveFlag_ [<>]] private parameterTail_ ::= <> paramNames paramTypeTail_ | '..' | expr paramNames ::= paramName (',' paramName)* paramName ::= ['$' | '$$'] IDENT @@ -399,7 +406,7 @@ private returnDefault_ ::= '---' | <> // `#foreign lib`, `#modify { ... }`, `#c_call`, `#expand`, … // `#foreign lib "symbol"`, `#modify { ... }`, `#c_call`, `#expand`, `#align 8` — the // same shape as any other directive, so there is one rule for both positions. -procModifier ::= directiveExpr +procModifier ::= !<> directiveExpr structExpr ::= 'struct' ['(' [paramList] ')'] structModifier_* block enumExpr ::= ('enum' | 'enum_flags') [!'{' expr] structModifier_* block diff --git a/src/main/kotlin/dev/hgh/jai/parser/JaiParserUtil.kt b/src/main/kotlin/dev/hgh/jai/parser/JaiParserUtil.kt index 3f2c84e..111a021 100644 --- a/src/main/kotlin/dev/hgh/jai/parser/JaiParserUtil.kt +++ b/src/main/kotlin/dev/hgh/jai/parser/JaiParserUtil.kt @@ -37,7 +37,6 @@ object JaiParserUtil : GeneratedParserUtilBase() { "insert", "code", "run_and_insert", - "compile_time", "bytes", "no_reset", ) @@ -79,6 +78,9 @@ object JaiParserUtil : GeneratedParserUtilBase() { "poke_name", "dump", "foreign", + "compiler", + "intrinsic", + "program_export", "define", "include", "undef", @@ -100,7 +102,40 @@ object JaiParserUtil : GeneratedParserUtilBase() { * that declaration's scope. Taken from the corpus, where only these four appear * directly in front of a `name :` / `name ::` / `name :=`. */ - private val DIRECTIVES_ON_DECLARATIONS = setOf("as", "overlay", "add_context", "no_reset") + private val DIRECTIVES_ON_DECLARATIONS = + setOf("as", "overlay", "add_context", "no_reset", "program_export") + + /** Scope switches are statements, never procedure modifiers. */ + private val NON_PROC_DIRECTIVES = setOf("scope_file", "scope_module", "scope_export") + + /** Keyword tokens accepted by the grammar's `keywordAsName_` name rule. */ + private val DECL_NAME_KEYWORDS = + setOf( + T.KW_IF, + T.KW_FOR, + T.KW_CASE, + T.KW_ENUM, + T.KW_CAST, + T.KW_UNION, + T.KW_STRUCT, + T.KW_REMOVE, + T.KW_INLINE, + T.KW_CONTEXT, + T.KW_OPERATOR, + T.KW_INTERFACE, + T.KW_TYPE_INFO, + T.KW_USING, + T.KW_DEFER, + T.KW_RETURN, + T.KW_BREAK, + T.KW_CONTINUE, + T.KW_THEN, + T.KW_ELSE, + T.KW_WHILE, + T.KW_NULL, + T.KW_TRUE, + T.KW_FALSE, + ) // ---------------------------------------------------------------- directives @@ -154,6 +189,22 @@ object JaiParserUtil : GeneratedParserUtilBase() { return true } + /** + * True when the current directive is a scope switch, which is a statement + * rather than a procedure modifier. Keeping this as a lookahead predicate lets + * the grammar's normal `directiveExpr` consume the modifier's operand, such + * as the library name in `#foreign kernel32`. + */ + @JvmStatic + fun nonProcDirectiveAhead( + b: PsiBuilder, + level: Int, + ): Boolean { + if (b.tokenType !== T.DIRECTIVE) return false + val text = b.tokenText ?: return false + return text.removePrefix("#") in NON_PROC_DIRECTIVES + } + /** * A directive that takes neither an operand nor a block, so it stands alone as a * statement without a `;` — `#scope_file`, `#no_padding`, `#compiler`. @@ -166,7 +217,12 @@ object JaiParserUtil : GeneratedParserUtilBase() { if (b.tokenType !== T.DIRECTIVE) return false val text = b.tokenText ?: return false val name = text.removePrefix("#") - if (name in DIRECTIVES_WITH_OPERAND || name in DIRECTIVES_WITH_BLOCK) return false + if (name in DIRECTIVES_WITH_BLOCK) return false + if (name in DIRECTIVES_WITH_OPERAND && + !(name == "program_export" && b.lookAhead(1) !== T.STRING) + ) { + return false + } b.advanceLexer() return true } @@ -345,6 +401,8 @@ object JaiParserUtil : GeneratedParserUtilBase() { * `#as using,except(vtable) iunknown: IUnknown;` and the macro-exported * `` `it := entry.value; ``. */ + private fun isDeclarationNameToken(token: IElementType?): Boolean = token === T.IDENT || token in DECL_NAME_KEYWORDS + @JvmStatic fun isDeclarationAhead( b: PsiBuilder, @@ -352,17 +410,24 @@ object JaiParserUtil : GeneratedParserUtilBase() { ): Boolean { var i = 0 var inPrefix = false + var sawUsingPrefix = false loop@ while (true) { when (b.lookAhead(i)) { T.DIRECTIVE -> { + // After `using,except`, a directive is the operand of the + // using modifier (`using,except #run ... name: T`), not a + // second declaration modifier. + if (sawUsingPrefix && !inPrefix) break@loop if (!isDeclarationDirective(b, i)) break@loop i++ inPrefix = true + if (b.lookAhead(i) === T.STRING) i++ } T.KW_USING -> { i++ inPrefix = true + sawUsingPrefix = true } T.LPAREN, T.LBRACKET, T.BEGIN_ARRAY_LITERAL, T.BEGIN_STRUCT_LITERAL -> { @@ -376,6 +441,17 @@ object JaiParserUtil : GeneratedParserUtilBase() { if (!inPrefix || b.lookAhead(i + 1) !== T.IDENT) break@loop i += 2 inPrefix = false + // `using,except SKIP_THESE orientation: T` has a bare + // identifier as the flag's operand. The declaration grammar + // consumes it through `exprNoAssign`; skip it here so the + // actual field name is what declNames sees. + if (sawUsingPrefix && + b.lookAhead(i) === T.IDENT && + b.lookAhead(i + 1) === T.IDENT && + b.lookAhead(i + 2) === T.COLON + ) { + i++ + } } else -> { @@ -383,14 +459,66 @@ object JaiParserUtil : GeneratedParserUtilBase() { } } } + // A using-except operand can be a directive expression, for example: + // `using,except #run names_to_skip(Type) business: Business;`. The + // prefix loop intentionally stops at that directive, so find the first + // declaration name after its (possibly grouped) operand. + if (sawUsingPrefix && !isDeclarationNameToken(b.lookAhead(i))) { + var candidate = i + while (candidate <= MAX_LOOKAHEAD) { + when (b.lookAhead(candidate)) { + T.LPAREN, T.LBRACKET, T.BEGIN_ARRAY_LITERAL, T.BEGIN_STRUCT_LITERAL -> { + candidate = skipBalancedGroup(b, candidate) + if (candidate < 0) return false + continue + } + + T.SEMICOLON, T.RBRACE -> { + return false + } + + else -> {} + } + if (isDeclarationNameToken(b.lookAhead(candidate)) && + b.lookAhead(candidate + 1) in setOf(T.COLON, T.COLON_COLON, T.COLON_EQ) + ) { + return true + } + candidate++ + } + } while (true) { if (b.lookAhead(i) === T.BACKTICK) i++ - if (b.lookAhead(i) !== T.IDENT) return false + if (!isDeclarationNameToken(b.lookAhead(i))) return false i++ when (b.lookAhead(i)) { - T.COLON, T.COLON_COLON, T.COLON_EQ -> return true - T.COMMA -> i++ - else -> return false + T.COLON, T.COLON_COLON, T.COLON_EQ -> { + return true + } + + // In a mixed declaration/assignment list, an existing lvalue is + // marked with `=` before the next declared name: + // `success, output=, error := adb(...)`. + T.EQ -> { + when (b.lookAhead(i + 1)) { + T.COMMA -> i += 2 + + // The final assigned target may be followed directly by + // the declaration's `:=` tail: + // `target, existing= := make_target()`. + T.COLON_EQ -> return true + + else -> return false + } + } + + T.COMMA -> { + i++ + } + + else -> { + return false + } } } } @@ -526,12 +654,43 @@ object JaiParserUtil : GeneratedParserUtilBase() { return t !== T.DOLLAR && t !== T.DOUBLE_DOLLAR } - /** `IDENT ':'` — a loop label. */ + /** Consumes `if (condition) #if FLAG statement` as one nested statement. */ + @JvmStatic + fun parseNestedInlineIfStatement( + b: PsiBuilder, + level: Int, + ): Boolean { + val marker = b.mark() + + fun fail(): Boolean { + marker.rollbackTo() + return false + } + + if (b.tokenType !== T.KW_IF || b.lookAhead(1) !== T.LPAREN) return fail() + b.advanceLexer() + if (!JaiParser.parenExpr(b, level + 1)) return fail() + if (b.tokenType !== T.DIRECTIVE || b.tokenText != "#if") return fail() + b.advanceLexer() + if (b.tokenType !== T.IDENT) return fail() + b.advanceLexer() + if (!JaiParser.statement(b, level + 1)) return fail() + + marker.drop() + return true + } + + /** `IDENT ':'` or `IDENT ':='` — a loop label. */ @JvmStatic fun isLabelAhead( b: PsiBuilder, level: Int, - ): Boolean = b.tokenType === T.IDENT && b.lookAhead(1) === T.COLON + ): Boolean { + val nameOffset = if (b.tokenType === T.BACKTICK) 1 else 0 + if (b.lookAhead(nameOffset) !== T.IDENT) return false + val separator = b.lookAhead(nameOffset + 1) + return separator === T.COLON || separator === T.COLON_EQ + } private const val MAX_LOOKAHEAD = 4096 diff --git a/src/test/kotlin/dev/hgh/jai/parser/JaiCorpusParserTest.kt b/src/test/kotlin/dev/hgh/jai/parser/JaiCorpusParserTest.kt index f416944..cc9df38 100644 --- a/src/test/kotlin/dev/hgh/jai/parser/JaiCorpusParserTest.kt +++ b/src/test/kotlin/dev/hgh/jai/parser/JaiCorpusParserTest.kt @@ -106,7 +106,7 @@ class JaiCorpusParserTest : BasePlatformTestCase() { private companion object { /** Ratchet. Raise it as the grammar improves; never lower it. */ - const val MIN_CLEAN_FILES = 650 + const val MIN_CLEAN_FILES = 714 const val REPORTED_ERRORS = 25 const val REPORTED_KINDS = 20 const val GOT_WIDTH = 14 diff --git a/src/test/kotlin/dev/hgh/jai/parser/JaiParserLongTailTest.kt b/src/test/kotlin/dev/hgh/jai/parser/JaiParserLongTailTest.kt new file mode 100644 index 0000000..4c5d0a5 --- /dev/null +++ b/src/test/kotlin/dev/hgh/jai/parser/JaiParserLongTailTest.kt @@ -0,0 +1,68 @@ +package dev.hgh.jai.parser + +import com.intellij.psi.PsiErrorElement +import com.intellij.psi.PsiFileFactory +import com.intellij.psi.PsiRecursiveElementWalkingVisitor +import com.intellij.testFramework.fixtures.BasePlatformTestCase +import dev.hgh.jai.JaiLanguage + +/** + * Focused regressions for syntax recovered while closing the corpus gate. + * + * [JaiCorpusParserTest] proves acceptance across the full local distribution; + * these small examples make the individual grammar contracts easy to exercise + * while iterating on generated PSI. + */ +class JaiParserLongTailTest : BasePlatformTestCase() { + fun testLongTailConstructsParseWithoutErrors() { + val snippets = + listOf( + "result=, output, error := call();", + """ + Thing :: struct { + using,except SKIP_THESE field: Field; + } + proc :: (using,except(x,y,z) value: Value) {} + """.trimIndent(), + """ + factory :: () { + callback = p => { release(p); }; + result := { callback, p }; + } + """.trimIndent(), + """ + while `loop := ready { break loop; } + if #compile_time { report(); } else { fallback(); } + if (release) #if ENABLE_STATISTICS report(); + """.trimIndent(), + """ + operator ! :: inline (value: Flag) -> bool { return value; } + push_context,defer_pop ctx; + return first, second,; + """.trimIndent(), + ) + val factory = PsiFileFactory.getInstance(project) + + for ((index, text) in snippets.withIndex()) { + val file = factory.createFileFromText("long-tail-$index.jai", JaiLanguage, text) + val errors = errorsIn(file) + assertTrue( + "Long-tail snippet $index produced parser errors: ${errors.joinToString { it.errorDescription }}\n$text", + errors.isEmpty(), + ) + } + } + + private fun errorsIn(file: com.intellij.psi.PsiFile): List { + val errors = mutableListOf() + file.accept( + object : PsiRecursiveElementWalkingVisitor() { + override fun visitErrorElement(element: PsiErrorElement) { + errors.add(element) + super.visitErrorElement(element) + } + }, + ) + return errors + } +}