The packaged change-notes still described only the lexer and highlighting, so an
installed build understated what it does and said nothing about the 58 corpus
files that still fail.
JaiFileTypeTest could previously only check the virtual file's type, because
without a ParserDefinition the PSI file fell back to plain text. It now asserts
the PSI file is a JaiFile in the Jai language and that JaiParserDefinition is the
registered definition - the wiring every PSI-dependent feature needs.
Also records the in-memory-fixture flake ('Cannot create child file at /src') as
infrastructure, with the note that JaiCorpusParserTest sidesteps it by using
PsiFileFactory directly.
Records the parser design facts a future agent needs before touching Jai.bnf:
the token-factory requirement, what lives in JaiParserUtil and why, the PSI
element that truncates an enclosing expression, longest-first operator ordering,
and directive-flag adjacency. Adds the stale-output symptoms and the sticky
--tests filter to the environment gotchas, and writes down the grammar
iteration loop.
Phase 3 is marked done; the remaining 58 corpus files are listed as the pick-up
point with the three known gaps named.
isDeclarationAhead used to skip *any* directive, so '#scope_file' was absorbed as
a modifier of the declaration after it. That misreports scope, since a scope
directive governs everything that follows it. Only #as, #overlay, #add_context
and #no_reset actually prefix a declaration in the corpus; the rest are
statements now, which the regenerated Directives.txt shows.
Golden fixtures cover the reference doc's gotcha list: declaration forms,
procedures (named/multiple/variadic/polymorphic/foreign returns), structs, enums,
unions, array and pointer types, the 'if x == { case }' switch form, for
modifiers, ifx, and directives including a here-string.
DebugParseTest is now inert by default and documented as the scratch loop.
generateParser wipes its output through a Delete task rather than a doFirst,
because a lambda added from a Kotlin build script captures the script object and
breaks the configuration cache ('./jaigradle check' failed to store it).
check and verifyPlugin (IU-253/261/262) both green.
operatorSymbol now tries multi-token forms first, so 'operator *[]' is no longer
cut short by the bare '*'. '#code'/'#insert' take a whole statement, and a
statement that already consumed its ';' satisfies the enclosing declaration.
Ratchet 650/714.
Two staleness traps cost real debugging time, and both made grammar edits look
like they had no effect:
- compileJava/compileKotlin read src/main/gen with no task dependency, so an
edit to Jai.bnf could be compiled against the previous generated parser. The
compiled JaiParser.class was three minutes older than its source while the
tests reported green. They now depend on generateParser.
- purgeOldFiles is not enough. Grammar-Kit rewrites only the files whose rule
changed, so JaiDeclaration kept a getArgumentList() its regenerated Impl no
longer had and the build failed inside generated code. generateParser now
deletes src/main/gen first.
Grammar: named return defaults use a private rule rather than 'initializer'.
Giving the default its own PSI element made the enclosing procLiteralExpr end
early, so '-> a: int = 1 { }' silently lost its body.
Corpus parse 90.9% (649/714), ratchet at 645.
A bare procedure *type* parameter such as 'proc: (info: *Info, data: T)' has no
arrow, directive or body after the parentheses, so isProcHeaderAhead now accepts
any parenthesised group that contains a ':' or '$' at depth 1.
Also: compound-assignment operator overloads, 'for <=REVERSE *=BY_POINTER',
backticked loop variables and returns, 'cast(T, value)', postfix 'x.(T)' casts,
'{}' as the zero value, '#asm AVX, AVX2 { … }', 'push_context,defer_pop', and
'#string,\%' here-strings (used in how_to/018_print_functions.jai, which the
shipped Jai_Lexer module itself rejects).
A directive prefixing a declaration no longer takes a bare operand: '#add_context
deep_copy_depth: int;' had the directive eating the declared name.
Ratchet raised to 640/714.
Grammar: 'using,except(x)' modifiers, '#ifx' with block branches, unions with
parameters, multi-value initializers, mixed 'name:, lvalue = call()' targets,
'push_context' with no explicit context, 'operator []=', '#foreign lib "symbol"',
keyword-named arguments such as 'remove={...}', and '#insert (options) body'.
Directive flags now require adjacency: without it, in
'(callback: (*GUID) #c_call, lpContext: *void)' the flag swallowed ', lpContext'.
procModifier is now just directiveExpr — one rule for directives in both
positions, so fixes apply everywhere.
jaitest: a '--tests' filter persisted through Gradle's configuration cache, so a
later unfiltered ./jaitest silently re-ran only that class and printed OK.
Unfiltered runs now disable the configuration cache.
MIN_CLEAN_FILES ratchets the corpus gate at 575/714.
Grammar gains: array types, directive flags, declaration-level directive
modifiers, parameter markers, backticked declaration names, notes after a
block-valued declaration, 'if cond then stmt', 'ifx c else v', '#module_parameters'
two-group form, '(.*)' prefix dereference, and '#asm' bodies consumed opaquely
(they are x86-64 assembly, not Jai).
Return items no longer take a default value: allowing one made
'f: (K) -> u32 = null' swallow the enclosing parameter's default and read the
rest of the parameter list as extra return values.
Adds src/main/grammar/Jai.bnf, the generated parser/PSI in src/main/gen, a
JaiParserDefinition, and the Tier 3 corpus parse gate (406/714 files clean).
Two things were not obvious:
- Grammar-Kit mints its own token instances from the .bnf 'tokens' block, which
are different objects from the ones JaiLexer emits, so every rule silently
failed to match. Fixed with tokenTypeFactory -> JaiTokenTypes.byName.
- A backslash inside an identifier is a continuation in the compiler's lexer
(Jai_Lexer/module.jai:444): 'left\_margin' is one identifier. JaiLexer now
does the same; the Tier 0 round-trip still holds because the token span
covers the backslash and the skipped spaces.
Constructs that BNF alone cannot express live in JaiParserUtil: directive-name
tests (#ident is one token), '==' before '{' for the switch form, procedure
header vs parenthesised expression, and the declaration lookahead.
- README documents install-from-disk and what actually works today
- jaitest prints the recovery steps when a poisoned build cache makes
:test report NO-SOURCE, and AGENTS.md records the failure mode
- JaiSyntaxHighlighter maps every one of the lexer's token types; asserted over
the whole corpus (105 distinct token types seen, all mapped)
- JaiHighlightingLexer refines IDENT into built-in types and loop variables for
colouring only, so the parser never sees them as reserved
- colour settings page with a demo file that is itself asserted to lex cleanly
Lexes all 714 .jai files (17.1M chars, 3.0M tokens) and asserts round-trip
tiling, strict offset progress, and zero BAD_CHARACTER. Green on first run.
jaitest: use cleanTest and retry once when a concurrent Gradle run clobbers
build/test-results (java.io.EOFException with no XML).
JaiLexer mirrors compose_new_token in the compiler's own lexer. Nested block
comments and here-strings are each consumed inside a single token, so the lexer
state is always 0 and it can restart at any token boundary.
13 plain-JUnit tests, one per gotcha in the language reference §14.
