intellijai/docs/JAI_LANGUAGE_REFERENCE.md

# Jai Language Reference (for plugin implementors)

**Provenance:** Everything here was derived from the local Jai distribution at
`~/.local/jai` — primarily `modules/Jai_Lexer/module.jai` (the compiler's own
lexer, authoritative for tokens/keywords), plus `how_to/` and `modules/`.
**No online sources were used; online Jai material is outdated.**

Anything marked ⚠️ is inferred rather than directly confirmed in the lexer.

Re-verify against: `~/.local/jai/modules/Jai_Lexer/module.jai`

---

## 1. Source layout

- Extension: `.jai`
- Encoding: UTF-8 assumed; the language imposes no encoding on `string`.
- Line endings: `\n` or `\r\n` (here-strings normalize to `\n`).
- No zero-terminated strings; no preprocessor in the C sense.

---

## 2. Comments

```jai
// line comment to end of line

/* block comment
   /* THESE NEST — track depth, do not stop at the first */
   still inside */
```

**Nested block comments are real.** `Jai_Lexer/module.jai:1436` maintains
`comment_depth`, incrementing on `/*` and decrementing on `*/`.
A lexer that stops at the first `*/` is wrong.

Note the source comment at line 1433: the lexer will treat `/////*` as a nested
open-comment (the author flags this as possibly undesirable). Match the simple
depth-counting behavior.

---

## 3. Keywords

Exact list, from `check_for_keyword` (`module.jai:758`). These are the *only*
identifiers promoted to keyword tokens.

| Len | Keywords |
| ----- | ---------- |
| 2 | `if` `xx` |
| 3 | `ifx` `for` |
| 4 | `then` `else` `null` `case` `enum` `true` `cast` |
| 5 | `while` `break` `using` `defer` `false` `union` |
| 6 | `return` `struct` `remove` `inline` |
| 7 | `size_of` `type_of` `code_of` `context` |
| 8 | `continue` `operator` |
| 9 | `type_info` `no_inline` `interface` |
| 10 | `enum_flags` |
| 11 | `is_constant` |
| 12 | `push_context` |
| 14 | `initializer_of` |

Flat list:

```text
if xx ifx for then else null case enum true cast while break using defer false
union return struct remove inline size_of type_of code_of context continue
operator type_info no_inline interface enum_flags is_constant push_context
initializer_of
```

Notes:

- `xx` is the auto-cast operator (`KEYWORD_AUTO_CAST`), not an identifier.
- `remove` is a loop-body statement for removing the current element.
- `then` is used by `ifx` (`ifx cond then a else b`).
- `interface` appears in type restrictions: `(x: $T/interface Matchable)`.
- `struct`, `union`, `enum`, `enum_flags` are all type-constructor keywords.

**Not keywords** (contrary to what one might assume): `int`, `float`, `bool`,
`string`, `s8`..`s64`, `u8`..`u64`, `float32`, `float64`, `void`, `Type`, `Any`,
`Code`. These are ordinary identifiers resolving to built-in types. Highlight
them as *built-in types*, in a distinct style from keywords, and do not let a
parser depend on them being reserved.

### Built-in type names (identifiers, highlight separately)

```text
s8 s16 s32 s64 u8 u16 u32 u64 int
float float32 float64 bool string void
Type Any Code
```

`int` is an alias for `s64`; `float` is an alias for `float32`.

---

## 4. Operators and punctuation

From `Token_Type` (`module.jai:17`). Multi-character tokens must be matched
longest-first.

### Arithmetic / assignment

```text
+  -  *  /  %
+= -= *= /= %=
```

### Comparison / logical

```text
==  !=  <  >  <=  >=  &&  ||  !
&&= ||=
```

`ISEQUAL_FOR_SWITCH_STATEMENT` is a distinct token: `==` immediately followed by
a `{` block introduces the switch form (`if x == { case ...; }`). The lexer
distinguishes it; a highlighter can ignore the distinction, a parser cannot.

### Bitwise

```text
&  |  ^  ~
&= |= ^=
<<  >>       (SHIFT_LEFT, SHIFT_RIGHT)
<<= >>=
<<< >>>      (ROTATE_LEFT, ROTATE_RIGHT — disambiguated by the parser, not the lexer)
<<<= >>>=
```

### Distinctive Jai tokens

| Token | Name | Meaning |
| ------- | ------ | --------- |
| `->` | `RIGHT_ARROW` | procedure return type |
| `..` | `DOUBLE_DOT` | inclusive range, `for 0..7` |
| `$` | | polymorphic type/value capture |
| `$$` | `DOUBLE_DOLLAR` | optionally-constant parameter |
| `---` | `TRIPLE_MINUS` | "do not initialize", `x: T = ---;` |
| `--` | `DOUBLE_MINUS` | distinct token (⚠️ not a decrement operator in normal code) |
| `,,` | `DOUBLE_COMMA` | inline context modification, `join(a,, allocator=temp)` |
| `.{` | `BEGIN_STRUCT_LITERAL` | struct literal |
| `.[` | `BEGIN_ARRAY_LITERAL` | array literal |
| `=>` | `QUICK_LAMBDA` | quick lambda, `x => x.count` |
| `.*` | `POSTFIX_DEREFERENCE` | pointer dereference, `ptr.*` |
| `*` | `POINTER_DEREFERENCE` | prefix: address-of AND pointer-type marker |
| `===` | `TRIPLE_EQUALS` | `#asm` register pinning |
| `` ` `` | backtick | see §9 |
| `:` `::` `:=` | | declaration forms, see §6 |

**Critical `*` semantics — inverted vs C.** `*T` is "pointer to T" *and* `*value`
is "address of value". Dereference is postfix `.*`. There is no prefix `*`
dereference. This matters for any expression parser.

`-` lexing (`module.jai:1680`): `->` then `---` then `--` then `-=` then `-`.

---

## 5. Literals

### Integers

```jai
123
1_000_000        // underscores allowed anywhere as separators
0xfade_deaf      // hex
0b101101101      // binary
```

Default type of an integer literal is `s64`, but literals implicitly convert to
any type they fit in.

### Floats

```jai
37.0
1.111
0h7fbf_ffff              // IEEE-754 bits in hex ("hexfloat"), 16 hex digits max
0h8000_0000_0000_0000    // 64-bit negative zero
```

Default is `float32` unless precision demands `float64`.
`Value_Flags` tracks `HEX`, `BINARY`, `FLOAT`, `REQUIRES_FLOAT64`, `OVERFLOWED`.

### Strings

```jai
"Hello, Sailor!"     // escapes: \n \t \" \\ etc.
```

Strings are `{count: s64, data: *u8}` views — **not** zero-terminated, and
subscripting yields `u8` (there is no character type).

### Here-strings (`#string`)

```jai
THE_STRING :: #string DONE
Anything at all, including "quotes" and \n literally.
DONE
```

Syntax: `#string <IDENT>` then a newline, then raw text, terminated by a line
that **starts with** `<IDENT>`. The terminator identifier is arbitrary (`DONE`
is only convention). Flagged as `Value_Flags.HERE_STRING`.
There is a `#string` variant used with imports: `#import,string #string DONE`.

For an IntelliJ lexer this needs a dedicated state with the terminator captured,
much like heredocs in shell/Perl.

### Character literals

```jai
#char "a"        // yields a u8
```

`#char` is a directive, not a `'x'` literal form. **There is no single-quote
character literal in Jai.** Do not lex `'` as a string delimiter.

### Notes (`@`)

```jai
foo :: () { }  @PrintLike @Deprecated
x: int;        @Cleanup
```

`@Identifier` is a `NOTE` token. Common in the wild: `@Cleanup`, `@Incomplete`,
`@Speed`, `@Robustness`, `@Temporary`, `@ToDo`, `@Hack`, `@Copypasta`, `@test`,
`@PrintLike`, `@NoProfile`. Treat as metadata/annotation for highlighting.
`Note_Flags.IS_SYSTEM_LEVEL` distinguishes compiler-known notes.

---

## 6. Declarations

The universal declaration form is `name : type = value`, with parts omissible:

```jai
a : float = 37.0;    // explicit type + value
b : float;           // explicit type, default-initialized to zero
c := 111.0;          // type inferred  (: and = merge into :=)
d : float : -123.45; // CONSTANT (second colon instead of =)
e :: 42;             // constant, type inferred
f : T = ---;         // explicitly UNINITIALIZED (no zeroing)
x, y, z: float;      // compound declaration
```

**Everything is zero-initialized by default** unless `= ---` is used.

`::` (constant) is how procedures, structs, and enums are declared — they are
just constant values:

```jai
main :: () { }
Person :: struct { }
Fruit :: enum u32 { }
```

This is the single most important structural fact for a parser: there is no
`func`/`fn`/`class` keyword. A top-level declaration is
`IDENT :: <struct|enum|enum_flags|union|(params)...|expr>`.

---

## 7. Procedures

```jai
name :: (a: int, b: float) -> ReturnType { ... }
name :: (a: int) -> named: ReturnType { ... }          // named return value
name :: (a: int) -> (out: [] T) { ... }                // parenthesized
name :: () -> A, B { ... }                             // multiple returns
name :: (x: int, y := 10) { ... }                      // default argument
name :: (fmt: string, args: ..Any) { ... }             // variadic
```

Calls support named arguments: `f(flavor = "chocolate", num_scoops = 3)`.

### Quick lambdas

```jai
quick_sort(to_sort, x => x.count);
quick_sort(to_sort, (a, b) => ifx a.count != b.count then b.count-a.count else 0);
count1 :: x => x.count;
```

### Operator overloading

```jai
operator + :: (x: Complex, y: Complex) -> Complex { ... }
operator == :: (a: T, b: T) -> bool #symmetric { ... }
operator [] :: (a: Bit_Array, index: int) -> bool { ... }
```

`operator` is a keyword; the token following it is the operator symbol, then
`::` and the procedure. `OPERATOR_ARRAY_SUBSCRIPT` / `OPERATOR_ASSIGNMENT_TO_ARRAY_SUBSCRIPT`
exist as parser-internal token types for `[]`.

### Polymorphism

```jai
square :: (x: $T) -> T { return x*x; }        // capture T from argument type
array_add :: (array: *[..] $T, item: T) { }   // capture inside a compound type
multiplier :: ($T: Type) { }                  // $ = must be compile-time constant
divider :: (x: int, $$ y: int) -> int { }     // $$ = optionally constant
discuss :: (x: $T/interface Matchable) { }    // type restriction via interface
proc :: (x: $T/SomeStruct) { }                // ⚠️ restriction by type
Holder :: struct ($T: Type, $N: s64) { }      // polymorphic struct
```

`$` on a parameter type captures it; `$` on a parameter name requires the
*value* be compile-time constant. `/` after a polymorphic capture introduces a
restriction.

---

## 8. Types

```jai
[8] int          // fixed-size array
[] float         // array view  {count, data}
[..] int         // resizable/dynamic array
*T               // pointer to T
[9] u8           // e.g. Phone_Number :: [9] u8;
```

### Structs

```jai
Rectangle :: struct {
    x0, y0: float;
    color_name: string;
    temperature := -10.0;      // default value
    info: Ice_Cream_Info;
    info.flavor = "chocolate"; // override a sub-struct default, inside the struct body
}
```

Struct bodies may contain assignment statements that set nested defaults.

Layout is declaration order, always, with no compiler reordering.

```jai
Video_File :: struct {
    #as using base: Document;   // #as enables implicit cast to Document
                                // using imports Document's names
}
```

### Enums

```jai
Fruits :: enum u32 {
    BANANA :: 5;      // explicit value
    APPLE;            // auto-increments
}
Fruits :: enum u32 #specified { BANANA :: 1; }   // values locked for serialization
Flags :: enum_flags u8 { A :: 0x1; B :: 0x2; }
```

**Unary dot:** enum values can be written `.APRICOT` when the type is inferred.
This means a leading `.` followed by an identifier is a valid *expression*, and
must not be confused with member access. Same for `.{` and `.[`.

### Unions

```jai
union { a: int; b: float; }
```

---

## 9. Control flow

```jai
if cond { } else if cond { } else { }
if cond  single_statement;

ifx cond then a else b                  // expression form

if value == {                           // switch form
    case .BANANA;  print("...");
    case .APPLE;   x := 1; print("...");   // each case is its own scope
    case;          print("default");       // valueless case = default
}
```

Cases **do not** fall through. To fall through, end the block with `#through`.
`#complete` on the `if` asserts all enum cases are handled.

```jai
while cond { }
break;  continue;
```

### For loops — many forms

```jai
for 1..7 { print("%", it); }             // range; implicit `it`
for i: 1..7 { }                          // named index
for a..b { }                             // runtime bounds
for < b..a { }                           // REVERSED (note the `<`)
for array { print("%", it); }            // iterate values; `it` and `it_index`
for < numbers { }                        // reversed iteration
for value, i: numbers { }                // named value and index
for * teas { }                           // iterate BY POINTER (`it` is a pointer)
for outer: numbers { for inner: numbers { } }   // named for nesting
for numbers  if (it & 1) == 0  remove it;       // `remove` current element
```

Implicit loop variables are `it` (value) and `it_index` (index). They are
ordinary identifiers, but worth special highlighting.

`for <` and `for *` are modifier tokens between `for` and the iteration
expression — a parser must accept `for`, then optional `<` and/or `*`.

### Custom iteration (`for_expansion`)

```jai
for_expansion :: (holder: Holder, body: Code, flags: For_Flags) #expand {
    for slot_index: 0..holder.count-1 {
        `it_index := slot_index;      // backtick exports the name to the caller
        `it := holder.values[slot_index];
        #insert body;
    }
}
for :positive_vibes_only holder { }     // ⚠️ select a named expansion
```

### Backticked identifiers

`` `name `` marks an identifier that a macro exports into the caller's scope.
`ident_is_backticked` is a flag on the token. Only `defer`, `return`,
`push_context`, and overloaded operator functions may be backticked among
keywords (`module.jai:1537`).

### defer

```jai
defer x += 1;         // runs on ANY exit from the enclosing scope, incl. break
defer free(ptr);
```

### Context

```jai
new_context := context;
new_context.allocator = temp;
push_context new_context { some_function(); }

join(strings,, allocator=temp);     // ,, = inline context change for one call
```

---

## 10. Casting

```jai
cast(u8) b
cast,no_check(u8) c        // modifier on cast, suppresses range check
xx value                   // auto-cast (infer target type)
xx,no_check value          // ⚠️ modifier form
```

`cast` takes a parenthesized type and an optional `,modifier` between the
keyword and the paren.

---

## 11. Compiler directives (`#`)

Directives are `#` immediately followed by an identifier. Full list observed in
the distribution, ordered by frequency (counts from `modules/`):

**Very common**
`#foreign` `#c_call` `#type` `#cpp_method` `#if` `#import` `#char` `#overlay`
`#elsewhere` `#as` `#run` `#through` `#scope_file` `#load` `#no_context`
`#library` `#type_info_none` `#expand` `#align` `#asm` `#assert` `#scope_module`
`#scope_export` `#caller_location` `#string`

**Common**
`#define` `#cpp_return_type_is_non_pod` `#no_padding` `#no_aoc`
`#bake_arguments` `#insert` `#complete` `#compiler` `#no_abc` `#include`
`#Context` `#code` `#module_parameters` `#endif` `#add_context` `#modify`
`#location` `#deprecated` `#ifdef` `#no_debug` `#symmetric` `#filepath`
`#compile_time` `#intrinsic`

**Occasional**
`#type_info_procedures_are_void_pointers` `#program_export` `#placeholder`
`#procedure_of_call` `#pragma` `#version` `#this` `#bytes`
`#type_info_no_size_complaint` `#discard` `#file` `#else` `#run_and_insert`
`#undef` `#place` `#bake` `#bake_constants` `#no_reset` `#poke_name`
`#specified` `#no_alias` `#dump` `#placeholders`

> ⚠️ Some of these (`#define`, `#include`, `#ifdef`, `#endif`, `#undef`,
> `#pragma`, `#elsewhere`) appear predominantly inside **C-binding files
> generated by `Bindings_Generator`**, which embed C preprocessor text. Do not
> assume they are core language directives.
>
> ⚠️ **`#must` does not exist in this distribution** — a full-tree grep of
> `~/.local/jai` returns zero hits. Older online documentation mentions it. This
> is exactly the kind of drift the "don't search online" instruction guards
> against. Do not add it to the keyword list.

### Semantics of the important ones

| Directive | Meaning |
| ----------- | --------- |
| `#import "Basic"` | import a module; also `#import,string`, `#import,file`, `#import,dir` |
| `#load "file.jai"` | textually include another file into this scope |
| `#run expr` | execute at compile time |
| `#if cond { }` / `#else` | static (compile-time) conditional; can appear at top level |
| `#assert cond` | compile-time assertion |
| `#scope_file` / `#scope_module` / `#scope_export` | change visibility of everything below, until the next scope directive |
| `#expand` | mark a procedure as a macro |
| `#insert code` | splice a `Code` value in |
| `#code expr` | produce a `Code` value |
| `#through` | fall through to next `case` |
| `#complete` | require exhaustive `case` coverage |
| `#specified` | lock enum values for forward compatibility |
| `#as` | allow implicit cast from this struct member |
| `#char "a"` | character byte literal |
| `#string ID` | here-string |
| `#asm { }` | inline x86-64 assembly block |
| `#foreign` / `#library` / `#c_call` | FFI |
| `#caller_location` / `#location()` / `#file` / `#filepath` | source location introspection |
| `#no_abc` / `#no_aoc` | disable array-bounds-check / arithmetic-overflow-check |
| `#place` | overlay a struct member at another member's offset |
| `#module_parameters` | parameterize a module |
| `#modify` | compile-time hook to inspect/alter polymorph resolution |
| `#deprecated "msg"` | deprecation warning |
| `#body_text` / `#poke_name` / `#dump` | metaprogramming utilities |

`#scope_file` etc. are *statement-position* directives that affect everything
following them in the file — relevant if the plugin does symbol visibility.

---

## 12. Modules and imports

```jai
#import "Basic";
#import "Math";
Sort :: #import "Sort";       // bind a module to a name
#load "other_file.jai";
#import "Foo"(PARAM = 3);     // ⚠️ module parameters
```

Module search path: `~/.local/jai/modules/`. A module is either
`Name.jai` or a directory `Name/module.jai`. Both forms exist in the
distribution (e.g. `Bit_Array.jai` vs `Basic/module.jai`).

This is the resolution rule a "go to definition" / import-completion feature
must implement.

---

## 13. Highlighting recommendations (IntelliJ token groups)

| Group | Contents |
| ------- | ---------- |
| Keyword | §3 list |
| Built-in type | §3 built-in type names |
| Directive | `#ident` |
| Note / annotation | `@Ident` |
| Number | int/hex/binary/float/hexfloat, incl. `_` separators |
| String | `"..."` |
| Here-string | `#string ID ... ID` (own token, own lexer state) |
| Line comment | `//...` |
| Block comment | `/*...*/` **nesting** |
| Operator | §4 |
| Loop variable | `it`, `it_index` |
| Polymorph | `$T`, `$$x` |
| Backtick ident | `` `name `` |
| Uninitialized | `---` |

---

## 14. Gotchas that break naive implementations

1. **Nested block comments.** Must count depth.
2. **No `'c'` char literal.** `'` is not a string delimiter. Use `#char "c"`.
3. **`*` is address-of (prefix) and pointer-type; `.*` is dereference.** Inverted from C.
4. **`.{`, `.[`, `.IDENT`** — a leading `.` is not always member access.
5. **Here-strings** need a lexer state carrying the terminator identifier.
6. **`--` vs `---` vs `->` vs `-=`** — longest-match order matters.
7. **`,,`** is one token, not two commas.
8. **`==` before `{`** is a distinct token (switch form).
9. **Primitive type names are not keywords** — don't reserve them.
10. **`#must` is not in this version.** Don't trust old online docs.
11. **`for` takes modifiers** (`<`, `*`) before the iterable.
12. **Declarations have no introducer keyword.** `Foo :: struct {}` and
    `foo :: () {}` and `foo :: 3;` are all the same syntactic shape.
13. **`remove` is a keyword**, valid only in loop bodies.
14. **`then` is a keyword**, used only by `ifx`.
15. **Struct bodies can contain assignments**, not just declarations.

---

## 15. Primary sources to re-read

| Topic | Path (under `~/.local/jai`) |
| ------- | ------------------------------ |
| Tokens, keywords, lexing | `modules/Jai_Lexer/module.jai` ← authoritative |
| Basics, declarations | `how_to/001_first.jai`, `002_number_types.jai` |
| Arrays / strings | `how_to/004_arrays.jai`, `005_strings.jai` |
| Structs / literals | `how_to/006_structs.jai`, `007_struct_literals.jai` |
| Types as values | `how_to/008_types.jai` |
| Enums | `how_to/013_enums.jai`, `014_enum_unary_dot.jai` |
| Loops | `how_to/019_looping.jai`, `730_for_expansions.jai` |
| if / ifx / switch | `how_to/022_if.jai`, `025_ifx.jai`, `027_if_case.jai` |
| Imports / scopes | `how_to/040_import_and_load/`, `080_scopes.jai`, `151_file_and_global_scopes/` |
| using | `how_to/042_using.jai`, `044_using_advanced/` |
| Operator overloading | `how_to/093_operator_overloading.jai`, `094_array_operators.jai` |
| Polymorphism | `how_to/100_`, `110_`, `115_`, `120_`, `160_type_restrictions.jai` |
| Context / `,,` | `how_to/011_context.jai`, `225_comma_comma.jai` |
| Metaprogramming | `how_to/450_basic_metaprogram/`, `600_insert.jai`, `630_compiler_get_nodes.jai` |
| Inline asm | `how_to/900_inline_assembly.jai` |
| Compiler API | `modules/Compiler/` |