From 3d03299c062d3f1427dc416ac0248182e760c5a4 Mon Sep 17 00:00:00 2001 From: Harold Cindy <120691094+HaroldCindy@users.noreply.github.com> Date: Tue, 20 Jan 2026 11:34:46 -0800 Subject: [PATCH 1/2] Add draft of static require RFC --- rfcs/static-require-bundle.md | 219 ++++++++++++++++++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 rfcs/static-require-bundle.md diff --git a/rfcs/static-require-bundle.md b/rfcs/static-require-bundle.md new file mode 100644 index 00000000..2e8a65b3 --- /dev/null +++ b/rfcs/static-require-bundle.md @@ -0,0 +1,219 @@ +# RFC: Static require() for ServerLua + +* Status: **DRAFT** + +## Summary + +Compile-time static `require()` resolution that bundles multiple modules into a single deployable unit. Client-side bundling +avoids the kind of security issues (see MySQL's `LOAD LOCAL`) where the server has to tell the client what local resources are needed +to continue bundling, and the client can't reason about what resources are needed on its own. Preserves original filenames and line numbers +for debugging via per-Proto source info in bytecode. + +## Motivation + +**Problem:** No code reuse beyond copy-paste in-viewer. Large scripts are unwieldy. LSL's `#include` loses dependency info after processing, + if you don't have the local dependencies you have no option but to edit the post-processed source code soup. + +**Why static `require()`:** SL's creation and permission models don't play nice with dynamic runtime dependencies. Bundling together dependencies when + building makes it easy for us to support multiple development styles (external editor with custom build toolchain, in-viewer editor have different opinions about + how to include files and from where) + +**Why it's tricky:** Server compiles, but dependencies live on client. Can't reasonably have server request files (i.e. `LOAD LOCAL` from MySQL). +Need debug info across modules. Want future tree shaking/inlining for optimization purposes. + +## Path Resolution + +All paths are explicit - no relative paths (`./`, `../`). + +| Pattern | Resolves to | +|---------|---------------------------------------------------------------------------------------| +| `require("foo")` | `package.path`-like semantics. Includes from the top of the package, or from libs dir | +| `require("@myalias/utils")` | User-configured alias → local path | +| `require("@sl/json")` | Platform libs (reserved namespace for LL-provided native Lua libs) | + +**Client resolves aliases before bundling.** Bundle stores alias paths; runtime does simple string lookup when actually + calling `require()` + +## Bundle Format + +Text-based, valid Luau syntax, lightly inspired by MIME multipart RFC: + +```lua +--[[!!SLUA:BUNDLE!!]] +-- NOTE: May have some metadata in the header too +-- MAIN is implicit (first section after header) +local foo = require("@myproject/lib/foo") +local json = require("@sl/json") +foo.bar() + +--[[!!SLUA:MODULE:@myproject/lib/foo!!]] +local helpers = require("@myproject/lib/helpers") +return { bar = function() return helpers.helper() end } + +--[[!!SLUA:MODULE:@myproject/lib/helpers!!]] +return { helper = function() return "hello" end } +``` + +**Rules:** +- `--[[!!SLUA:BUNDLE!!]]` header must be first line +- MAIN is implicit (content between header and first MODULE) +- `--[[!!SLUA:MODULE:path!!]]` marks each dependency +- `--[[!!SLUA:` in user source is rejected +- Platform libs (`@sl/...`) not included - provided by runtime +- Generally the user only sees and directly edits the `MAIN` bit of the bundle +- Users may define their own global aliases that refer to particular libs on their disk +- - For ex. you might `require("@textlib/v2")` to pull in v2 of your text rendering library + +Conceptually, the bundle provides a sort of virtual filesystem for the runtime `require()` implementation. + +Notably, since the bundle format includes the source code as-is, before any optimization and tree-shaking, file name +and line mappings for errors are automatically correct, without `--!@line` directives or similar. + +## Editing Workflow + +```mermaid +flowchart TD + subgraph Start + direction LR + OpenExisting[Open existing script] + CreateNew[Create new script] + end + + DownloadArchive[Download source archive] + IsValidBundle{Valid bundle?} + MainView[MAIN-only view] + RawView[Raw view] + UserEdits[User edits] + + CreateNew --> MainView + OpenExisting --> DownloadArchive + DownloadArchive --> IsValidBundle + IsValidBundle -->|Yes| MainView + IsValidBundle -->|No| RawView + MainView --> UserEdits + RawView --> UserEdits + + UserEdits -->|Toggle view| CheckUnsaved + UserEdits -->|Save| StartSave + + subgraph Toggle[Toggle View] + CheckUnsaved{Unsaved changes?} + TogglePrompt{Discard or cancel?} + CheckToggleDir{Raw to MAIN?} + ValidBundle{Valid bundle?} + DoToggle[Toggle view] + ShowParseError[Show parse error] + + CheckUnsaved -->|No| CheckToggleDir + CheckUnsaved -->|Yes| TogglePrompt + TogglePrompt -->|Discard| CheckToggleDir + CheckToggleDir -->|No| DoToggle + CheckToggleDir -->|Yes| ValidBundle + ValidBundle -->|Yes| DoToggle + ValidBundle -->|No| ShowParseError + end + + subgraph Save + StartSave[User saves] + IsRawView{Viewing raw?} + SendRaw[Send as-is] + HasLocalDep{Local dep exists?} + CheckOwnership{Same user last saved?} + BundleLocal[Bundle in local dep] + CheckMatch{Local matches bundle?} + ConfirmUntrusted{Confirm: pull local dep\ninto untrusted script?} + DepInBundle{Dep in existing bundle?} + BundleFromBundle[Bundle from existing] + ResolveError[Error: can't resolve] + SendBundle[Send bundle] + AbortSave[Abort] + + StartSave --> IsRawView + IsRawView -->|Yes| SendRaw + IsRawView -->|No| HasLocalDep + HasLocalDep -->|Yes| CheckOwnership + CheckOwnership -->|Yes| BundleLocal + CheckOwnership -->|No| CheckMatch + CheckMatch -->|Yes| BundleLocal + CheckMatch -->|No| ConfirmUntrusted + ConfirmUntrusted -->|Yes| BundleLocal + ConfirmUntrusted -->|No| AbortSave + HasLocalDep -->|No| DepInBundle + DepInBundle -->|Yes| BundleFromBundle + DepInBundle -->|No| ResolveError + ResolveError --> AbortSave + BundleLocal --> SendBundle + BundleFromBundle --> SendBundle + end + + TogglePrompt -->|Cancel| UserEdits + ShowParseError --> UserEdits + DoToggle --> UserEdits + AbortSave --> UserEdits + + SendRaw --> Compile + SendBundle --> Compile + + subgraph Server + Compile[Compile & store] + Done[Return result] + Compile --> Done + end +``` + +**Key points:** +- MAIN-only view: user sees entry script, client bundles on save +- Raw view: user sees/edits full archive, sent as-is +- Ownership check: security measure to prevent leaking local code into scripts others control +- - Open question as to how this should work, we'll need to do some server-side enrichment +- Fallback: if local dep missing, use version from downloaded bundle +- - This allows users to do one-off edits to scripts from the viewer even if they don't have all the + constituent parts on their drives. + +## Runtime + +- `require()` implemented in C (hides module table/cache from scripts) +- Module results cached after first execution +- Each module runs in sandboxed environment via `dangerouslyexecuterequiredmodule()` +- - This function will be hidden from view and not directly usable. +- Simple string lookup - no alias resolution at runtime + +## Bytecode Extension + +**Problem:** Standard Luau shares one `chunkname` across all functions. Bundles have multiple + source files. It's useful to be able to have proper filename and line mappings in errors. + +**Solution:** Per-Proto source in bytecode. Each function stores its source filename. Loader reads per-function source for correct stack traces. + +```cpp +// In lvmload.cpp +TString* protoSource = hasPerProtoSource + ? strings[readVarInt(data, size, offset)] + : source; // Fallback for non-bundles +p->source = protoSource; +``` + +## Errors + +| Error | When | Message | +|-------|------|---------| +| Dynamic require | Compile | `require() argument must be a string literal` | +| Relative path | Compile | `relative paths not supported` | +| Unknown alias | Compile | `unknown alias '@foo' in require` | +| Module not found | Compile | `cannot find module ''` | +| Circular dependency | Compile | `circular dependency: -> -> ` | +| Delimiter in source | Compile | `source cannot contain '--[[!!SLUA:'` | +| Depth exceeded | Compile | `require depth exceeds maximum` | +| Module not in bundle | Runtime | `module not found: ` | + +## Limits + +- Max dependency depth: 100 +- Max total modules: 1000 +- No circular dependencies (no different from typical Luau here!) + +## Future Work + +- Tree shaking (eliminate unused exports) +- Cross-module inlining (`--!pure` modules) +- Inventory-based module resolution From a9a2726dbf4b299fd9aab6bb31735826c7e39649 Mon Sep 17 00:00:00 2001 From: Harold Cindy <120691094+HaroldCindy@users.noreply.github.com> Date: Thu, 7 May 2026 16:13:15 -0700 Subject: [PATCH 2/2] Add bundler MVP, update RFC to match bundler --- .github/workflows/slua_bundle.yml | 51 ++ .gitignore | 7 + rfcs/static-require-bundle.md | 306 +++++++++--- tools/slua_bundle/README.md | 78 +++ tools/slua_bundle/pyproject.toml | 64 +++ tools/slua_bundle/slua_bundle/__init__.py | 63 +++ tools/slua_bundle/slua_bundle/__main__.py | 5 + tools/slua_bundle/slua_bundle/bundler.py | 217 +++++++++ tools/slua_bundle/slua_bundle/canonicalize.py | 97 ++++ tools/slua_bundle/slua_bundle/cli.py | 111 +++++ tools/slua_bundle/slua_bundle/errors.py | 12 + tools/slua_bundle/slua_bundle/extractor.py | 135 ++++++ tools/slua_bundle/slua_bundle/fs.py | 173 +++++++ tools/slua_bundle/slua_bundle/luaurc.py | 58 +++ tools/slua_bundle/slua_bundle/py.typed | 0 tools/slua_bundle/slua_bundle/resolver.py | 124 +++++ tools/slua_bundle/slua_bundle/runtime.py | 222 +++++++++ tools/slua_bundle/tests/__init__.py | 0 tools/slua_bundle/tests/_helpers.py | 10 + tools/slua_bundle/tests/test_bundler.py | 447 ++++++++++++++++++ tools/slua_bundle/tests/test_canonicalize.py | 191 ++++++++ tools/slua_bundle/tests/test_disk_fs.py | 56 +++ tools/slua_bundle/tests/test_e2e.py | 103 ++++ tools/slua_bundle/tests/test_extractor.py | 201 ++++++++ tools/slua_bundle/tests/test_memory_fs.py | 49 ++ tools/slua_bundle/tests/test_resolver.py | 106 +++++ tools/slua_bundle/tests/test_runtime.py | 227 +++++++++ 27 files changed, 3060 insertions(+), 53 deletions(-) create mode 100644 .github/workflows/slua_bundle.yml create mode 100644 tools/slua_bundle/README.md create mode 100644 tools/slua_bundle/pyproject.toml create mode 100644 tools/slua_bundle/slua_bundle/__init__.py create mode 100644 tools/slua_bundle/slua_bundle/__main__.py create mode 100644 tools/slua_bundle/slua_bundle/bundler.py create mode 100644 tools/slua_bundle/slua_bundle/canonicalize.py create mode 100644 tools/slua_bundle/slua_bundle/cli.py create mode 100644 tools/slua_bundle/slua_bundle/errors.py create mode 100644 tools/slua_bundle/slua_bundle/extractor.py create mode 100644 tools/slua_bundle/slua_bundle/fs.py create mode 100644 tools/slua_bundle/slua_bundle/luaurc.py create mode 100644 tools/slua_bundle/slua_bundle/py.typed create mode 100644 tools/slua_bundle/slua_bundle/resolver.py create mode 100644 tools/slua_bundle/slua_bundle/runtime.py create mode 100644 tools/slua_bundle/tests/__init__.py create mode 100644 tools/slua_bundle/tests/_helpers.py create mode 100644 tools/slua_bundle/tests/test_bundler.py create mode 100644 tools/slua_bundle/tests/test_canonicalize.py create mode 100644 tools/slua_bundle/tests/test_disk_fs.py create mode 100644 tools/slua_bundle/tests/test_e2e.py create mode 100644 tools/slua_bundle/tests/test_extractor.py create mode 100644 tools/slua_bundle/tests/test_memory_fs.py create mode 100644 tools/slua_bundle/tests/test_resolver.py create mode 100644 tools/slua_bundle/tests/test_runtime.py diff --git a/.github/workflows/slua_bundle.yml b/.github/workflows/slua_bundle.yml new file mode 100644 index 00000000..6398f9d2 --- /dev/null +++ b/.github/workflows/slua_bundle.yml @@ -0,0 +1,51 @@ +name: slua_bundle + +on: + push: + branches: [main, develop] + paths: + - 'tools/slua_bundle/**' + - '.github/workflows/slua_bundle.yml' + pull_request: + paths: + - 'tools/slua_bundle/**' + - '.github/workflows/slua_bundle.yml' + +defaults: + run: + working-directory: tools/slua_bundle + +jobs: + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - run: pip install ruff + - run: ruff check . + + typecheck: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - run: pip install --upgrade pip + - run: pip install . --group dev + - run: mypy slua_bundle + + test: + runs-on: ubuntu-22.04 + strategy: + matrix: + python-version: ["3.7", "3.13"] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + - run: pip install . pytest + - run: pytest diff --git a/.gitignore b/.gitignore index 4f34a306..3a48a9a1 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,10 @@ __pycache__ /fuzz/corpus/json/* !/fuzz/corpus/json/*.json *.code-workspace + +# Python tooling +*.egg-info/ +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +*.swp diff --git a/rfcs/static-require-bundle.md b/rfcs/static-require-bundle.md index 2e8a65b3..4d7b1bf3 100644 --- a/rfcs/static-require-bundle.md +++ b/rfcs/static-require-bundle.md @@ -1,6 +1,7 @@ # RFC: Static require() for ServerLua * Status: **DRAFT** +* Reference implementation: [`tools/slua_bundle/`](../tools/slua_bundle/) ## Summary @@ -9,6 +10,10 @@ avoids the kind of security issues (see MySQL's `LOAD LOCAL`) where the server h to continue bundling, and the client can't reason about what resources are needed on its own. Preserves original filenames and line numbers for debugging via per-Proto source info in bytecode. +SLua intentionally reuses upstream Luau ecosystem conventions where they exist - `.luaurc` for aliases, `init.luau` for module-as-directory entries, +Wally-style project layout - so existing Roblox/Luau tooling works without modification. SLua adds the bundle format and per-Proto source bytecode +on top; configuration is not a SLua extension surface. + ## Motivation **Problem:** No code reuse beyond copy-paste in-viewer. Large scripts are unwieldy. LSL's `#include` loses dependency info after processing, @@ -23,51 +28,230 @@ Need debug info across modules. Want future tree shaking/inlining for optimizati ## Path Resolution -All paths are explicit - no relative paths (`./`, `../`). +**Aliases are pure identity, not location.** A `require("@alias/path")` call is a logical name for a module. The path +never encodes where the module is fetched from or what kind of resolver produced it. Two scripts that share an alias +path are referring to the same logical module regardless of how each environment resolved it. + +| Pattern | Meaning | +|---------|-----------------------------------------------------------------------------------------------| +| `require("@root/utils")` | The built-in `@root/` alias always points to the project root. See [Built-in aliases](#built-in-aliases). | +| `require("@myalias/utils")` | User-declared alias from `.luaurc`. Resolved by the bundler; canonical bundle key is the alias path itself. | +| `require("./foo")`, `require("../foo")`, `require("@self/foo")` | Location-dependent. Canonicalized to absolute alias form at bundle time; see [Canonicalization](#canonicalization). | + +Bare identifiers (`require("foo")`) are not valid - upstream Luau already rejects anything without a `./`, `../`, or +`@` prefix. The `@sl/` namespace is reserved for future use. + +**Bundlers resolve aliases at bundle time.** The bundle stores each module under its canonical absolute alias key. +Runtime `require()` is a simple string lookup against the bundle's module table - no alias resolution at runtime. + +### Built-in aliases + +- `@root/` — always points to the project root. Lets scripts reference project-internal modules without depending on + the project's PROJECT name. Reserved: `.luaurc` may not declare an alias named `root`. +- `@self/` — upstream Luau convention: "current module's directory" (relative to the requiring file). Treated by the + resolver, not by `.luaurc`. Reserved — `.luaurc` may not declare an alias named `self`. + +Project-internal canonical keys always have the form `@root/...`, regardless of the bundle's PROJECT name. PROJECT is +viewer-linkage metadata only and never appears in canonical keys. + +### Canonicalization + +Every module is keyed in the bundle under an absolute alias path. The bundler rewrites the require's string constant +in emitted bytecode so runtime lookup matches that key directly. + +```mermaid +flowchart TD + Start[require S from module R] + Start --> Kind{S starts with} + Kind -->|@alias/...| Direct[Canonical key = S] + Kind -->|./, ../, @self/| Resolve[Resolve against R's location -> physical path L] + Resolve --> FindAlias{Most-specific covering alias for L} + FindAlias -->|Found| Build["Canonical key = @alias/{L relative to target}"] + FindAlias -->|None| NoAliasErr["Error: no alias spans L (shouldn't happen: @root always covers project_root)"] + Direct --> Done[Stored in bundle module table] + Build --> Done +``` + +The covering-alias set always includes `@root` for `project_root`, so files under `project_root` always canonicalize +cleanly. Files outside `project_root` need an explicit `.luaurc` alias spanning them. + +MAIN is canonicalized identically: the `MAIN` directive's value acts as MAIN's anchor key, exactly +as the MODULE marker acts for module bodies. MAIN is always present in well-formed bundles. -| Pattern | Resolves to | -|---------|---------------------------------------------------------------------------------------| -| `require("foo")` | `package.path`-like semantics. Includes from the top of the package, or from libs dir | -| `require("@myalias/utils")` | User-configured alias → local path | -| `require("@sl/json")` | Platform libs (reserved namespace for LL-provided native Lua libs) | +**Tiebreaker for identical-target aliases.** Specificity is by number of path components in the alias target (deeper +wins). When two aliases point at the exactly-same directory at the deepest covering tier: +- If `@root` is one of them, `@root` wins silently. Common case: a user declared a Wally-style alias for + `project_root` in `.luaurc` not realizing `@root` already covers it. +- Otherwise, the alphabetically-first alias name wins (ASCII byte order), and the bundler emits a warning. This is a + config smell — usually a copy-paste mistake in `.luaurc`. Alias names are assumed ASCII (Luau / Wally convention); + non-ASCII alias names are unspecified. -**Client resolves aliases before bundling.** Bundle stores alias paths; runtime does simple string lookup when actually - calling `require()` +**Worked example.** Inside `Packages/SomeLib/init.luau` with `.luaurc` defining `"SomeLib": "Packages/SomeLib"`: + +```lua +local Module = require("./src/Module") +``` + +resolves physically to `Packages/SomeLib/src/Module.luau`. The most-specific covering alias is `@SomeLib`, so the +bundle key becomes `@SomeLib/src/Module` - identical to what `require("@SomeLib/src/Module")` would produce. Wally +packages canonicalize this way without any SLua-specific handling. + +Same source tree + same `.luaurc` produces identical canonical keys regardless of which bundler ran; `.luaurc` ships +alongside the source. Resolvers returning virtual source (e.g. inventory) must declare an alias prefix for what they +return, or reject location-dependent requires inside it - a resolver concern, not the format's. ## Bundle Format -Text-based, valid Luau syntax, lightly inspired by MIME multipart RFC: +Text-based, valid Luau syntax, lightly inspired by MIME multipart RFC. The format is **producer-agnostic**: the viewer +and any external bundler emit the same artifact, and the server cannot tell which built it. ```lua ---[[!!SLUA:BUNDLE!!]] --- NOTE: May have some metadata in the header too --- MAIN is implicit (first section after header) -local foo = require("@myproject/lib/foo") -local json = require("@sl/json") +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:PROJECT myhud +-- !!LUABUNDLE:MAIN @root/HudController +-- !!LUABUNDLE:BODY +local foo = require("@root/lib/foo") foo.bar() - ---[[!!SLUA:MODULE:@myproject/lib/foo!!]] -local helpers = require("@myproject/lib/helpers") +-- !!LUABUNDLE:MODULE @root/lib/foo +local helpers = require("./helpers") return { bar = function() return helpers.helper() end } - ---[[!!SLUA:MODULE:@myproject/lib/helpers!!]] +-- !!LUABUNDLE:MODULE @root/lib/helpers return { helper = function() return "hello" end } ``` -**Rules:** -- `--[[!!SLUA:BUNDLE!!]]` header must be first line -- MAIN is implicit (content between header and first MODULE) -- `--[[!!SLUA:MODULE:path!!]]` marks each dependency -- `--[[!!SLUA:` in user source is rejected -- Platform libs (`@sl/...`) not included - provided by runtime -- Generally the user only sees and directly edits the `MAIN` bit of the bundle -- Users may define their own global aliases that refer to particular libs on their disk -- - For ex. you might `require("@textlib/v2")` to pull in v2 of your text rendering library +**Directives** (each on its own line, line-comment form): -Conceptually, the bundle provides a sort of virtual filesystem for the runtime `require()` implementation. +| Directive | Required | Purpose | +|-----------|----------|---------| +| `-- !!LUABUNDLE:VERSION ` | yes, first line | Bundle format version. Consumers MUST reject unknown versions. Future version bumps are non-back-compatible unless explicitly stated. | +| `-- !!LUABUNDLE:PROJECT ` | no | Advisory viewer-linkage metadata; associates the bundle with a disk project. Not used for canonicalization. See [Project Linkage](#project-linkage). | +| `-- !!LUABUNDLE:MAIN ` | yes | Canonical key for the MAIN section (e.g. `@root/HudController`). Anchors `./`, `../`, and `@self/` requires inside MAIN. Always emitted by the reference bundler; computed from MAIN's path under `project_root`. | +| `-- !!LUABUNDLE:BODY` | yes | Separates the header from MAIN's source body. Everything between BODY and the first MODULE marker (or EOF) is MAIN. | +| `-- !!LUABUNDLE:MODULE ` | per module | Marks each dependency. The canonical key identifies the module for runtime `require()` lookup. | -Notably, since the bundle format includes the source code as-is, before any optimization and tree-shaking, file name -and line mappings for errors are automatically correct, without `--!@line` directives or similar. +**Other rules:** +- Lines matching `^-- *!!LUABUNDLE:` in user source are rejected at bundle time. Users may not author lines that look like bundle directives. +- Generally the user only sees and directly edits the `MAIN` body of the bundle. +- Users may define their own aliases (via `.luaurc`) referring to libs on disk; see [Configuration & Project Layout](#configuration--project-layout). + +The bundle conceptually provides a virtual filesystem for the runtime `require()` implementation. Because the format +includes source code as-is - before any optimization or tree-shaking - filenames and line mappings for errors are +automatically correct without `--!@line` directives. + +## Configuration & Project Layout + +SLua reuses Luau ecosystem conventions for project configuration. **No SLua-specific config file is introduced.** + +### Alias config: `.luaurc` + +`.luaurc` is the standard Luau alias config (JSON, Wally-compatible). SLua bundlers read it the same way `lune`, the +`luau` CLI, and Roblox Studio do: + +```json +{ + "languageMode": "strict", + "aliases": { + "Pkg": "Packages" + } +} +``` + +Project-internal modules don't need a user alias — the built-in `@root/` covers `project_root`. `.luaurc` is for +declaring external aliases (Wally packages, vendored libs, virtual-source resolvers, etc.). The alias name `root` is +reserved. + +**SLua bundlers read only the `.luaurc` at `project_root`.** Nested `.luaurc` files deeper in the tree are ignored; +files above `project_root` are never consulted. This deviates from upstream Luau's walking behavior on purpose: +bundle reproducibility requires the alias set to be a function of the project tree alone. Walking up past +`project_root` would let aliases bleed in from whatever directory the developer happened to bundle from, breaking +identical-source-tree-gives-identical-canonical-keys. + +### Recommended directory layout + +Mirror Rojo/Wally idiom: + +``` +myhud/ # project_root (any name; not load-bearing) + .luaurc # aliases for external deps (Wally-friendly, Roblox-compatible) + wally.toml # optional, Wally manifest + Packages/ # Wally-vendored deps (e.g. @SomeLib) + src/ + HudController.luau # top of src/ -> SL script "HudController" (canonical: @root/src/HudController) + DataLink.luau # top of src/ -> SL script "DataLink" (canonical: @root/src/DataLink) + lib/ + utils.luau # nested -> helper, @root/src/lib/utils + shared.luau # nested -> helper, @root/src/lib/shared +``` + +**Convention:** files at the **immediate root of `src/`** become SL scripts (one each, named after the disk filename). +Files in **any subdirectory** are project-local helpers, never deployed as standalone scripts. A CLI deploy tool +(`slua deploy ` or similar) reads the top of `src/`, bundles each top-level file with its required modules, +and uploads one SL script per file. No manifest is needed; the directory shape is the manifest. + +This dissolves the question of what "the main file" is named at the script level: the SL script name *is* the disk +filename. There is no `main.luau` magic. + +### Module-as-directory entry: `init.luau` + +When an alias resolves to a directory (e.g., `require("@coollib")` where `coollib` is a directory), the entry file is +`init.luau`. **This is upstream Luau's convention** (see `Require/include/Luau/Require.h:33-34`), adopted unchanged so +SLua require behaves identically to Lute, Roblox, and the `luau` CLI for directory aliases. `init.luau` has no special +meaning at the SL-script level - only for module-as-directory. + +### Wally compatibility + +SLua projects are normal Luau projects. `wally install` populates `Packages/`, `.luaurc` aliases (`@Pkg/...`) point at +it, the bundler reads disk normally. No SLua-side configuration is needed beyond what any other Luau tool already +supports. + +## Resolver Behaviour + +A bundler maps an alias path to source code at bundle time. The RFC commits only to **minimum** behaviour and stays +silent on the rest: + +- Bundlers **SHOULD** support `.luaurc`-based disk alias resolution. This is the Wally/Roblox compatibility floor. +- Bundlers **MAY** support additional resolvers (inventory, marketplace, http registries, etc.). These are entirely an + implementation choice; the RFC neither enumerates nor specifies them. +- The bundle's embedded source is the **universal last-resort resolver**. Any bundler can rebundle a bundle it received - + even if it cannot reach the original sources - by reusing the bundled copies of each module. + +The last-resort resolver is what keeps a viewer-authored bundle re-buildable from an external CLI without inventory +access: when an alias has no local resolver, the bundler falls back to the source already embedded in the previous +bundle. The script never breaks just because the new environment lacks the original resolver. + +## Project Linkage + +The bundle header's optional `PROJECT ` directive lets a viewer associate a script with a local disk project +for editing. The RFC specifies the **contract** only: + +- A viewer MAY maintain bindings from project names (and optionally per-object UUID overrides) to local disk directories. +- Resolvers see the path resulting from the binding; how it is stored is viewer business. +- Bindings are per-user-per-machine, never synced to SL servers, and not assumed identical between collaborators. +- Header is metadata only; an unset project name just means "no disk linkage known here." A bundler that ignores it + works correctly. + +### Non-normative appendix: registry sketch + +A minimal viewer-internal registry could look like: + +```json +{ + "projects": { + "myhud": "/Users/me/slua/myhud", + "vendor-system": "/Users/me/slua/vending" + }, + "objects": { + "01234567-89ab-cdef-0123-456789abcdef": "/Users/me/slua/oldhud" + } +} +``` + +Resolution order: object UUID override -> project header name -> unbound. + +This appendix is illustrative. TPVs may store bindings however suits their codebase; only the contract above is +load-bearing. + +Viewer-to-disk **sync-back** (writing MAIN edits back to a bound disk project) is deferred to a follow-up RFC. ## Editing Workflow @@ -117,11 +301,11 @@ flowchart TD StartSave[User saves] IsRawView{Viewing raw?} SendRaw[Send as-is] - HasLocalDep{Local dep exists?} + ResolverSucceeds{Resolver produces source?} CheckOwnership{Same user last saved?} - BundleLocal[Bundle in local dep] - CheckMatch{Local matches bundle?} - ConfirmUntrusted{Confirm: pull local dep\ninto untrusted script?} + BundleResolved[Bundle from resolver] + CheckMatch{Resolver matches bundle?} + ConfirmUntrusted{Confirm: pull resolved source\ninto untrusted script?} DepInBundle{Dep in existing bundle?} BundleFromBundle[Bundle from existing] ResolveError[Error: can't resolve] @@ -130,19 +314,19 @@ flowchart TD StartSave --> IsRawView IsRawView -->|Yes| SendRaw - IsRawView -->|No| HasLocalDep - HasLocalDep -->|Yes| CheckOwnership - CheckOwnership -->|Yes| BundleLocal + IsRawView -->|No| ResolverSucceeds + ResolverSucceeds -->|Yes| CheckOwnership + CheckOwnership -->|Yes| BundleResolved CheckOwnership -->|No| CheckMatch - CheckMatch -->|Yes| BundleLocal + CheckMatch -->|Yes| BundleResolved CheckMatch -->|No| ConfirmUntrusted - ConfirmUntrusted -->|Yes| BundleLocal + ConfirmUntrusted -->|Yes| BundleResolved ConfirmUntrusted -->|No| AbortSave - HasLocalDep -->|No| DepInBundle + ResolverSucceeds -->|No| DepInBundle DepInBundle -->|Yes| BundleFromBundle DepInBundle -->|No| ResolveError ResolveError --> AbortSave - BundleLocal --> SendBundle + BundleResolved --> SendBundle BundleFromBundle --> SendBundle end @@ -162,13 +346,14 @@ flowchart TD ``` **Key points:** -- MAIN-only view: user sees entry script, client bundles on save -- Raw view: user sees/edits full archive, sent as-is -- Ownership check: security measure to prevent leaking local code into scripts others control -- - Open question as to how this should work, we'll need to do some server-side enrichment -- Fallback: if local dep missing, use version from downloaded bundle -- - This allows users to do one-off edits to scripts from the viewer even if they don't have all the - constituent parts on their drives. +- MAIN-only view: user sees entry script, client bundles on save by running its resolver chain. +- Raw view: user sees/edits full archive, sent as-is. This path also serves externally-built bundles uploaded directly. +- Resolver chain: each alias is tried against the bundler's resolvers (`.luaurc` disk minimum; viewer may add inventory or others). Chain order and resolver types are bundler-specific (see [Resolver Behaviour](#resolver-behaviour)). +- Ownership check: security measure to prevent leaking resolved source into a script the current user does not own. +- - Open question as to how this should work; needs server-side enrichment. +- Universal fallback: if no resolver succeeds, use the version embedded in the previous bundle. This lets users do + one-off edits even when they lack the original resolver context (e.g., editing in an external CLI a script that was + originally bundled by a viewer with inventory access). ## Runtime @@ -198,14 +383,28 @@ p->source = protoSource; | Error | When | Message | |-------|------|---------| | Dynamic require | Compile | `require() argument must be a string literal` | -| Relative path | Compile | `relative paths not supported` | +| Bare identifier | Compile | `require path must start with './', '../', or '@'` | | Unknown alias | Compile | `unknown alias '@foo' in require` | -| Module not found | Compile | `cannot find module ''` | +| Require escapes alias | Compile | `require '' from : '..' traverses past alias root` | +| Invalid path component | Compile | `component '' starts with '.'` / `component contains NUL` | +| Relative require without anchor | Compile | MAIN uses `./`, `../`, or `@self/` but the bundle has no `MAIN` directive | +| No covering alias | Bundle time | `no alias spans ; add a .luaurc entry covering it` (won't fire for files under `project_root` since `@root` always covers it) | +| Reserved alias | Bundle time | `.luaurc` declares an alias name reserved by the format (`root`, `self`) | +| Alias collision | Bundle time | Two `.luaurc` aliases point at the exact same directory; emitted as a warning, not an error | +| Ambiguous file resolution | Bundle time | Both `.luau` and `/init.luau` exist; remove one to disambiguate | +| No resolver succeeded | Bundle time | `cannot resolve module '': no resolver produced source and no copy in existing bundle` | | Circular dependency | Compile | `circular dependency: -> -> ` | -| Delimiter in source | Compile | `source cannot contain '--[[!!SLUA:'` | +| Delimiter in source | Compile | `source cannot contain '-- !!LUABUNDLE:'` | | Depth exceeded | Compile | `require depth exceeds maximum` | +| Duplicate MODULE marker | Parse | Bundle contains two `MODULE` directives with the same canonical key | +| Malformed module key | Parse | A `MODULE` key does not begin with `@` | +| Missing MAIN directive | Parse | Bundle has no `MAIN` directive | | Module not in bundle | Runtime | `module not found: ` | +The reference Python bundler raises the bundle-time and parse errors directly. The production C++ Luau compiler will +surface the compile-time errors at compile time; the reference bundler raises analogous errors at bundle time as a +stand-in. + ## Limits - Max dependency depth: 100 @@ -216,4 +415,5 @@ p->source = protoSource; - Tree shaking (eliminate unused exports) - Cross-module inlining (`--!pure` modules) -- Inventory-based module resolution +- Viewer-to-disk sync-back semantics (separate RFC, after viewer implementation experience) +- Marketplace / registry resolver designs (whatever shape they take, they sit alongside disk and inventory as additional bundler-side resolvers; the bundle format does not need to know) diff --git a/tools/slua_bundle/README.md b/tools/slua_bundle/README.md new file mode 100644 index 00000000..3fec4d80 --- /dev/null +++ b/tools/slua_bundle/README.md @@ -0,0 +1,78 @@ +# slua-bundle + +Reference implementation of the SLua static-require-bundle algorithm. Bundle a Luau project rooted at a `MAIN` module into a single text artifact, inspect bundle artifacts, and extract bundles back into a self-contained project tree. + +For the format spec and design rationale, see [`rfcs/static-require-bundle.md`](../../rfcs/static-require-bundle.md) at the repo root. + +## Install + +Requires Python 3.7+. + +``` +pip install ./tools/slua_bundle +``` + +Or, for development: + +``` +pip install -e ./tools/slua_bundle +``` + +Either form puts a `slua-bundle` command on `$PATH`. + +## CLI + +### `slua-bundle bundle` + +Bundle a project tree into a single text artifact. Project-internal modules canonicalize under the built-in `@root` alias. `--project` is an optional advisory project name emitted as the bundle's `PROJECT` directive (for viewer-side linkage to a disk project); it does not affect canonicalization. + +``` +slua-bundle bundle --root ./src ./src/Main.luau -o bundle.lua +slua-bundle bundle --root ./src --project myhud ./src/Main.luau -o bundle.lua +``` + +Omit `-o` to write to stdout. + +Pass `--input-bundle PATH` to use an existing bundle as a last-resort resolver: when a require's source is missing on disk (or its alias isn't declared in `.luaurc`), the embedded copy from the prior bundle is used instead. Disk wins when both are available. This is the rebundle-without-source flow - useful when a CLI receives a viewer-built bundle and needs to rebuild it without access to the original resolvers (inventory, etc.). + +### `slua-bundle inspect` + +Show a bundle's structure: format version, advisory project name (if any), MAIN entry point, and a per-module byte breakdown sorted by canonical key. + +``` +slua-bundle inspect bundle.lua +``` + +### `slua-bundle extract` + +Reverse a bundle into a project tree. `@root` modules land flat under `/`, non-root aliases under `//`. A `.luaurc` is generated when non-root aliases are present. Re-bundling the extracted tree reproduces the original bundle byte-for-byte. + +``` +slua-bundle extract -o ./extracted bundle.lua +``` + +## Library + +```python +from pathlib import Path +from slua_bundle import DiskFS, bundle, parse_bundle +from slua_bundle.extractor import extract_to_dir + +fs = DiskFS(Path("./src")) +text = bundle(fs, project_root=Path("./src").resolve(), + main_path=Path("./src/Main.luau").resolve()) +# Optionally pass project_name="myhud" to emit a PROJECT directive. + +parsed = parse_bundle(text) +extract_to_dir(parsed, DiskFS(Path("./extracted")), Path("./extracted").resolve()) +``` + +`MemoryFS` is also available for dict-backed in-memory use, mirroring `DiskFS`. + +## Development + +``` +pip install -e ./tools/slua_bundle +cd tools/slua_bundle +python -m pytest +``` diff --git a/tools/slua_bundle/pyproject.toml b/tools/slua_bundle/pyproject.toml new file mode 100644 index 00000000..734e306f --- /dev/null +++ b/tools/slua_bundle/pyproject.toml @@ -0,0 +1,64 @@ +[build-system] +requires = ["setuptools"] +build-backend = "setuptools.build_meta" + +[project] +name = "slua-bundle" +version = "0.1.0" +description = "Reference implementation of the SLua static-require-bundle algorithm: bundle, inspect, and extract Luau projects." +requires-python = ">=3.7" +readme = "README.md" + +[project.scripts] +slua-bundle = "slua_bundle.cli:main" + +[tool.setuptools] +packages = ["slua_bundle"] + +[tool.setuptools.package-data] +slua_bundle = ["py.typed"] + +[dependency-groups] +dev = [ + "mypy>=1.0", + "pytest>=8", + "pytest-cov>=6", + "ruff>=0.8", +] +test = [ + "pytest>=8", + "pytest-cov", +] + +[tool.pytest.ini_options] +testpaths = ["tests"] +pythonpath = ["."] + +[tool.ruff] +line-length = 100 +target-version = "py37" + +[tool.ruff.lint] +select = ["B", "C", "E", "F", "W", "I", "C90", "RUF"] +ignore = ["E501", "RUF012"] + +[tool.ruff.lint.isort] +combine-as-imports = true + +[tool.ruff.lint.mccabe] +max-complexity = 25 + +[tool.mypy] +follow_imports = "normal" +strict_optional = true + +[tool.coverage.run] +source = ["slua_bundle"] +branch = true + +[tool.coverage.report] +exclude_lines = [ + "pragma: no cover", + "if TYPE_CHECKING:", + "raise NotImplementedError", +] diff --git a/tools/slua_bundle/slua_bundle/__init__.py b/tools/slua_bundle/slua_bundle/__init__.py new file mode 100644 index 00000000..6e794e59 --- /dev/null +++ b/tools/slua_bundle/slua_bundle/__init__.py @@ -0,0 +1,63 @@ +from .bundler import ( + AmbiguousResolutionError, + DepthExceededError, + MarkerInjectionError, + ModuleCountExceededError, + NoResolverError, + bundle, +) +from .canonicalize import ( + AliasCollisionWarning, + NoCoveringAliasError, + ReservedAliasError, + canonicalize, +) +from .errors import BundleError +from .fs import DiskFS, FSBackend, MemoryFS +from .luaurc import InvalidLuaurcError +from .resolver import ( + BareIdentifierError, + InvalidPathComponentError, + RelativeRequireWithoutAnchorError, + RequireEscapesAliasError, + UnknownAliasError, + resolve, +) +from .runtime import ( + BundleParseError, + BundleParser, + CircularDependencyError, + ParsedBundle, + parse_bundle, + simulate, +) + +__all__ = [ + "AliasCollisionWarning", + "AmbiguousResolutionError", + "BareIdentifierError", + "BundleError", + "BundleParseError", + "BundleParser", + "CircularDependencyError", + "DepthExceededError", + "DiskFS", + "FSBackend", + "InvalidLuaurcError", + "InvalidPathComponentError", + "MarkerInjectionError", + "MemoryFS", + "ModuleCountExceededError", + "NoCoveringAliasError", + "NoResolverError", + "ParsedBundle", + "RelativeRequireWithoutAnchorError", + "RequireEscapesAliasError", + "ReservedAliasError", + "UnknownAliasError", + "bundle", + "canonicalize", + "parse_bundle", + "resolve", + "simulate", +] diff --git a/tools/slua_bundle/slua_bundle/__main__.py b/tools/slua_bundle/slua_bundle/__main__.py new file mode 100644 index 00000000..dd8a8c90 --- /dev/null +++ b/tools/slua_bundle/slua_bundle/__main__.py @@ -0,0 +1,5 @@ +import sys + +from .cli import main + +sys.exit(main()) diff --git a/tools/slua_bundle/slua_bundle/bundler.py b/tools/slua_bundle/slua_bundle/bundler.py new file mode 100644 index 00000000..19e1b8d7 --- /dev/null +++ b/tools/slua_bundle/slua_bundle/bundler.py @@ -0,0 +1,217 @@ +from __future__ import annotations + +import re +from collections import deque +from pathlib import PurePath + +from .canonicalize import build_alias_map, canonicalize +from .errors import BundleError +from .fs import FSBackend, normalize +from .resolver import _split_canonical, resolve +from .runtime import parse_bundle + +BUNDLE_VERSION = 1 + +MAX_DEPTH = 100 +MAX_MODULES = 1000 + +REQUIRE_RE = re.compile(r'\brequire\s*\(\s*(?:"([^"]*)"|\'([^\']*)\')\s*\)') +_MARKER_RE = re.compile(r"^--\s*!!LUABUNDLE:", re.MULTILINE) + + +class MarkerInjectionError(BundleError): + pass + + +class AmbiguousResolutionError(BundleError): + """Both .luau and /init.luau exist for a resolved require.""" + + +class NoResolverError(BundleError): + """No resolver produced source and no copy was found in the existing bundle.""" + + +class DepthExceededError(BundleError): + """The require graph exceeds MAX_DEPTH levels from MAIN.""" + + +class ModuleCountExceededError(BundleError): + """The require graph exceeds MAX_MODULES total modules.""" + + +def _check_no_marker_injection(origin: str, source: str) -> None: + if _MARKER_RE.search(source): + raise MarkerInjectionError( + f"{origin} contains a line that looks like a bundle marker; " + "source files must not contain '-- !!LUABUNDLE:' at the start of a line" + ) + + +def bundle( + vfs: FSBackend, + project_root: PurePath, + main_path: PurePath, + project_name: str | None = None, + existing_bundle: str | None = None, +) -> str: + """Bundle MAIN and everything it transitively requires. + + Pure-trace: starts at MAIN, regex-finds `require()` calls, resolves each + to a physical file, and enqueues unvisited targets. Files unreachable + from MAIN never enter the bundle. Aliases come from project_root's + .luaurc only (nested .luaurc files are ignored for reproducibility); + the built-in @root alias always covers project_root. + + project_name, if provided, is emitted as the bundle's PROJECT directive + (advisory viewer-linkage metadata) but is not used for canonicalization. + + existing_bundle, if provided, acts as the universal last-resort resolver: + when an alias has no on-disk source (or no .luaurc entry), the bundler + falls back to the source already embedded in that bundle. Disk wins + when both are available. + """ + project_root = normalize(project_root) + main_path = normalize(main_path) + + alias_map = build_alias_map(vfs, project_root) + main_key = canonicalize(vfs, main_path, project_root) + main_source = vfs.read(main_path) + _check_no_marker_injection(str(main_path), main_source) + + fallback_modules: dict[str, str] = {} + if existing_bundle is not None: + parsed = parse_bundle(existing_bundle) + fallback_modules.update(parsed.modules) + fallback_modules[parsed.fields["main"]] = parsed.main_source + + known_aliases = set(alias_map.keys()) + for key in fallback_modules: + known_aliases.add(key[1:].split("/", 1)[0]) + + visited: dict[str, str] = {} + queue: deque[tuple[str, str, int]] = deque() + queue.append((main_key, main_source, 0)) + enqueued: set[str] = {main_key} + + while queue: + key, source, depth = queue.popleft() + if depth > MAX_DEPTH: + raise DepthExceededError( + f"require depth {depth} exceeds maximum {MAX_DEPTH} at module {key}" + ) + if key in visited: + continue + visited[key] = source + + for dq, sq in REQUIRE_RE.findall(source): + req_str = dq or sq + target_key, target_source = _resolve_source( + vfs, alias_map, fallback_modules, project_root, + key, req_str, known_aliases, + ) + if target_key in enqueued: + continue + _check_no_marker_injection(target_key, target_source) + if len(enqueued) >= MAX_MODULES: + raise ModuleCountExceededError( + f"bundle exceeds maximum of {MAX_MODULES} modules" + ) + enqueued.add(target_key) + queue.append((target_key, target_source, depth + 1)) + + main_body = visited.pop(main_key) + + parts: list[str] = [f"-- !!LUABUNDLE:VERSION {BUNDLE_VERSION}\n"] + if project_name is not None: + parts.append(f"-- !!LUABUNDLE:PROJECT {project_name}\n") + parts.append(f"-- !!LUABUNDLE:MAIN {main_key}\n") + parts.append("-- !!LUABUNDLE:BODY\n") + parts.append(_terminate(main_body)) + for key, source in visited.items(): + parts.append(f"-- !!LUABUNDLE:MODULE {key}\n") + parts.append(_terminate(source)) + return "".join(parts) + + +def _resolve_source( + vfs: FSBackend, + alias_map: dict[str, PurePath], + fallback_modules: dict[str, str], + project_root: PurePath, + requirer_anchor_key: str, + require_str: str, + known_aliases: set[str], +) -> tuple[str, str]: + """Resolve a require() string to (canonical_key, source). + + Disk resolver wins; the fallback bundle's embedded copy is the universal + last resort (RFC: 'Resolver Behaviour'). When the alias isn't even known + to disk, fall back directly. + + On disk hits, re-canonicalize from the physical path so colliding aliases + (two .luaurc entries targeting the same directory) de-dup to a single + canonical key and the collision warning fires from canonicalize(). + """ + target_key = resolve(require_str, requirer_anchor_key, known_aliases) + alias, rel_parts = _split_canonical(target_key) + + if alias in alias_map: + target_path = _luau_resolve_to_file(vfs, alias_map[alias], rel_parts) + if target_path is not None: + canonical_key = canonicalize(vfs, target_path, project_root) + return canonical_key, vfs.read(target_path) + + if target_key in fallback_modules: + return target_key, fallback_modules[target_key] + + raise NoResolverError( + f"cannot resolve module '{require_str}' to '{target_key}': " + "no resolver produced source and no copy in existing bundle" + ) + + +def _luau_resolve_to_file( + vfs: FSBackend, + base_dir: PurePath, + rel_parts: list[str], +) -> PurePath | None: + """Apply Luau's file resolution: try /.../.luau, then /...//init.luau. + + Returns None when no file is found - the caller decides whether to try + another resolver (existing-bundle fallback) or surface NoResolverError. + Raises AmbiguousResolutionError when both candidate files exist. + """ + if not rel_parts: + # `@SomeAlias` with no tail: the alias root itself is the module. + # Look for init.luau directly in the alias root. + candidate = base_dir / "init.luau" + return candidate if vfs.is_file(candidate) else None + + leaf = rel_parts[-1] + parent_dir = base_dir + for p in rel_parts[:-1]: + parent_dir = parent_dir / p + + leaf_file = parent_dir / f"{leaf}.luau" + init_file = parent_dir / leaf / "init.luau" + + leaf_exists = vfs.is_file(leaf_file) + init_exists = vfs.is_file(init_file) + + if leaf_exists and init_exists: + raise AmbiguousResolutionError( + f"both {leaf_file} and {init_file} exist; remove one to disambiguate" + ) + if leaf_exists: + return leaf_file + if init_exists: + return init_file + return None + + +def _terminate(content: str) -> str: + """Force exactly one trailing newline so the next marker lands at column 0. + + Body content is otherwise preserved byte-for-byte. + """ + return content if content.endswith("\n") else content + "\n" diff --git a/tools/slua_bundle/slua_bundle/canonicalize.py b/tools/slua_bundle/slua_bundle/canonicalize.py new file mode 100644 index 00000000..c229ae84 --- /dev/null +++ b/tools/slua_bundle/slua_bundle/canonicalize.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import warnings +from pathlib import PurePath + +from .errors import BundleError +from .fs import FSBackend, normalize +from .luaurc import load_config + + +RESERVED_ALIASES = frozenset({"root", "self"}) + + +class NoCoveringAliasError(BundleError): + pass + + +class ReservedAliasError(BundleError): + pass + + +class AliasCollisionWarning(UserWarning): + pass + + +def _is_prefix(prefix: PurePath, path: PurePath) -> bool: + if prefix == path: + return True + return prefix in path.parents + + +def build_alias_map( + vfs: FSBackend, + project_root: PurePath, +) -> dict[str, PurePath]: + """Project-root .luaurc + the built-in @root alias. + + Nested .luaurc files are ignored: alias set must be a function of the + project tree alone for bundle reproducibility. Reserved alias names + (currently just 'root') cannot be declared in .luaurc. + """ + project_root = normalize(project_root) + aliases = load_config(vfs, project_root) + for reserved in RESERVED_ALIASES: + if reserved in aliases: + raise ReservedAliasError( + f"alias '{reserved}' is reserved and cannot be declared in .luaurc" + ) + aliases["root"] = project_root + return aliases + + +def canonicalize( + vfs: FSBackend, + file_path: PurePath, + project_root: PurePath, +) -> str: + file_path = normalize(file_path) + project_root = normalize(project_root) + + aliases = build_alias_map(vfs, project_root) + covering = [(target, name) for name, target in aliases.items() if _is_prefix(target, file_path)] + if not covering: + raise NoCoveringAliasError( + f"no alias spans {file_path}, add a .luaurc entry covering it" + ) + + max_depth = max(len(t.parts) for t, _ in covering) + most_specific = [(t, n) for t, n in covering if len(t.parts) == max_depth] + # Two aliases at identical depth that both cover the same file must share + # the same target path (only one directory at any given depth can be an + # ancestor of a given file), so target_dir below is unambiguous. + target_dir = most_specific[0][0] + + if len(most_specific) > 1: + names = sorted(n for _, n in most_specific) + if "root" in names: + alias_name = "root" + else: + alias_name = names[0] + warnings.warn( + f"multiple .luaurc aliases target the same directory {target_dir}: " + f"{names}; using '{alias_name}' (ASCII-first) for canonical keys", + AliasCollisionWarning, + stacklevel=2, + ) + else: + alias_name = most_specific[0][1] + + rel = file_path.relative_to(target_dir) + rel_parts = list(rel.with_suffix("").parts) + if rel_parts and rel_parts[-1] == "init": + rel_parts.pop() + + if not rel_parts: + return f"@{alias_name}" + return f"@{alias_name}/" + "/".join(rel_parts) diff --git a/tools/slua_bundle/slua_bundle/cli.py b/tools/slua_bundle/slua_bundle/cli.py new file mode 100644 index 00000000..ebf7e5da --- /dev/null +++ b/tools/slua_bundle/slua_bundle/cli.py @@ -0,0 +1,111 @@ +"""Command-line interface to the bundler. + +Subcommands: + bundle -- produce a bundle from a project on disk + inspect -- pretty-print a bundle's structure + extract -- reverse a bundle into a self-contained project tree + +Errors raised by the library are caught at the top of `main()` and +printed to stderr with exit code 1; unexpected exceptions propagate so +stack traces stay visible during prototype development. +""" + +from __future__ import annotations + +import argparse +import pathlib +import sys + +from .bundler import bundle +from .errors import BundleError +from .extractor import extract_to_dir +from .fs import DiskFS +from .runtime import parse_bundle + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(prog="python -m slua_bundle") + sub = parser.add_subparsers(dest="cmd", required=True) + + p_bundle = sub.add_parser("bundle", help="Bundle a project tree") + p_bundle.add_argument("--project", default=None, help="Optional advisory project name (emitted as PROJECT directive for viewer linkage)") + p_bundle.add_argument("--root", required=True, type=pathlib.Path, help="Project root directory (covered by built-in @root alias)") + p_bundle.add_argument("--input-bundle", dest="input_bundle", type=pathlib.Path, default=None, help="Existing bundle whose embedded modules act as a last-resort resolver when disk sources are missing") + p_bundle.add_argument("-o", "--output", type=pathlib.Path, default=None, help="Write bundle to this file (default: stdout)") + p_bundle.add_argument("main", type=pathlib.Path, help="Path to MAIN .luau file") + + p_inspect = sub.add_parser("inspect", help="Show a bundle's structure") + p_inspect.add_argument("bundle_file", type=pathlib.Path, nargs="?", default=None, help="Bundle file to inspect (default: stdin)") + + p_extract = sub.add_parser("extract", help="Reverse a bundle into a project tree") + p_extract.add_argument("-o", "--output", required=True, type=pathlib.Path, help="Output directory") + p_extract.add_argument("bundle_file", type=pathlib.Path, help="Bundle file to extract") + + return parser + + +def _cmd_bundle(args: argparse.Namespace) -> int: + disk = DiskFS(args.root) + existing = args.input_bundle.read_text() if args.input_bundle else None + text = bundle( + disk, + project_root=args.root.resolve(), + main_path=args.main.resolve(), + project_name=args.project, + existing_bundle=existing, + ) + if args.output: + args.output.write_text(text) + else: + sys.stdout.write(text) + return 0 + + +def _read_bundle_text(arg: pathlib.Path | None) -> str: + if arg is None: + return sys.stdin.read() + return arg.read_text() + + +def _cmd_inspect(args: argparse.Namespace) -> int: + text = _read_bundle_text(args.bundle_file) + parsed = parse_bundle(text) + print(f"VERSION {parsed.fields.get('version', '?')}") + project = parsed.fields.get('project') + print(f"PROJECT {project}" if project else "PROJECT ") + main_key = parsed.fields.get("main") + main_size = len(parsed.main_source.encode("utf-8")) + if main_key: + print(f"MAIN {main_key} ({main_size} bytes)") + else: + print(f"MAIN ({main_size} bytes)") + modules = sorted(parsed.modules.items()) + total = sum(len(s.encode("utf-8")) for _, s in modules) + print(f"MODULES ({len(modules)}, total {total} bytes):") + for key, source in modules: + print(f" {key} ({len(source.encode('utf-8'))} bytes)") + return 0 + + +def _cmd_extract(args: argparse.Namespace) -> int: + text = args.bundle_file.read_text() + parsed = parse_bundle(text) + extract_to_dir(parsed, DiskFS(args.output), args.output.resolve()) + n = 1 + len(parsed.modules) + print(f"extracted {n} modules to {args.output}") + return 0 + + +def main(argv: list[str] | None = None) -> int: + parser = _build_parser() + args = parser.parse_args(argv) + handlers = { + "bundle": _cmd_bundle, + "inspect": _cmd_inspect, + "extract": _cmd_extract, + } + try: + return handlers[args.cmd](args) + except BundleError as e: + print(f"error: {e}", file=sys.stderr) + return 1 diff --git a/tools/slua_bundle/slua_bundle/errors.py b/tools/slua_bundle/slua_bundle/errors.py new file mode 100644 index 00000000..c4ee25dd --- /dev/null +++ b/tools/slua_bundle/slua_bundle/errors.py @@ -0,0 +1,12 @@ +"""Root exception for everything raised by slua_bundle. + +Catch `BundleError` to handle any user-surfaceable error the bundler, +parser, resolver, or extractor can raise. Specific subclasses live next +to the code that raises them. +""" + +from __future__ import annotations + + +class BundleError(Exception): + pass diff --git a/tools/slua_bundle/slua_bundle/extractor.py b/tools/slua_bundle/slua_bundle/extractor.py new file mode 100644 index 00000000..82bc77be --- /dev/null +++ b/tools/slua_bundle/slua_bundle/extractor.py @@ -0,0 +1,135 @@ +"""Reverse a parsed bundle into a self-contained on-disk project tree. + +Layout convention: @root/... files land at /, files under other +aliases at //. A .luaurc declaring every non-root alias is +written when any are present. The extracted tree is hermetic: re-bundling +it with `--root ` reproduces the original bundle byte-for-byte. +""" + +from __future__ import annotations + +import json +from pathlib import PurePath + +from .errors import BundleError +from .fs import FSBackend +from .resolver import _split_canonical +from .runtime import ParsedBundle + + +class ExtractError(BundleError): + pass + + +class ExtractCollisionError(ExtractError): + """Two canonical keys map to the same physical path.""" + + +class ExtractClobberError(ExtractError): + """Output path already contains files that extract would overwrite.""" + + +class ExtractMissingMainError(ExtractError): + """Bundle has no MAIN directive; cannot place MAIN's body.""" + + +class ExtractUnsafeKeyError(ExtractError): + """Canonical key has a component that would escape the output dir or + otherwise produce an unsafe path.""" + + +_UNSAFE_COMPONENTS = frozenset({"", ".", ".."}) + + +def _validate_key_parts(canonical_key: str, alias: str, parts: list[str]) -> None: + """Reject keys whose components could traverse outside or + produce malformed paths. Our own bundler never emits these; this is + the trust boundary for bundles from unknown sources. + """ + for component in (alias, *parts): + if component in _UNSAFE_COMPONENTS: + raise ExtractUnsafeKeyError( + f"canonical key {canonical_key!r} contains unsafe component {component!r}" + ) + if any(c in component for c in ("/", "\\", "\x00")): + raise ExtractUnsafeKeyError( + f"canonical key {canonical_key!r} contains illegal char in component {component!r}" + ) + + +def physical_path_for_key( + canonical_key: str, + output: PurePath, +) -> PurePath: + """Map @alias/path -> on-disk path under . + + @root files go flat under ; other aliases go under a same-named + subdir. Bare alias keys (stripped init.luau) become init.luau in the + appropriate dir. + """ + alias, parts = _split_canonical(canonical_key) + _validate_key_parts(canonical_key, alias, parts) + base = output if alias == "root" else output / alias + if not parts: + return base / "init.luau" + return base.joinpath(*parts[:-1], f"{parts[-1]}.luau") + + +def extract_to_dir( + parsed: ParsedBundle, + out_fs: FSBackend, + output: PurePath, +) -> None: + main_key = parsed.fields.get("main") + if not main_key: + raise ExtractMissingMainError( + "bundle has no MAIN directive; cannot place MAIN's body. " + "Re-bundle with current code -- older bundles produced without " + "MAIN are not extractable." + ) + + sources: dict[str, str] = {main_key: parsed.main_source} + sources.update(parsed.modules) + + layout: dict[str, PurePath] = { + key: physical_path_for_key(key, output) + for key in sources + } + + seen: dict[PurePath, str] = {} + for key, path in layout.items(): + prior = seen.get(path) + if prior is not None: + raise ExtractCollisionError( + f"keys {prior!r} and {key!r} both map to {path}; " + "rename one alias or restructure the bundle" + ) + seen[path] = key + + luaurc_path = output / ".luaurc" + aliases_for_luaurc = sorted({ + _split_canonical(key)[0] + for key in sources + if _split_canonical(key)[0] != "root" + }) + + pre_existing: list[PurePath] = [ + path for path in layout.values() if out_fs.is_file(path) + ] + if aliases_for_luaurc and out_fs.is_file(luaurc_path): + pre_existing.append(luaurc_path) + if pre_existing: + listing = ", ".join(str(p) for p in pre_existing) + raise ExtractClobberError( + f"refusing to overwrite existing files: {listing}" + ) + + for key, path in layout.items(): + out_fs.write(path, sources[key]) + + if aliases_for_luaurc: + body = json.dumps( + {"aliases": {a: a for a in aliases_for_luaurc}}, + indent=2, + ) + "\n" + out_fs.write(luaurc_path, body) diff --git a/tools/slua_bundle/slua_bundle/fs.py b/tools/slua_bundle/slua_bundle/fs.py new file mode 100644 index 00000000..4efbcaf5 --- /dev/null +++ b/tools/slua_bundle/slua_bundle/fs.py @@ -0,0 +1,173 @@ +""" +Filesystem backends for the bundler. + +Two backends ship: MemoryFS (dict-backed, used by tests) and DiskFS +(real on-disk projects). Each declares a `Path` class attribute -- the +PurePath subclass it works with -- so application code can stay generic +over `PurePath` without mixing POSIX and Windows path types within a +single backend. + +Bundle keys are constructed from `PurePath.parts`, which returns plain +string tuples for any subclass, so the wire format is POSIX-shaped on +every host. +""" + +from __future__ import annotations + +import os +import pathlib +from abc import ABC, abstractmethod +from dataclasses import dataclass, field +from pathlib import PurePath, PurePosixPath +from typing import ClassVar, Iterator + + +def _is_anchor(s: str) -> bool: + return s in ("/", "\\") or s.endswith((":\\", ":/")) + + +def normalize(p: PurePath) -> PurePath: + """Collapse `.` and `..` segments, preserving the path subclass.""" + cls = type(p) + parts: list[str] = [] + for part in p.parts: + if part == "..": + if parts and parts[-1] != ".." and not _is_anchor(parts[-1]): + parts.pop() + else: + parts.append(part) + elif part == ".": + continue + else: + parts.append(part) + if not parts: + return cls(".") + return cls(*parts) + + +class FSBackend(ABC): + Path: ClassVar[type[PurePath]] + + @abstractmethod + def is_file(self, path: PurePath | str) -> bool: ... + + @abstractmethod + def read(self, path: PurePath | str) -> str: ... + + @abstractmethod + def is_dir(self, path: PurePath | str) -> bool: ... + + @abstractmethod + def iter_files(self) -> Iterator[PurePath]: ... + + @abstractmethod + def write(self, path: PurePath | str, content: str) -> None: + """Write `content` to `path`, creating parent dirs as needed.""" + + def to_path(self, p: PurePath | str) -> PurePath: + if isinstance(p, str): + p = self.Path(p) + return normalize(p) + + +@dataclass +class MemoryFS(FSBackend): + """ + In-memory dict-backed filesystem. Used by tests for deterministic, + cross-platform behavior. Keys are normalized at construction and at + lookup, mimicking how `open()` resolves `..`/`.` during traversal. + """ + + Path: ClassVar[type[PurePath]] = PurePosixPath + files: dict[PurePosixPath, str] = field(default_factory=dict) + + @classmethod + def from_dict(cls, d: dict[str, str]) -> "MemoryFS": + return cls({_as_posix(k): v for k, v in d.items()}) + + def is_file(self, path: PurePath | str) -> bool: + return _as_posix(path) in self.files + + def read(self, path: PurePath | str) -> str: + key = _as_posix(path) + if key not in self.files: + raise FileNotFoundError(f"not a file: {path}") + return self.files[key] + + def is_dir(self, path: PurePath | str) -> bool: + key = _as_posix(path) + return any(key in p.parents for p in self.files) + + def iter_files(self) -> Iterator[PurePosixPath]: + return iter(self.files.keys()) + + def write(self, path: PurePath | str, content: str) -> None: + self.files[_as_posix(path)] = content + + +def _as_posix(path: PurePath | str) -> PurePosixPath: + if isinstance(path, str): + path = PurePosixPath(path) + elif not isinstance(path, PurePosixPath): + path = PurePosixPath(*path.parts) + result = normalize(path) + assert isinstance(result, PurePosixPath) + return result + + +class DiskFS(FSBackend): + """ + Real on-disk filesystem rooted at a single directory. + + Path type is `pathlib.Path` (PosixPath on POSIX, WindowsPath on NT -- + both inherit from PurePath). Paths passed in are typically absolute + paths under the root; paths returned from `iter_files()` are + absolute. + + Symlinks are followed: a symlink in a developer's project is + intentional, pathlib detects cycles, and any file landing outside an + alias root surfaces cleanly via NoCoveringAliasError. + """ + + Path: ClassVar[type[PurePath]] = pathlib.Path + + def __init__(self, root: pathlib.Path) -> None: + self._root = root.resolve() + + @property + def root(self) -> pathlib.Path: + return self._root + + def _coerce(self, path: PurePath | str) -> pathlib.Path: + if isinstance(path, str): + return pathlib.Path(path) + if isinstance(path, pathlib.Path): + return path + return pathlib.Path(*path.parts) + + def is_file(self, path: PurePath | str) -> bool: + return self._coerce(path).is_file() + + def read(self, path: PurePath | str) -> str: + return self._coerce(path).read_text() + + def is_dir(self, path: PurePath | str) -> bool: + return self._coerce(path).is_dir() + + def iter_files(self) -> Iterator[pathlib.Path]: + # Skip hidden directories (.git/, node_modules-style hidden trees, + # etc.) and hidden files except .luaurc, which carries config we + # need. Dotfile filtering keeps real-world projects bundleable + # without spurious MarkerInjectionError hits or perf cliffs. + for cur, dirs, files in os.walk(self._root, followlinks=True): + dirs[:] = [d for d in dirs if not d.startswith(".")] + cur_path = pathlib.Path(cur) + for name in files: + if name.startswith(".") and name != ".luaurc": + continue + yield cur_path / name + + def write(self, path: PurePath | str, content: str) -> None: + target = self._coerce(path) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(content) diff --git a/tools/slua_bundle/slua_bundle/luaurc.py b/tools/slua_bundle/slua_bundle/luaurc.py new file mode 100644 index 00000000..17bac870 --- /dev/null +++ b/tools/slua_bundle/slua_bundle/luaurc.py @@ -0,0 +1,58 @@ +from __future__ import annotations + +import json +import re +from pathlib import PurePath + +from .errors import BundleError +from .fs import FSBackend, normalize + +CONFIG_NAME = ".luaurc" + + +class InvalidLuaurcError(BundleError): + """A .luaurc file exists but is not valid JSONC.""" + +# Match Luau's actual .luaurc parser (Config/src/Config.cpp): // line +# comments and trailing commas in objects/arrays. Block comments +# (`/* */`) are NOT accepted by Luau and stay in the stream so +# json.loads fails on them. +# +# Two passes so a trailing comma hidden behind a line comment isn't +# missed -- a single combined regex consumes the comment and never +# revisits the comma underneath. String literals are preserved by the +# alternation in each pattern. +_LINE_COMMENT_RE = re.compile(r'"(?:\\.|[^"\\])*"|//[^\n]*') +_TRAILING_COMMA_RE = re.compile(r'"(?:\\.|[^"\\])*"|,(?=\s*[}\]])') + + +def _strip_jsonc(s: str) -> str: + def keep_strings(m: re.Match[str]) -> str: + text = m.group(0) + return text if text.startswith('"') else "" + s = _LINE_COMMENT_RE.sub(keep_strings, s) + s = _TRAILING_COMMA_RE.sub(keep_strings, s) + return s + + +def load_config(vfs: FSBackend, config_dir: PurePath) -> dict[str, PurePath]: + config_path = config_dir / CONFIG_NAME + if not vfs.is_file(config_path): + return {} + try: + raw = json.loads(_strip_jsonc(vfs.read(config_path))) + except json.JSONDecodeError as e: + raise InvalidLuaurcError( + f"{config_path}: invalid JSONC ({e.msg} at line {e.lineno}, column {e.colno})" + ) from e + aliases = raw.get("aliases", {}) + out: dict[str, PurePath] = {} + for name, target in aliases.items(): + target_path = vfs.Path(target) + if target_path.is_absolute(): + out[name] = normalize(target_path) + else: + out[name] = normalize(config_dir / target_path) + return out + + diff --git a/tools/slua_bundle/slua_bundle/py.typed b/tools/slua_bundle/slua_bundle/py.typed new file mode 100644 index 00000000..e69de29b diff --git a/tools/slua_bundle/slua_bundle/resolver.py b/tools/slua_bundle/slua_bundle/resolver.py new file mode 100644 index 00000000..85745483 --- /dev/null +++ b/tools/slua_bundle/slua_bundle/resolver.py @@ -0,0 +1,124 @@ +from __future__ import annotations + +from .errors import BundleError + + +class BareIdentifierError(BundleError): + pass + + +class UnknownAliasError(BundleError): + pass + + +class RequireEscapesAliasError(BundleError): + pass + + +class RelativeRequireWithoutAnchorError(BundleError): + pass + + +class InvalidPathComponentError(BundleError): + pass + + +def _split_canonical(key: str) -> tuple[str, list[str]]: + assert key.startswith("@"), f"canonical key must start with @: {key}" + rest = key[1:] + if "/" in rest: + alias, tail = rest.split("/", 1) + return alias, tail.split("/") + return rest, [] + + +def _join_canonical(alias: str, parts: list[str]) -> str: + if not parts: + return f"@{alias}" + return f"@{alias}/" + "/".join(parts) + + +def _normalize(parts: list[str], context: str) -> list[str]: + out: list[str] = [] + for p in parts: + if p == "" or p == ".": + continue + if p == "..": + if not out: + raise RequireEscapesAliasError( + f"{context}: '..' traverses past alias root" + ) + out.pop() + else: + if p.startswith("."): + raise InvalidPathComponentError( + f"{context}: component {p!r} starts with '.' " + "(would resolve to a hidden file)" + ) + if "\x00" in p: + raise InvalidPathComponentError( + f"{context}: component contains NUL" + ) + out.append(p) + return out + + +def resolve(require_str: str, anchor_key: str | None, known_aliases: set[str]) -> str: + if require_str.startswith("@"): + rest = require_str[1:] + if "/" in rest: + alias, tail = rest.split("/", 1) + else: + alias, tail = rest, "" + + if alias == "self": + if anchor_key is None: + raise RelativeRequireWithoutAnchorError( + f"require '{require_str}' uses '@self' but the requiring module has no " + "anchor key (set 'main=' in the BUNDLE header for MAIN)" + ) + # Match Luau's RequireNavigator: @self resets to the requirer's + # *module path* and navigates from there. The module path is the + # file with its extension (and `init` suffix) stripped, which is + # exactly the canonical key. So @self/x from a requirer with key + # @p/lib/foo resolves to @p/lib/foo/x -- inside a subdirectory + # named after the file. That only points somewhere real for + # init.luau-style modules (whose canonical key already has no + # leaf); for leaf files @self is mostly useless. + anchor_alias, anchor_parts = _split_canonical(anchor_key) + tail_parts = tail.split("/") if tail else [] + new_parts = _normalize( + anchor_parts + tail_parts, + f"require '{require_str}' from {anchor_key}", + ) + return _join_canonical(anchor_alias, new_parts) + + if alias not in known_aliases: + raise UnknownAliasError( + f"unknown alias '@{alias}' in require '{require_str}'" + ) + tail_parts = tail.split("/") if tail else [] + new_parts = _normalize(tail_parts, f"require '{require_str}'") + return _join_canonical(alias, new_parts) + + if require_str.startswith("./") or require_str.startswith("../") or require_str in ( + ".", + "..", + ): + if anchor_key is None: + raise RelativeRequireWithoutAnchorError( + f"relative require '{require_str}' has no anchor " + "(set 'main=' in the BUNDLE header for MAIN)" + ) + anchor_alias, anchor_parts = _split_canonical(anchor_key) + anchor_dir_parts = anchor_parts[:-1] if anchor_parts else [] + rel_parts = require_str.split("/") + new_parts = _normalize( + anchor_dir_parts + rel_parts, + f"require '{require_str}' from {anchor_key}", + ) + return _join_canonical(anchor_alias, new_parts) + + raise BareIdentifierError( + f"require path must start with './', '../', or '@' (got '{require_str}')" + ) diff --git a/tools/slua_bundle/slua_bundle/runtime.py b/tools/slua_bundle/slua_bundle/runtime.py new file mode 100644 index 00000000..1560a4ea --- /dev/null +++ b/tools/slua_bundle/slua_bundle/runtime.py @@ -0,0 +1,222 @@ +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Iterator + +from .errors import BundleError +from .resolver import resolve + +SUPPORTED_VERSIONS = frozenset({1}) + +MARKER_RE = re.compile(r"^--\s*!!LUABUNDLE:(\S+)(?:\s+(.+?))?\s*$") +REQUIRE_RE = re.compile(r'\brequire\s*\(\s*"([^"]*)"\s*\)') + + +class BundleParseError(BundleError): + pass + + +class CircularDependencyError(BundleError): + pass + + +@dataclass +class ParsedBundle: + fields: dict[str, str] + main_source: str + modules: dict[str, str] + + +@dataclass(frozen=True) +class _Line: + lineno: int + raw: str + marker_kind: str | None + marker_arg: str + + @property + def is_marker(self) -> bool: + return self.marker_kind is not None + + +def _scan(text: str) -> Iterator[_Line]: + for lineno, raw in enumerate(text.splitlines(keepends=True), start=1): + stripped = raw.rstrip("\n").rstrip("\r") + m = MARKER_RE.match(stripped) + kind = m.group(1) if m else None + arg = (m.group(2) or "").strip() if m else "" + yield _Line(lineno=lineno, raw=raw, marker_kind=kind, marker_arg=arg) + + +class BundleParser: + def __init__(self, text: str) -> None: + self._lines = _scan(text) + self.fields: dict[str, str] = {} + self.main_source: str = "" + self.modules: dict[str, str] = {} + + def parse(self) -> ParsedBundle: + self._consume_version() + self._consume_header() + if "main" not in self.fields: + raise BundleParseError("missing MAIN directive") + self.main_source, trailing = self._consume_body() + self._consume_modules(trailing) + return ParsedBundle( + fields=self.fields, + main_source=self.main_source, + modules=self.modules, + ) + + def _consume_version(self) -> None: + try: + line = next(self._lines) + except StopIteration: + raise BundleParseError("empty bundle (missing VERSION)") from None + if not line.is_marker or line.marker_kind != "VERSION": + what = f"marker {line.marker_kind}" if line.is_marker else "body content" + raise BundleParseError( + f"line {line.lineno}: expected VERSION marker first, got {what}" + ) + try: + version = int(line.marker_arg) + except ValueError as e: + raise BundleParseError( + f"line {line.lineno}: VERSION arg must be an integer, got {line.marker_arg!r}" + ) from e + if version not in SUPPORTED_VERSIONS: + raise BundleParseError( + f"line {line.lineno}: unsupported VERSION {version} " + f"(supported: {sorted(SUPPORTED_VERSIONS)})" + ) + self.fields["version"] = str(version) + + def _consume_header(self) -> None: + for line in self._lines: + if not line.is_marker: + raise BundleParseError( + f"line {line.lineno}: body content before BODY marker" + ) + kind = line.marker_kind + if kind == "PROJECT": + self._set_unique(line, "project") + elif kind == "MAIN": + self._set_unique(line, "main") + elif kind == "BODY": + if line.marker_arg: + raise BundleParseError(f"line {line.lineno}: BODY takes no argument") + return + else: + raise BundleParseError( + f"line {line.lineno}: unknown header directive {kind} " + "(at this VERSION, only PROJECT, MAIN, BODY are valid)" + ) + raise BundleParseError("missing BODY marker") + + def _set_unique(self, line: _Line, field_name: str) -> None: + if not line.marker_arg: + raise BundleParseError( + f"line {line.lineno}: {line.marker_kind} requires an argument" + ) + if field_name in self.fields: + raise BundleParseError( + f"line {line.lineno}: duplicate {line.marker_kind} directive" + ) + self.fields[field_name] = line.marker_arg + + def _consume_body(self) -> tuple[str, _Line | None]: + """Consume non-marker lines until the next marker or EOF. + + Returns (body source, the marker that ended the body or None for EOF). + Body content is preserved verbatim -- no stripping -- so round-trip + through the bundler is stable. + """ + body: list[str] = [] + for line in self._lines: + if line.is_marker: + return "".join(body), line + body.append(line.raw) + return "".join(body), None + + def _consume_modules(self, first_marker: _Line | None) -> None: + current = first_marker + while current is not None: + if current.marker_kind != "MODULE": + raise BundleParseError( + f"line {current.lineno}: unexpected marker {current.marker_kind} after BODY " + "(only MODULE markers are valid)" + ) + key = current.marker_arg + if not key: + raise BundleParseError( + f"line {current.lineno}: MODULE requires a canonical key" + ) + if key in self.modules: + raise BundleParseError( + f"line {current.lineno}: duplicate MODULE marker for {key}" + ) + body, next_marker = self._consume_body() + self.modules[key] = body + current = next_marker + + +def parse_bundle(text: str) -> ParsedBundle: + return BundleParser(text).parse() + + +def simulate(text: str) -> dict[str, int]: + """ + Simulate a Luau VM running the bundle with `require()`-cache semantics. + + Starts at MAIN, recursively walks `require()` calls, and counts how + many times each module body would execute under correct caching -- + once on first require, zero on cache hits. The return value maps each + canonical key to its body-execution count; a correctly-bundled, + correctly-traversed program has every reachable key at exactly 1. + + Counts > 1 indicate a require-cache bug; missing keys indicate + unreachable (over-bundled) modules. Cycles raise CircularDependencyError. + + No Lua code is actually evaluated -- this is a static walk over the + require graph, comparing dedup behavior against the contract. + """ + parsed = parse_bundle(text) + main_anchor = parsed.fields["main"] # required by the parser + + all_modules = dict(parsed.modules) + if main_anchor in all_modules: + raise BundleParseError( + f"BUNDLE main={main_anchor} collides with a module key in the same bundle" + ) + all_modules[main_anchor] = parsed.main_source + + known_aliases: set[str] = {"root"} + for key in all_modules: + if not key.startswith("@"): + raise BundleParseError(f"module key does not start with @: {key}") + alias = key[1:].split("/", 1)[0] + known_aliases.add(alias) + + body_runs: dict[str, int] = {} + cached: set[str] = set() + in_progress: list[str] = [] + + def run(key: str) -> None: + if key in in_progress: + chain = " -> ".join([*in_progress, key]) + raise CircularDependencyError(f"circular dependency: {chain}") + if key in cached: + return + if key not in all_modules: + raise BundleParseError(f"required module {key} not in bundle") + in_progress.append(key) + for req in REQUIRE_RE.findall(all_modules[key]): + target = resolve(req, key, known_aliases) + run(target) + in_progress.pop() + body_runs[key] = body_runs.get(key, 0) + 1 + cached.add(key) + + run(main_anchor) + return body_runs diff --git a/tools/slua_bundle/tests/__init__.py b/tools/slua_bundle/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tools/slua_bundle/tests/_helpers.py b/tools/slua_bundle/tests/_helpers.py new file mode 100644 index 00000000..0046caba --- /dev/null +++ b/tools/slua_bundle/tests/_helpers.py @@ -0,0 +1,10 @@ +"""Shared test helpers.""" + +from __future__ import annotations + +import json + + +def luaurc(aliases: dict[str, str]) -> str: + """Render a minimal .luaurc body. Use raw strings for JSONC-shape tests.""" + return json.dumps({"aliases": aliases}) diff --git a/tools/slua_bundle/tests/test_bundler.py b/tools/slua_bundle/tests/test_bundler.py new file mode 100644 index 00000000..68b49107 --- /dev/null +++ b/tools/slua_bundle/tests/test_bundler.py @@ -0,0 +1,447 @@ +"""Bundle production from an FSBackend.""" + +import warnings +from pathlib import PurePosixPath + +import pytest + +from slua_bundle import ( + AliasCollisionWarning, + AmbiguousResolutionError, + DepthExceededError, + MarkerInjectionError, + MemoryFS, + ModuleCountExceededError, + NoResolverError, + ReservedAliasError, + bundle, + parse_bundle, +) +from slua_bundle.bundler import MAX_DEPTH, MAX_MODULES + +from ._helpers import luaurc + + +def test_bundle_emits_main_anchor_and_modules(): + vfs = MemoryFS.from_dict({ + "/project/src/HudController.luau": 'require("./lib/foo")', + "/project/src/lib/foo.luau": 'return "foo"', + }) + text = bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/HudController.luau"), + "myhud", + ) + assert text == """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:PROJECT myhud +-- !!LUABUNDLE:MAIN @root/HudController +-- !!LUABUNDLE:BODY +require("./lib/foo") +-- !!LUABUNDLE:MODULE @root/lib/foo +return "foo" +""" + + +def test_bundle_alias_pointing_outside_project_root_is_fine(): + """An explicit alias to a directory outside project_root is honored. + + The user knows what they're declaring; the bundler maps the file under + the alias and ships it. This is what enables vendored deps that don't + live under project_root. + """ + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"shared": "/elsewhere"}), + "/project/src/Main.luau": 'require("@shared/util")', + "/elsewhere/util.luau": "return {}", + }) + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + assert text == """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:PROJECT myhud +-- !!LUABUNDLE:MAIN @root/src/Main +-- !!LUABUNDLE:BODY +require("@shared/util") +-- !!LUABUNDLE:MODULE @shared/util +return {} +""" + + +def test_bundle_includes_wally_package(): + # Bundler reads only the project-root .luaurc. Wally writes its alias + # declarations there, which is sufficient for MAIN to resolve @SomeLib. + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"SomeLib": "Packages/SomeLib"}), + "/project/src/Main.luau": 'require("@SomeLib/util")', + "/project/Packages/SomeLib/util.luau": "return {}", + }) + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + assert text == """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:PROJECT myhud +-- !!LUABUNDLE:MAIN @root/src/Main +-- !!LUABUNDLE:BODY +require("@SomeLib/util") +-- !!LUABUNDLE:MODULE @SomeLib/util +return {} +""" + + +def test_bundle_rejects_marker_injection_in_source(): + """Source files must not contain '-- !!LUABUNDLE:' markers; bundler must reject.""" + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": 'require("./lib/evil")', + "/project/src/lib/evil.luau": "-- !!LUABUNDLE:MODULE!! @evil/injected\nreturn {}", + }) + with pytest.raises(MarkerInjectionError): + bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + + +def test_bundle_rejects_marker_injection_in_main(): + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": "-- !!LUABUNDLE:BUNDLE!! evil\nreturn {}", + }) + with pytest.raises(MarkerInjectionError): + bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + + +def test_bundle_rejects_ambiguous_resolution(): + """When both .luau and /init.luau exist, the require is ambiguous.""" + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": 'require("./foo")', + "/project/src/foo.luau": 'return "leaf"', + "/project/src/foo/init.luau": 'return "dir"', + }) + with pytest.raises(AmbiguousResolutionError): + bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + + +def test_bundle_only_includes_files_reachable_from_main(): + """Files not transitively required by MAIN are not bundled.""" + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": 'require("./foo")', + "/project/src/foo.luau": "return 1", + "/project/src/orphan.luau": 'return "should not appear"', + "/project/src/lib/sub.luau": 'return "also unused"', + }) + text = bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + assert text == """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:PROJECT myhud +-- !!LUABUNDLE:MAIN @root/Main +-- !!LUABUNDLE:BODY +require("./foo") +-- !!LUABUNDLE:MODULE @root/foo +return 1 +""" + + +def test_bundle_traces_single_quoted_requires(): + """require('./foo') is valid Lua and must be picked up like the double-quoted form.""" + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": "require('./foo')", + "/project/src/foo.luau": "return 1", + }) + text = bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + assert text == """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:PROJECT myhud +-- !!LUABUNDLE:MAIN @root/Main +-- !!LUABUNDLE:BODY +require('./foo') +-- !!LUABUNDLE:MODULE @root/foo +return 1 +""" + + +def test_bundle_rejects_require_to_missing_file(): + """A require pointing at a nonexistent file with no fallback raises NoResolverError.""" + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": 'require("./missing")', + }) + with pytest.raises(NoResolverError, match="no resolver produced source"): + bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + + +def test_bundle_round_trip_is_stable(): + """bundle -> parse -> compare; module bodies and main come back verbatim.""" + main_src = 'require("./lib/foo")\nreturn 1\n' + foo_src = 'return "foo"\n' + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": main_src, + "/project/src/lib/foo.luau": foo_src, + }) + text = bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + parsed = parse_bundle(text) + assert parsed.main_source == main_src + assert parsed.modules == {"@root/lib/foo": foo_src} + + +def test_bundle_round_trip_appends_missing_trailing_newline(): + """A source without a trailing newline gets exactly one appended -- once -- for marker alignment.""" + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": "return 1", # no trailing \n + }) + text = bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + parsed = parse_bundle(text) + # Body now has the forced \n. Re-parse is then idempotent. + assert parsed.main_source == "return 1\n" + + +def test_bundle_without_project_name_omits_project_directive(): + """PROJECT is optional viewer-linkage metadata; bundling without it emits no PROJECT line.""" + vfs = MemoryFS.from_dict({ + "/project/src/Main.luau": 'require("./foo")', + "/project/src/foo.luau": "return 1", + }) + text = bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/Main.luau"), + ) + assert text == """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:MAIN @root/Main +-- !!LUABUNDLE:BODY +require("./foo") +-- !!LUABUNDLE:MODULE @root/foo +return 1 +""" + parsed = parse_bundle(text) + assert "project" not in parsed.fields + assert parsed.modules == {"@root/foo": "return 1\n"} + + +def test_bundle_rejects_reserved_root_alias_in_luaurc(): + """@root is built-in; .luaurc cannot redefine it.""" + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"root": "/elsewhere"}), + "/project/src/Main.luau": "return 1", + }) + with pytest.raises(ReservedAliasError): + bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + + +def test_bundle_user_alias_at_project_root_loses_to_root_silently(): + """User-declared alias targeting project_root coexists with @root; @root wins canonicalization with no warning.""" + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"myproj": "."}), + "/project/src/Main.luau": "return 1", + }) + with warnings.catch_warnings(): + warnings.simplefilter("error", AliasCollisionWarning) + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + assert "-- !!LUABUNDLE:MAIN @root/src/Main" in text + assert "@myproj" not in text + + +def test_bundle_two_user_aliases_at_identical_target_warns(): + """Two non-root user aliases pointing at the exact same directory: ASCII-first wins, warning emitted.""" + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"Zeta": "/elsewhere", "Alpha": "/elsewhere"}), + "/project/src/Main.luau": 'require("@Alpha/util")', + "/elsewhere/util.luau": "return {}", + }) + with pytest.warns(AliasCollisionWarning, match="Alpha"): + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + assert "-- !!LUABUNDLE:MODULE @Alpha/util" in text + assert "-- !!LUABUNDLE:MODULE @Zeta/util" not in text + + +# ---- Last-resort resolver (existing_bundle fallback) ------------------------ + + +_PRIOR_BUNDLE = """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:MAIN @root/Main +-- !!LUABUNDLE:BODY +require("./lib/foo") +-- !!LUABUNDLE:MODULE @root/lib/foo +return "from prior bundle" +""" + + +def test_existing_bundle_used_when_disk_source_missing(): + """Disk has MAIN but not its dependency; embedded copy fills the gap.""" + vfs = MemoryFS.from_dict({ + "/project/Main.luau": 'require("./lib/foo")', + }) + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/Main.luau"), + existing_bundle=_PRIOR_BUNDLE, + ) + assert "-- !!LUABUNDLE:MODULE @root/lib/foo" in text + assert 'return "from prior bundle"' in text + + +def test_existing_bundle_used_when_alias_missing_from_luaurc(): + """Require references an alias the new env has no .luaurc entry for.""" + prior = """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:MAIN @root/Main +-- !!LUABUNDLE:BODY +require("@SomeLib/util") +-- !!LUABUNDLE:MODULE @SomeLib/util +return "vendored" +""" + vfs = MemoryFS.from_dict({ + "/project/Main.luau": 'require("@SomeLib/util")', + }) + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/Main.luau"), + existing_bundle=prior, + ) + assert "-- !!LUABUNDLE:MODULE @SomeLib/util" in text + assert 'return "vendored"' in text + + +def test_disk_preferred_over_existing_bundle_when_both_present(): + """Disk version wins when both resolvers can produce source.""" + vfs = MemoryFS.from_dict({ + "/project/Main.luau": 'require("./lib/foo")', + "/project/lib/foo.luau": 'return "from disk"', + }) + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/Main.luau"), + existing_bundle=_PRIOR_BUNDLE, + ) + assert 'return "from disk"' in text + assert 'return "from prior bundle"' not in text + + +def test_no_resolver_succeeded_raises(): + """Neither disk nor existing bundle has the required module.""" + vfs = MemoryFS.from_dict({ + "/project/Main.luau": 'require("./lib/missing")', + }) + with pytest.raises(NoResolverError, match="no resolver produced source"): + bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/Main.luau"), + existing_bundle=_PRIOR_BUNDLE, + ) + + +# ---- Depth + module-count limits -------------------------------------------- + + +def _chain_vfs(n: int) -> MemoryFS: + """MAIN -> m1 -> m2 -> ... -> m{n-1}, requires written as ./mK.""" + files: dict[str, str] = { + "/p/Main.luau": 'require("./m1")' if n > 0 else "return 1", + } + for i in range(1, n): + body = f'require("./m{i+1}")' if i < n - 1 else "return 1" + files[f"/p/m{i}.luau"] = body + return MemoryFS.from_dict(files) + + +def test_depth_at_limit_allowed(): + """A chain at exactly MAX_DEPTH depth bundles cleanly.""" + # MAIN at depth 0, deepest module at depth MAX_DEPTH. + vfs = _chain_vfs(MAX_DEPTH + 1) + text = bundle( + vfs, + PurePosixPath("/p"), + PurePosixPath("/p/Main.luau"), + ) + assert f"-- !!LUABUNDLE:MODULE @root/m{MAX_DEPTH}" in text + + +def test_depth_exceeded_raises(): + """A chain one step deeper than MAX_DEPTH trips the limit.""" + vfs = _chain_vfs(MAX_DEPTH + 2) + with pytest.raises(DepthExceededError, match=f"exceeds maximum {MAX_DEPTH}"): + bundle( + vfs, + PurePosixPath("/p"), + PurePosixPath("/p/Main.luau"), + ) + + +def test_module_count_exceeded_raises(): + """MAIN with > MAX_MODULES distinct dependencies trips the count limit.""" + requires = "\n".join(f'require("./m{i}")' for i in range(MAX_MODULES + 1)) + files: dict[str, str] = {"/p/Main.luau": requires} + for i in range(MAX_MODULES + 1): + files[f"/p/m{i}.luau"] = "return 1" + vfs = MemoryFS.from_dict(files) + with pytest.raises(ModuleCountExceededError, match=f"maximum of {MAX_MODULES}"): + bundle( + vfs, + PurePosixPath("/p"), + PurePosixPath("/p/Main.luau"), + ) diff --git a/tools/slua_bundle/tests/test_canonicalize.py b/tools/slua_bundle/tests/test_canonicalize.py new file mode 100644 index 00000000..804d4777 --- /dev/null +++ b/tools/slua_bundle/tests/test_canonicalize.py @@ -0,0 +1,191 @@ +"""Canonicalize physical paths to alias-prefixed canonical keys.""" + +import warnings +from pathlib import PurePosixPath + +import pytest + +from slua_bundle import ( + AliasCollisionWarning, + InvalidLuaurcError, + MemoryFS, + NoCoveringAliasError, + ReservedAliasError, + canonicalize, +) + +from ._helpers import luaurc + + +def test_implicit_root_alias_no_luaurc(): + """No .luaurc -- @root implicitly covers project_root.""" + vfs = MemoryFS.from_dict({ + "/project/src/HudController.luau": "return {}", + "/project/src/lib/foo.luau": "return {}", + }) + project_root = PurePosixPath("/project/src") + assert canonicalize(vfs, PurePosixPath("/project/src/HudController.luau"), project_root) == "@root/HudController" + assert canonicalize(vfs, PurePosixPath("/project/src/lib/foo.luau"), project_root) == "@root/lib/foo" + + +def test_init_luau_strips_to_directory(): + vfs = MemoryFS.from_dict({ + "/project/src/init.luau": "", + "/project/src/lib/init.luau": "", + "/project/src/lib/foo.luau": "", + }) + project_root = PurePosixPath("/project/src") + assert canonicalize(vfs, PurePosixPath("/project/src/init.luau"), project_root) == "@root" + assert canonicalize(vfs, PurePosixPath("/project/src/lib/init.luau"), project_root) == "@root/lib" + + +def test_wally_alias_at_project_root_beats_root(): + """Top-level .luaurc declares @SomeLib; files inside it canonicalize under @SomeLib. + + Most-specific covering alias wins (more path parts), so files under + /project/Packages/SomeLib/ get @SomeLib/... rather than @root/Packages/SomeLib/... + """ + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"SomeLib": "Packages/SomeLib"}), + "/project/Packages/SomeLib/init.luau": "", + "/project/Packages/SomeLib/util.luau": "", + "/project/src/HudController.luau": "", + }) + project_root = PurePosixPath("/project") + assert canonicalize(vfs, PurePosixPath("/project/Packages/SomeLib/util.luau"), project_root) == "@SomeLib/util" + assert canonicalize(vfs, PurePosixPath("/project/Packages/SomeLib/init.luau"), project_root) == "@SomeLib" + assert canonicalize(vfs, PurePosixPath("/project/src/HudController.luau"), project_root) == "@root/src/HudController" + + +def test_nested_luaurc_is_ignored(): + """Aliases declared in a nested .luaurc are not honored. + + Bundle reproducibility requires the alias set to be fully determined by + project_root's .luaurc; a child .luaurc that isn't mirrored at the project + root has no effect, and a file it would have named is canonicalized under + @root instead. + """ + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({}), + "/project/Packages/SomeLib/.luaurc": luaurc({"SomeLib": "."}), + "/project/Packages/SomeLib/util.luau": "", + }) + project_root = PurePosixPath("/project") + assert canonicalize(vfs, PurePosixPath("/project/Packages/SomeLib/util.luau"), project_root) == "@root/Packages/SomeLib/util" + + +def test_no_covering_alias_for_outside_path(): + """A file outside the project root with no covering .luaurc raises.""" + vfs = MemoryFS.from_dict({ + "/project/src/HudController.luau": "", + "/elsewhere/orphan.luau": "", + }) + project_root = PurePosixPath("/project/src") + with pytest.raises(NoCoveringAliasError): + canonicalize(vfs, PurePosixPath("/elsewhere/orphan.luau"), project_root) + + +def test_external_alias_with_explicit_target(): + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"shared": "shared-modules"}), + "/project/shared-modules/util.luau": "", + "/project/src/main.luau": "", + }) + project_root = PurePosixPath("/project") + assert canonicalize(vfs, PurePosixPath("/project/shared-modules/util.luau"), project_root) == "@shared/util" + + +def test_jsonc_line_comments_and_trailing_commas_supported(): + """Match Luau's actual .luaurc parser: // line comments and trailing commas.""" + config = """ + { + // Wally-style comment + "aliases": { + "shared": "shared-modules", // trailing comment after value + }, + } + """ + vfs = MemoryFS.from_dict({ + "/project/.luaurc": config, + "/project/shared-modules/util.luau": "", + }) + project_root = PurePosixPath("/project") + assert canonicalize(vfs, PurePosixPath("/project/shared-modules/util.luau"), project_root) == "@shared/util" + + +def test_jsonc_block_comments_rejected(): + """Luau's parser does not accept /* */; the prototype mirrors that.""" + config = '{"aliases": {/* not allowed */ "shared": "shared-modules"}}' + vfs = MemoryFS.from_dict({ + "/project/.luaurc": config, + "/project/shared-modules/util.luau": "", + }) + project_root = PurePosixPath("/project") + with pytest.raises(InvalidLuaurcError, match="invalid JSONC"): + canonicalize(vfs, PurePosixPath("/project/shared-modules/util.luau"), project_root) + + +def test_reserved_root_alias_in_luaurc_rejected(): + """.luaurc may not declare an alias named `root`; it's the built-in project-root alias.""" + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"root": "shared-modules"}), + "/project/Main.luau": "", + }) + project_root = PurePosixPath("/project") + with pytest.raises(ReservedAliasError): + canonicalize(vfs, PurePosixPath("/project/Main.luau"), project_root) + + +def test_reserved_self_alias_in_luaurc_rejected(): + """.luaurc may not declare `self`; it's a Luau-ecosystem reserved name (current module's dir).""" + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"self": "shared-modules"}), + "/project/Main.luau": "", + }) + project_root = PurePosixPath("/project") + with pytest.raises(ReservedAliasError): + canonicalize(vfs, PurePosixPath("/project/Main.luau"), project_root) + + +def test_user_alias_at_project_root_loses_to_root_silently(): + """User declares an alias targeting project_root; @root wins canonicalization, no warning.""" + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"myhud": "."}), + "/project/Main.luau": "", + }) + project_root = PurePosixPath("/project") + with warnings.catch_warnings(): + warnings.simplefilter("error", AliasCollisionWarning) + assert canonicalize(vfs, PurePosixPath("/project/Main.luau"), project_root) == "@root/Main" + + +def test_two_user_aliases_at_identical_target_warns_and_picks_ascii_first(): + """Two non-root aliases at the same target dir: ASCII-first wins, warning emitted.""" + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"Zeta": "/elsewhere", "Alpha": "/elsewhere"}), + "/elsewhere/util.luau": "", + }) + project_root = PurePosixPath("/project") + with pytest.warns(AliasCollisionWarning, match="Alpha"): + assert canonicalize(vfs, PurePosixPath("/elsewhere/util.luau"), project_root) == "@Alpha/util" + + +def test_unnormalized_project_root_normalizes(): + """A caller-supplied project_root with `.` segments must normalize before lookups.""" + vfs = MemoryFS.from_dict({ + "/project/src/.luaurc": luaurc({}), + "/project/src/Main.luau": "", + }) + project_root = PurePosixPath("/project/./src") + assert canonicalize(vfs, PurePosixPath("/project/src/Main.luau"), project_root) == "@root/Main" + + +def test_jsonc_does_not_strip_inside_strings(): + """Comment-like sequences inside string literals must be preserved.""" + config = '{"aliases": {"path-with-slashes": "weird//name"}}' + vfs = MemoryFS.from_dict({ + "/project/.luaurc": config, + "/project/weird/name/foo.luau": "", + }) + project_root = PurePosixPath("/project") + assert canonicalize(vfs, PurePosixPath("/project/weird/name/foo.luau"), project_root) == "@path-with-slashes/foo" diff --git a/tools/slua_bundle/tests/test_disk_fs.py b/tools/slua_bundle/tests/test_disk_fs.py new file mode 100644 index 00000000..fde6dfe7 --- /dev/null +++ b/tools/slua_bundle/tests/test_disk_fs.py @@ -0,0 +1,56 @@ +"""DiskFS: real-FS backend smoke tests against pytest's tmp_path.""" + +import pathlib + +from slua_bundle import DiskFS, bundle + + +def _write(root: pathlib.Path, rel: str, content: str) -> pathlib.Path: + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + return p + + +def test_bundle_smoke(tmp_path: pathlib.Path): + """End-to-end against real disk: bundle a small project. Full-text + equality also enforces the cross-platform contract that bundle keys are + POSIX-shaped on every host (the expected text uses forward slashes).""" + _write(tmp_path, "src/Main.luau", 'require("./lib/foo")') + _write(tmp_path, "src/lib/foo.luau", 'require("./bar")') + _write(tmp_path, "src/lib/bar.luau", "return 1") + + fs = DiskFS(tmp_path) + project_root = (tmp_path / "src").resolve() + text = bundle(fs, project_root, project_root / "Main.luau", "myhud") + + assert text == """\ +-- !!LUABUNDLE:VERSION 1 +-- !!LUABUNDLE:PROJECT myhud +-- !!LUABUNDLE:MAIN @root/Main +-- !!LUABUNDLE:BODY +require("./lib/foo") +-- !!LUABUNDLE:MODULE @root/lib/foo +require("./bar") +-- !!LUABUNDLE:MODULE @root/lib/bar +return 1 +""" + + +def test_iter_files_skips_dotfiles_except_luaurc(tmp_path: pathlib.Path): + """Hidden files (other than .luaurc) and hidden directories are pruned.""" + _write(tmp_path, "src/Main.luau", "return 1") + _write(tmp_path, ".luaurc", "{}") # kept + _write(tmp_path, ".env", "SECRET=1") # skipped + _write(tmp_path, ".cache/index", "stale") # whole dir pruned + _write(tmp_path, ".cache/blobs/blob", "binary-junk") + _write(tmp_path, "node_modules/.hidden/foo.luau", "x") # outer dir kept, inner pruned + _write(tmp_path, "node_modules/visible.luau", "y") # kept (not hidden) + + fs = DiskFS(tmp_path) + rels = sorted(str(p.relative_to(tmp_path)) for p in fs.iter_files()) + assert rels == [ + ".luaurc", + "node_modules/visible.luau", + "src/Main.luau", + ] diff --git a/tools/slua_bundle/tests/test_e2e.py b/tools/slua_bundle/tests/test_e2e.py new file mode 100644 index 00000000..f9ba957e --- /dev/null +++ b/tools/slua_bundle/tests/test_e2e.py @@ -0,0 +1,103 @@ +"""End-to-end: FSBackend -> bundle -> simulate.""" + +from pathlib import PurePosixPath + +from slua_bundle import MemoryFS, bundle, parse_bundle, simulate +from slua_bundle.extractor import extract_to_dir + +from ._helpers import luaurc + + +def test_happy_path(): + """MAIN -> ./lib/foo -> ./bar -> return.""" + vfs = MemoryFS.from_dict({ + "/project/src/HudController.luau": 'require("./lib/foo")', + "/project/src/lib/foo.luau": 'require("./bar")', + "/project/src/lib/bar.luau": "return 42", + }) + text = bundle( + vfs, + PurePosixPath("/project/src"), + PurePosixPath("/project/src/HudController.luau"), + "myhud", + ) + body_runs = simulate(text) + assert body_runs == { + "@root/HudController": 1, + "@root/lib/foo": 1, + "@root/lib/bar": 1, + } + + +def test_bare_project_no_luaurc(): + """@root covers project_root automatically; no .luaurc needed.""" + vfs = MemoryFS.from_dict({ + "/work/src/Main.luau": 'require("./lib/foo")', + "/work/src/lib/foo.luau": "return {}", + }) + text = bundle( + vfs, + PurePosixPath("/work/src"), + PurePosixPath("/work/src/Main.luau"), + "myhud", + ) + assert "-- !!LUABUNDLE:MAIN @root/Main" in text + body_runs = simulate(text) + assert body_runs["@root/Main"] == 1 + assert body_runs["@root/lib/foo"] == 1 + + +def test_wally_style_dedup(): + """SomeLib's util reached two ways from its init.luau (@self/util and @SomeLib/util) dedups. + + The project-root .luaurc declares @SomeLib; that's the only config the + bundler reads. Inside SomeLib/init.luau, both @self/util (resolves under + init.luau's anchor module path) and @SomeLib/util reach the same file. + """ + vfs = MemoryFS.from_dict({ + "/project/.luaurc": luaurc({"SomeLib": "Packages/SomeLib"}), + "/project/src/Main.luau": 'require("@SomeLib")', + "/project/Packages/SomeLib/init.luau": 'require("@self/util"); require("@SomeLib/util")', + "/project/Packages/SomeLib/util.luau": "return {}", + }) + text = bundle( + vfs, + PurePosixPath("/project"), + PurePosixPath("/project/src/Main.luau"), + "myhud", + ) + body_runs = simulate(text) + assert body_runs["@SomeLib/util"] == 1 + assert body_runs["@SomeLib"] == 1 + assert body_runs["@root/src/Main"] == 1 + + +def test_rebundle_via_extractor_uses_last_resort_resolver(): + """Extract a bundle, drop a source file, rebundle with the original as + fallback. Demonstrates the RFC's rebundle-without-source flow: a CLI + receives a bundle, can't reach the original resolvers, but can still + rebuild because the prior bundle is the universal last-resort resolver. + """ + src_vfs = MemoryFS.from_dict({ + "/project/Main.luau": 'require("./lib/foo")', + "/project/lib/foo.luau": 'return "hi"', + }) + initial = bundle( + src_vfs, + PurePosixPath("/project"), + PurePosixPath("/project/Main.luau"), + "myhud", + ) + + out_fs = MemoryFS() + extract_to_dir(parse_bundle(initial), out_fs, PurePosixPath("/out")) + del out_fs.files[PurePosixPath("/out/lib/foo.luau")] + + rebuilt = bundle( + out_fs, + PurePosixPath("/out"), + PurePosixPath("/out/Main.luau"), + "myhud", + existing_bundle=initial, + ) + assert rebuilt == initial diff --git a/tools/slua_bundle/tests/test_extractor.py b/tools/slua_bundle/tests/test_extractor.py new file mode 100644 index 00000000..644c599e --- /dev/null +++ b/tools/slua_bundle/tests/test_extractor.py @@ -0,0 +1,201 @@ +"""Reverse-engineer a bundle into a self-contained project tree.""" + +import json +from pathlib import PurePosixPath + +import pytest + +from slua_bundle import ( + MemoryFS, + bundle, + parse_bundle, +) +from slua_bundle.extractor import ( + ExtractClobberError, + ExtractCollisionError, + ExtractMissingMainError, + ExtractUnsafeKeyError, + extract_to_dir, + physical_path_for_key, +) +from slua_bundle.runtime import ParsedBundle + +OUT = PurePosixPath("/out") + + +@pytest.fixture +def fs() -> MemoryFS: + return MemoryFS() + + +def test_physical_path_root_alias_with_path(): + assert physical_path_for_key("@root/Main", OUT) == PurePosixPath("/out/Main.luau") + assert physical_path_for_key("@root/lib/foo", OUT) == PurePosixPath("/out/lib/foo.luau") + + +def test_physical_path_root_alias_bare(): + assert physical_path_for_key("@root", OUT) == PurePosixPath("/out/init.luau") + + +def test_physical_path_other_alias_with_path(): + assert physical_path_for_key("@SomeLib/util", OUT) == PurePosixPath("/out/SomeLib/util.luau") + + +def test_physical_path_other_alias_bare(): + assert physical_path_for_key("@SomeLib", OUT) == PurePosixPath("/out/SomeLib/init.luau") + + +def test_extract_writes_luaurc_for_external_aliases(fs: MemoryFS): + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/Main"}, + main_source='require("@SomeLib/util")\n', + modules={"@SomeLib/util": "return {}\n"}, + ) + extract_to_dir(parsed, fs, OUT) + assert set(fs.iter_files()) == { + PurePosixPath("/out/Main.luau"), + PurePosixPath("/out/SomeLib/util.luau"), + PurePosixPath("/out/.luaurc"), + } + luaurc = json.loads(fs.read(OUT / ".luaurc")) + assert luaurc == {"aliases": {"SomeLib": "SomeLib"}} + + +def test_extract_no_luaurc_when_only_project_alias(fs: MemoryFS): + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/Main"}, + main_source='require("./lib/foo")\n', + modules={"@root/lib/foo": "return 1\n"}, + ) + extract_to_dir(parsed, fs, OUT) + assert set(fs.iter_files()) == { + PurePosixPath("/out/Main.luau"), + PurePosixPath("/out/lib/foo.luau"), + } + + +def test_extract_places_modules_at_expected_paths(fs: MemoryFS): + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/Main"}, + main_source="MAIN\n", + modules={ + "@root/lib/foo": "FOO\n", + "@SomeLib/util": "UTIL\n", + "@SomeLib": "SOMELIB_INIT\n", + }, + ) + extract_to_dir(parsed, fs, OUT) + assert {p: fs.read(p) for p in fs.iter_files() if p.name != ".luaurc"} == { + PurePosixPath("/out/Main.luau"): "MAIN\n", + PurePosixPath("/out/lib/foo.luau"): "FOO\n", + PurePosixPath("/out/SomeLib/util.luau"): "UTIL\n", + PurePosixPath("/out/SomeLib/init.luau"): "SOMELIB_INIT\n", + } + + +def test_extract_round_trips_in_memory(): + src = MemoryFS.from_dict({ + "/proj/Main.luau": 'require("./lib/foo")\nreturn 1\n', + "/proj/lib/foo.luau": 'return "foo"\n', + }) + original = bundle(src, PurePosixPath("/proj"), PurePosixPath("/proj/Main.luau"), "myhud") + + out = MemoryFS() + extract_to_dir(parse_bundle(original), out, PurePosixPath("/extracted")) + + rebuilt = bundle(out, PurePosixPath("/extracted"), PurePosixPath("/extracted/Main.luau"), "myhud") + assert rebuilt == original + + +def test_extract_round_trips_with_external_alias_in_memory(): + src = MemoryFS.from_dict({ + "/proj/.luaurc": json.dumps({"aliases": {"SomeLib": "Packages/SomeLib"}}), + "/proj/Main.luau": 'require("@SomeLib/util")\n', + "/proj/Packages/SomeLib/util.luau": 'return "u"\n', + }) + original = bundle(src, PurePosixPath("/proj"), PurePosixPath("/proj/Main.luau"), "myhud") + + out = MemoryFS() + extract_to_dir(parse_bundle(original), out, PurePosixPath("/extracted")) + assert out.is_file(PurePosixPath("/extracted/SomeLib/util.luau")) + assert out.is_file(PurePosixPath("/extracted/.luaurc")) + + rebuilt = bundle(out, PurePosixPath("/extracted"), PurePosixPath("/extracted/Main.luau"), "myhud") + assert rebuilt == original + + +def test_extract_refuses_collision(fs: MemoryFS): + """Bare alias key and explicit `init` leaf both target init.luau.""" + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/Main"}, + main_source="\n", + modules={ + "@root": "return {}\n", # -> out/init.luau + "@root/init": "return {}\n", # -> out/init.luau (collision) + }, + ) + with pytest.raises(ExtractCollisionError): + extract_to_dir(parsed, fs, OUT) + assert list(fs.iter_files()) == [] + + +def test_extract_refuses_to_clobber(): + fs = MemoryFS.from_dict({"/out/Main.luau": "DO NOT TOUCH\n"}) + before = set(fs.iter_files()) + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/Main"}, + main_source='return 1\n', + modules={"@root/lib/foo": "FOO\n"}, + ) + with pytest.raises(ExtractClobberError): + extract_to_dir(parsed, fs, OUT) + assert set(fs.iter_files()) == before + assert fs.read(OUT / "Main.luau") == "DO NOT TOUCH\n" + + +def test_extract_rejects_traversal_in_module_key(fs: MemoryFS): + """A hand-crafted bundle with `..` components in a key must not write + outside . Our bundler never emits these; this defends against + malicious or corrupted bundles.""" + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/Main"}, + main_source="\n", + modules={"@root/../etc/passwd": "owned\n"}, + ) + with pytest.raises(ExtractUnsafeKeyError): + extract_to_dir(parsed, fs, OUT) + assert list(fs.iter_files()) == [] + + +def test_extract_rejects_traversal_in_main_key(fs: MemoryFS): + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/../escape"}, + main_source="\n", + modules={}, + ) + with pytest.raises(ExtractUnsafeKeyError): + extract_to_dir(parsed, fs, OUT) + assert list(fs.iter_files()) == [] + + +def test_extract_rejects_empty_component(fs: MemoryFS): + """`@root//foo` produces an empty middle component.""" + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud", "main": "@root/Main"}, + main_source="\n", + modules={"@root//foo": "x\n"}, + ) + with pytest.raises(ExtractUnsafeKeyError): + extract_to_dir(parsed, fs, OUT) + assert list(fs.iter_files()) == [] + + +def test_extract_errors_when_main_directive_missing(fs: MemoryFS): + parsed = ParsedBundle( + fields={"version": "1", "project": "myhud"}, + main_source="\n", + modules={}, + ) + with pytest.raises(ExtractMissingMainError): + extract_to_dir(parsed, fs, OUT) + assert list(fs.iter_files()) == [] diff --git a/tools/slua_bundle/tests/test_memory_fs.py b/tools/slua_bundle/tests/test_memory_fs.py new file mode 100644 index 00000000..7f3f059c --- /dev/null +++ b/tools/slua_bundle/tests/test_memory_fs.py @@ -0,0 +1,49 @@ +"""MemoryFS path normalization at construction and lookup.""" + +from pathlib import PurePosixPath + +import pytest + +from slua_bundle import MemoryFS + + +def test_from_dict_normalizes_keys(): + vfs = MemoryFS.from_dict({"/a/../b/foo.luau": "hello"}) + assert PurePosixPath("/b/foo.luau") in vfs.files + assert PurePosixPath("/a/../b/foo.luau") not in vfs.files + + +def test_read_normalizes_lookup(): + vfs = MemoryFS.from_dict({"/b/foo.luau": "hello"}) + assert vfs.read(PurePosixPath("/c/../b/foo.luau")) == "hello" + assert vfs.read(PurePosixPath("/b/./foo.luau")) == "hello" + + +def test_is_file_normalizes(): + vfs = MemoryFS.from_dict({"/x/y.luau": "."}) + assert vfs.is_file(PurePosixPath("/x/./y.luau")) + assert not vfs.is_file(PurePosixPath("/x/missing.luau")) + + +def test_is_dir_implicit_from_children(): + vfs = MemoryFS.from_dict({"/proj/src/main.luau": "."}) + assert vfs.is_dir(PurePosixPath("/proj")) + assert vfs.is_dir(PurePosixPath("/proj/src")) + assert vfs.is_dir(PurePosixPath("/proj/./src")) + assert not vfs.is_dir(PurePosixPath("/proj/src/main.luau")) + assert not vfs.is_dir(PurePosixPath("/missing")) + + +def test_read_missing_raises(): + vfs = MemoryFS.from_dict({"/a.luau": "."}) + with pytest.raises(FileNotFoundError): + vfs.read(PurePosixPath("/b.luau")) + + +def test_string_inputs_accepted(): + """All public MemoryFS methods accept either a string or a PurePosixPath.""" + vfs = MemoryFS.from_dict({"/proj/src/main.luau": "hello"}) + assert vfs.is_file("/proj/src/main.luau") + assert vfs.is_dir("/proj/src") + assert vfs.read("/proj/src/main.luau") == "hello" + assert vfs.read("/proj/./src/../src/main.luau") == "hello" diff --git a/tools/slua_bundle/tests/test_resolver.py b/tools/slua_bundle/tests/test_resolver.py new file mode 100644 index 00000000..8f1c01ba --- /dev/null +++ b/tools/slua_bundle/tests/test_resolver.py @@ -0,0 +1,106 @@ +"""Lexical resolution of require strings to canonical keys.""" + +import pytest + +from slua_bundle import ( + BareIdentifierError, + InvalidPathComponentError, + RelativeRequireWithoutAnchorError, + RequireEscapesAliasError, + UnknownAliasError, + resolve, +) + +KNOWN = {"myhud", "SomeLib"} + + +def test_absolute_alias_passthrough(): + assert resolve("@myhud/lib/foo", "@myhud/Main", KNOWN) == "@myhud/lib/foo" + + +def test_self_from_init_module_resolves_to_alias_root(): + """@self/x from an init.luau-style anchor (no leaf in the key) lands at the alias root. + This is the realistic / useful case -- @self in an init.luau is sibling-like. + """ + assert resolve("@self/lib/foo", "@myhud", KNOWN) == "@myhud/lib/foo" + + +def test_self_from_alias_root_with_no_leaf(): + """Same as above with a different alias -- explicit pin on Luau's behavior.""" + assert resolve("@self/util", "@SomeLib", KNOWN) == "@SomeLib/util" + + +def test_self_from_leaf_includes_filename_in_path(): + """@self/x from a leaf file resolves *under* a subdir named after the file + (extension stripped). This matches Luau's RequireNavigator semantics; in + practice the subdir rarely exists, so @self in a leaf is rarely useful. + """ + assert resolve("@self/lib/foo", "@myhud/HudController", KNOWN) == "@myhud/HudController/lib/foo" + assert resolve("@self/sibling", "@myhud/lib/bar", KNOWN) == "@myhud/lib/bar/sibling" + + +def test_dot_relative_resolves_to_anchor_dir(): + assert resolve("./bar", "@myhud/lib/foo", KNOWN) == "@myhud/lib/bar" + + +def test_dotdot_relative_pops_one_level(): + assert resolve("../sibling", "@myhud/lib/foo", KNOWN) == "@myhud/sibling" + + +def test_dot_relative_collapses_dot_segments(): + assert resolve("./a/./b/../c", "@myhud/lib/foo", KNOWN) == "@myhud/lib/a/c" + + +def test_bare_identifier_rejected(): + with pytest.raises(BareIdentifierError): + resolve("foo", "@myhud/Main", KNOWN) + + +def test_unknown_alias_rejected(): + with pytest.raises(UnknownAliasError): + resolve("@nope/x", "@myhud/Main", KNOWN) + + +def test_escape_via_dotdot_inside_absolute_alias(): + with pytest.raises(RequireEscapesAliasError): + resolve("@myhud/../escape", "@myhud/Main", KNOWN) + + +def test_escape_via_dotdot_in_relative(): + with pytest.raises(RequireEscapesAliasError): + resolve("../../escape", "@myhud/lib/foo", KNOWN) + + +def test_escape_via_self_dotdot(): + # Need three `..` to escape past @myhud/lib/foo's three components. + with pytest.raises(RequireEscapesAliasError): + resolve("@self/../../../escape", "@myhud/lib/foo", KNOWN) + + +def test_relative_without_anchor_rejected(): + """MAIN without main= cannot use relative requires.""" + with pytest.raises(RelativeRequireWithoutAnchorError): + resolve("./x", None, KNOWN) + + +def test_self_without_anchor_rejected(): + with pytest.raises(RelativeRequireWithoutAnchorError): + resolve("@self/x", None, KNOWN) + + +def test_absolute_alias_works_without_anchor(): + """MAIN without main= can still use absolute aliases.""" + assert resolve("@myhud/lib/foo", None, KNOWN) == "@myhud/lib/foo" + + +def test_path_component_starting_with_dot_rejected(): + """A component like '.bashrc' would resolve to a hidden file -- reject.""" + with pytest.raises(InvalidPathComponentError): + resolve("@myhud/.bashrc", "@myhud/Main", KNOWN) + with pytest.raises(InvalidPathComponentError): + resolve("./.config/foo", "@myhud/lib/x", KNOWN) + + +def test_path_component_with_nul_rejected(): + with pytest.raises(InvalidPathComponentError): + resolve("@myhud/lib/foo\x00bar", "@myhud/Main", KNOWN) diff --git a/tools/slua_bundle/tests/test_runtime.py b/tools/slua_bundle/tests/test_runtime.py new file mode 100644 index 00000000..04b70a22 --- /dev/null +++ b/tools/slua_bundle/tests/test_runtime.py @@ -0,0 +1,227 @@ +"""Bundle parsing and execution-graph traversal.""" + +from __future__ import annotations + +import pytest + +from slua_bundle import ( + BareIdentifierError, + BundleParseError, + CircularDependencyError, + UnknownAliasError, + parse_bundle, + simulate, +) + + +def _make(*lines: str) -> str: + return "\n".join(lines) + "\n" + + +def _v1_header(project: str | None = "myhud", main: str | None = None) -> tuple[str, ...]: + parts = ["-- !!LUABUNDLE:VERSION 1"] + if project is not None: + parts.append(f"-- !!LUABUNDLE:PROJECT {project}") + if main is not None: + parts.append(f"-- !!LUABUNDLE:MAIN {main}") + parts.append("-- !!LUABUNDLE:BODY") + return tuple(parts) + + +def test_parse_bundle_extracts_fields_and_modules(): + text = _make( + *_v1_header(project="myhud", main="@myhud/Main"), + "main body line 1", + "-- !!LUABUNDLE:MODULE @myhud/lib/foo", + "foo body", + ) + parsed = parse_bundle(text) + assert parsed.fields == {"version": "1", "project": "myhud", "main": "@myhud/Main"} + assert parsed.main_source == "main body line 1\n" + assert parsed.modules == {"@myhud/lib/foo": "foo body\n"} + + +def test_parse_bundle_rejects_missing_version(): + text = _make( + "-- !!LUABUNDLE:PROJECT myhud", + "-- !!LUABUNDLE:BODY", + "main body", + ) + with pytest.raises(BundleParseError, match="VERSION"): + parse_bundle(text) + + +def test_parse_bundle_rejects_unsupported_version(): + text = _make( + "-- !!LUABUNDLE:VERSION 99", + "-- !!LUABUNDLE:PROJECT myhud", + "-- !!LUABUNDLE:BODY", + ) + with pytest.raises(BundleParseError, match="unsupported VERSION"): + parse_bundle(text) + + +def test_parse_bundle_rejects_duplicate_project(): + text = _make( + "-- !!LUABUNDLE:VERSION 1", + "-- !!LUABUNDLE:PROJECT myhud", + "-- !!LUABUNDLE:PROJECT otherhud", + "-- !!LUABUNDLE:BODY", + ) + with pytest.raises(BundleParseError, match="duplicate PROJECT"): + parse_bundle(text) + + +def test_parse_bundle_rejects_duplicate_main(): + text = _make( + "-- !!LUABUNDLE:VERSION 1", + "-- !!LUABUNDLE:PROJECT myhud", + "-- !!LUABUNDLE:MAIN @myhud/A", + "-- !!LUABUNDLE:MAIN @myhud/B", + "-- !!LUABUNDLE:BODY", + ) + with pytest.raises(BundleParseError, match="duplicate MAIN"): + parse_bundle(text) + + +def test_parse_bundle_rejects_unknown_header_directive(): + text = _make( + "-- !!LUABUNDLE:VERSION 1", + "-- !!LUABUNDLE:PROJECT myhud", + "-- !!LUABUNDLE:FUTURE-THING something", + "-- !!LUABUNDLE:BODY", + "main body", + ) + with pytest.raises(BundleParseError, match="unknown header directive"): + parse_bundle(text) + + +def test_parse_bundle_rejects_missing_body_marker(): + text = _make( + "-- !!LUABUNDLE:VERSION 1", + "-- !!LUABUNDLE:PROJECT myhud", + ) + with pytest.raises(BundleParseError, match="missing BODY"): + parse_bundle(text) + + +def test_parse_bundle_accepts_missing_project(): + """PROJECT is optional (advisory viewer-linkage metadata).""" + text = _make( + "-- !!LUABUNDLE:VERSION 1", + "-- !!LUABUNDLE:MAIN @root/X", + "-- !!LUABUNDLE:BODY", + "main body", + ) + parsed = parse_bundle(text) + assert "project" not in parsed.fields + assert parsed.main_source == "main body\n" + + +def test_parse_bundle_rejects_missing_main(): + """MAIN is required; consumers reject bundles without it.""" + text = _make( + "-- !!LUABUNDLE:VERSION 1", + "-- !!LUABUNDLE:PROJECT myhud", + "-- !!LUABUNDLE:BODY", + "main body", + ) + with pytest.raises(BundleParseError, match="missing MAIN"): + parse_bundle(text) + + +def test_parse_bundle_rejects_unexpected_marker_after_body(): + text = _make( + *_v1_header(main="@myhud/Main"), + "main body", + "-- !!LUABUNDLE:FUTURE-THING something", + "-- !!LUABUNDLE:MODULE @myhud/x", + "x body", + ) + with pytest.raises(BundleParseError, match="unexpected marker"): + parse_bundle(text) + + +def test_parse_bundle_rejects_body_before_body_marker(): + text = _make( + "-- !!LUABUNDLE:VERSION 1", + "-- !!LUABUNDLE:PROJECT myhud", + "stray body line", + "-- !!LUABUNDLE:BODY", + ) + with pytest.raises(BundleParseError, match="body content before BODY"): + parse_bundle(text) + + +def test_simulate_runs_each_module_body_once_for_dedup(): + """Dedup via canonical key. MAIN is init.luau-style (canonical @myhud, no leaf) + so @self/x is sibling-like; bar uses ./foo to reach the same module.""" + text = _make( + *_v1_header(project="myhud", main="@myhud"), + 'require("@self/lib/foo"); require("@self/lib/bar")', + "-- !!LUABUNDLE:MODULE @myhud/lib/foo", + 'return "foo"', + "-- !!LUABUNDLE:MODULE @myhud/lib/bar", + 'require("./foo"); return "bar"', + ) + body_runs = simulate(text) + assert body_runs["@myhud/lib/foo"] == 1 + assert body_runs["@myhud/lib/bar"] == 1 + assert body_runs["@myhud"] == 1 + + +def test_simulate_circular_dependency_detected(): + text = _make( + *_v1_header(project="p", main="@p/Main"), + 'require("@p/A")', + "-- !!LUABUNDLE:MODULE @p/A", + 'require("@p/B")', + "-- !!LUABUNDLE:MODULE @p/B", + 'require("@p/A")', + ) + with pytest.raises(CircularDependencyError) as excinfo: + simulate(text) + msg = str(excinfo.value) + assert "@p/A" in msg and "@p/B" in msg + + +def test_simulate_main_as_import_target_is_a_cycle(): + """A module requiring MAIN's canonical key creates a cycle.""" + text = _make( + *_v1_header(project="p", main="@p/Main"), + 'require("@p/A")', + "-- !!LUABUNDLE:MODULE @p/A", + 'require("@p/Main")', + ) + with pytest.raises(CircularDependencyError): + simulate(text) + + +def test_simulate_main_with_anchor_accepts_relative_and_absolute(): + """init.luau-style MAIN: @self/x and ./x both land at the alias root and dedup.""" + text = _make( + *_v1_header(project="p", main="@p"), + 'require("@self/lib/foo"); require("./lib/foo")', + "-- !!LUABUNDLE:MODULE @p/lib/foo", + 'return "foo"', + ) + body_runs = simulate(text) + assert body_runs["@p/lib/foo"] == 1 + + +def test_simulate_bare_identifier_rejected(): + text = _make( + *_v1_header(project="p", main="@p/Main"), + 'require("foo")', + ) + with pytest.raises(BareIdentifierError): + simulate(text) + + +def test_simulate_unknown_alias_rejected(): + text = _make( + *_v1_header(project="p", main="@p/Main"), + 'require("@nope/x")', + ) + with pytest.raises(UnknownAliasError): + simulate(text)