Skip to content

Qt 6 GUI for SQL migrations on top of dbtool#474

Open
christianparpart wants to merge 4 commits intomasterfrom
feature/migrations-gui
Open

Qt 6 GUI for SQL migrations on top of dbtool#474
christianparpart wants to merge 4 commits intomasterfrom
feature/migrations-gui

Conversation

@christianparpart
Copy link
Copy Markdown
Member

@christianparpart christianparpart commented Apr 23, 2026

The headline change is dbtool-gui, a Qt 6 / QML front-end for browsing, previewing, and applying SQL migrations against any configured connection profile. The CLI / library work in the rest of this PR — new dbtool subcommands, profile + secrets handling, ODBC enumeration, cross-backend migration semantics — grew up while building the GUI and is needed to support it. To keep the public Lightweight API clean, the helpers that only dbtool and dbtool-gui use live in a new tools-private static library (src/tools/shared/, target tools_shared); they are not installed and not part of the Lightweight::Lightweight ABI.

The commits on this branch are deliberately split: the foundational CLI / library work first, the GUI on top.

Headline change — dbtool GUI

src/tools/dbtool-gui/ (opt-in via LIGHTWEIGHT_BUILD_GUI=ON): Qt 6 / QML front-end for dbtool against any configured connection profile.

  • Migration plan view grouped by LIGHTWEIGHT_SQL_RELEASE markers, with per-migration SQL preview and one-click apply / revert / backup-restore actions (MigrationRunner, BackupRunner running on QThread workers).
  • Connection panel backed by Config::ProfileStore + Secrets::SecretResolver, with ODBC DSN auto-discovery and an optional qtkeychain-based credential backend.
  • Ad-hoc SQL query panel (SqlQueryRunner, SqlSyntaxHighlighter, SqlResultModel) so the GUI is not migration-only.
  • Light/dark ThemeController and a small reusable QML component set (Card, StatusPill, KineticListView, WheelScrollAmplifier, ...).

Tools-private shared library (src/tools/shared/, target tools_shared)

These helpers are consumed by dbtool, dbtool-gui, lup2dbtool, and the test suite. They are STATIC, not installed, and not part of the public Lightweight CMake export. yaml-cpp is linked PRIVATE on tools_shared, so it no longer propagates to the installed Lightweight package.

  • Config::ProfileStore — YAML-backed store for named connection profiles, used by both dbtool and the GUI so credentials no longer live in plaintext on disk.
  • Secrets::SecretResolver with EnvBackend / FileBackend / StdinBackend — pluggable indirection for credentials referenced from profiles. FileBackend refuses files with mode wider than 0600.
  • Odbc::DataSourceEnumerator — wraps SQLDataSources and driver enumeration so the GUI can populate a DSN dropdown.
  • CodeGen::SplitFileWriter — codegen helper used by lup2dbtool to emit one C++ file per migration with a shared CMake snippet.

Lightweight library changes (public API)

  • Structured MigrationException — carries operation / timestamp / title / step index / failed SQL / driver message, so both the CLI and GUI can show actionable error context without parsing what().
  • Bounded migration apply/previewMigrationManager::ApplyPendingMigrationsUpTo / PreviewPendingMigrationsUpTo / FindReleaseByVersion, the forward counterparts to the existing rollback path. Plus FoldRegisteredMigrations as a primitive for callers that want to collapse a chain.
  • Cross-backend migration semantics for legacy SQL corpora — composite WhereExpression / SetExpression on Update / Delete, idempotent variants of AddColumn / DropColumn / AddForeignKey / DropIndex across SQLite / PostgreSQL / MSSQL, deterministic FK constraint names, a SQLite ALTER TABLE rebuild path for AddForeignKey / DropForeignKey, and per-migration compat policy with a lup-truncate renderer.
  • HardReset — drops tables in batched transactions; preserved-tables list compares by unqualified name only, so an unqualified plan resolves correctly against engine-specific default schemas (dbo / public / none).
  • MSSQL UTF-8 string literalsNCHAR concatenation so non-BMP characters survive round-trip.
  • SqlSchema cross-engine introspection fixesSQLForeignKeys row grouping now keys on FK_NAME (with a SQLite PRAGMA fallback), plus assorted reader gaps that previously caused identical migrations to look like drift across engines.
  • Robustness fixesSqlConnection now surfaces the original driver diagnostic when Connect() fails; SqlStatement tolerates SQL_NO_DATA from SQLExecDirect; SqlConnectInfo::EnsureSqliteDatabaseFileExists bootstraps a missing file-based SQLite database.

dbtool — new commands and reworks

  • Migrated onto Config::ProfileStore + Secrets::SecretResolver (drops the direct yaml-cpp link from dbtool and the bespoke ~/.config/dbtool/dbtool.yml parsing); adds --profile <name> and surfaces the latest applied release in status. An explicit --connection-string without --profile no longer silently merges profile defaults (schema, pluginsDir, ...) — the typed connection string pins the backend.
  • dbtool exec <QUERY> — ad-hoc SQL execution against a profile.
  • dbtool hard-reset — drop all tables in batched transactions (used by the GUI and CI).
  • dbtool unicode-upgrade-tables — bulk UTF-8 upgrade for legacy MSSQL schemas.
  • dbtool rewrite-checksums — rewrite stored migration checksums after an authorised content change, with plugin policy propagation.
  • dbtool migrate-to-release <VERSION> — forward-direction counterpart of rollback-to-release; resolves the version to its declared highestTimestamp and applies pending migrations up to that point.
  • dbtool status — aligned label column and reports the latest available release alongside the latest applied one.
  • --show-examples — long-form examples moved out of the default --help.
  • dbtool SqlLogger now routes through the standard logger.

lup2dbtool and LupMigrationsPlugin

  • New LupSqlParser, WhereClauseParser, StringUtils, expanded SqlStatementParser and CodeGenerator lift the legacy init_m_*.sql / upd_m_*.sql corpus into one C++ file per migration (lup_{version}.cpp), with --emit-cmake for the shared CMake snippet.
  • Per-file encoding detection (UTF-8 / UTF-16 / Windows-1252) with strict explicit-mode flags for reproducible CI runs.
  • --force-unicode is now the default (with --no-force-unicode opt-out).
  • LupMigrationsPlugin installs the lup-truncate compat policy by timestamp (cutoff extended to all LUP migrations), scrubs proprietary paths from placeholder comments, and cold-start bootstraps lup2dbtool itself when invoked from a parent build that is still mid-configure (with CONFIGURE_DEPENDS glob refresh).

Build / packaging

  • New tools_shared STATIC library at src/tools/shared/ — see above. Linked PUBLIC by dbtool_lib (so dbtool-gui inherits it transitively) and by lup2dbtool_lib; tests inherit it via those.
  • LIGHTWEIGHT_BUILD_GUI is gated by a Qt 6 auto-probe in the top-level CMake; if Qt isn't found the option is silently downgraded to OFF instead of failing the configure.
  • Lightweight_CPMAddPackage wrapper around CPMAddPackage so clang-tidy no longer fires on _deps/ (stdexec, libzip, nlohmann_json, reflection-cpp, tracy). First-party tidy coverage unchanged (89 invocations, all under src/).
  • MSVC C4251/C4275 suppression for shared builds.
  • Installed Lightweight header tree no longer contains Config/, Secrets/, Odbc/, or CodeGen/; yaml-cpp is no longer in the installed Lightweight-targets exports.

Documentation

  • docs/dbtool.md — covers the new subcommands and profile flow.

Risk

  • Surface area: large, but the GUI is opt-in (LIGHTWEIGHT_BUILD_GUI=ON) and the new dbtool subcommands are additive.
  • Behaviour change: dbtool no longer reads ~/.config/dbtool/dbtool.yml directly — existing users must migrate to a profile. --connection-string alone no longer pulls profile defaults; users who relied on that need to also pass --profile.
  • Public API: no breaking changes to the installed Lightweight::Lightweight headers — but the helpers that previously sat in src/Lightweight/{Config,Secrets,Odbc,CodeGen}/ (which were never exported by the C++20 module and never consumed inside the core library) have moved to src/tools/shared/. They remain in the Lightweight:: namespace; only the #include path and link target change for tool-side consumers.

Coverage

  • New unit tests: ConfigProfileStoreTests, SecretResolverTests, DataSourceEnumeratorTests, plus large additions to Lup2DbtoolTests, MigrationTests, and QueryBuilderTests.
  • MigrationTests now verifies HardReset / UnicodeUpgradeTables post-conditions through SqlSchema::ReadAllTables instead of sqlite_schema, so the cases run on every backend the suite is parameterised over.

Test plan

  • Build with LIGHTWEIGHT_BUILD_GUI=ON, launch dbtool-gui, exercise apply / revert / backup-restore against a SQLite profile.
  • Confirm the SQL preview, ad-hoc query panel, and theme toggle work end-to-end in the GUI.
  • Build with LIGHTWEIGHT_BUILD_GUI=OFF (default) and confirm CLI-only build is unchanged.
  • Run dbtool exec, dbtool hard-reset, dbtool unicode-upgrade-tables, dbtool rewrite-checksums, dbtool migrate-to-release against a throwaway profile.
  • Run the full unit test suite under each enabled backend (sqlite3, mssql2022, postgres).
  • Verify cmake --install does not ship Config/, Secrets/, Odbc/, or CodeGen/ under include/Lightweight/, and that the installed Lightweight-targets.cmake no longer references yaml-cpp.

@christianparpart christianparpart requested a review from a team as a code owner April 23, 2026 22:55
@github-actions github-actions Bot added documentation Improvements or additions to documentation CLI command line interface tools Query Builder tests Core API Query Formatter SQL dialect implementations CMake labels Apr 23, 2026
@christianparpart christianparpart marked this pull request as draft April 23, 2026 22:55
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 5 times, most recently from 103a585 to 156d6a7 Compare April 26, 2026 11:54
@christianparpart christianparpart changed the title Add Qt 6 migrations GUI and supporting infrastructure Add dbtool GUI plus diff/exec/fold and broader migration tooling Apr 27, 2026
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 3 times, most recently from 69c4852 to 2fcf48c Compare April 29, 2026 20:04
@github-actions github-actions Bot added the Data Binder SQL Data Binder support label Apr 29, 2026
@github-actions github-actions Bot removed the Data Binder SQL Data Binder support label Apr 30, 2026
@github-actions github-actions Bot added the Data Binder SQL Data Binder support label Apr 30, 2026
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 2 times, most recently from 49cd796 to 301326c Compare April 30, 2026 12:24
@github-actions github-actions Bot removed the Data Binder SQL Data Binder support label Apr 30, 2026
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 2 times, most recently from abcded1 to 8860114 Compare April 30, 2026 17:58
@christianparpart christianparpart force-pushed the feature/migrations-gui branch from 742a13a to da47294 Compare May 6, 2026 16:07
@christianparpart christianparpart changed the title Add dbtool GUI plus diff/exec/fold and broader migration tooling Add dbtool GUI plus diff/exec/fold/migrate-to-release and broader migration tooling May 6, 2026
christianparpart added a commit that referenced this pull request May 6, 2026
Three independent regressions were reported by the previous CI run on
this branch (see PR #474). Each is fixed at its root cause:

1. **`std::views::chunk` not in libc++ on the C++26 reflection job.**
   `SqlMigration.cpp:HardReset` used the C++23 `std::views::chunk`
   adapter to batch DROP TABLEs. The reflection-job toolchain ships
   an older libc++ that does not yet provide `__cpp_lib_ranges_chunk`
   (≥ 202202L) and the build aborted with "no member named 'chunk'
   in namespace 'std::ranges::views'". Gate the modern path behind
   the standard feature-test macro and fall back to a `std::span`
   subspan-based loop on stdlibs that lack it. The Linux-clang-debug
   job's newer libc++ keeps the modern path.

2. **SQLite FK-rebuild path silently disabled.**
   `SqlMigration::ExecuteScriptRespectingSqliteGuards` consults
   `SqlQueryFormatter::RequiresTableRebuildForForeignKeyChange()` to
   decide whether a sentinel-prefixed script triggers the table
   rebuild or is run as-is. The SQLite override of that hook was
   missing from `SQLiteQueryFormatter` on this branch, so the base
   class's `false` was returned and the sentinel script (which is
   intentionally a *commented-out* ALTER + sentinel) was executed
   verbatim. No FK was added but no error was raised either, which
   is exactly the failure the suite reported:
       MigrationTests.cpp:1064: CHECK( found ) — false
       MigrationTests.cpp:1145/1146: foundPa/foundPb — false
   Restore the override (returns `true`) and the rebuild path takes
   over again.

   While here, restore the canonical FK constraint name builder in
   `SQLiteQueryFormatter::BuildForeignKeyConstraint` — it had drifted
   to a hand-rolled `FK_{table}_{column}` while the runtime rebuild
   side (`SqliteRebuildAddForeignKey`) still calls
   `BuildForeignKeyConstraintName`. Now CREATE-table and ALTER-table
   produce identical names.

3. **Windows-cl `/WX` errors C4251 / C4275 on new DLL-exported
   classes.** `MigrationException` (this branch) and the new
   `Config::ProfileStore`, `Secrets::SecretResolver` and the three
   `Secrets::backends/*` classes carry `LIGHTWEIGHT_API` because
   their non-inline methods cross the DLL boundary. They expose
   STL-typed members (`std::string`, `std::vector`,
   `std::filesystem::path`) or derive from a non-DLL-interface base
   (`ISecretBackend`), which triggers the standard
   "STL-across-the-DLL-boundary" warnings under `/W4 /WX`. Suppress
   both warnings on the `Lightweight` target when built shared on
   MSVC. Producer and consumer always share the same runtime in our
   matrix, so neither warning indicates an ABI bug.

Verified locally against the full test suite on:
- sqlite3:   590 cases, 589 passed, 1 skipped, 5505 assertions
- mssql2022: 590 cases, 589 passed, 1 skipped, 5493 assertions
- postgres:  590 cases, 589 passed, 1 skipped, 5495 assertions

Risk: low. The chunk fallback is observationally identical to the
view-based path. The FK rebuild override restores prior behaviour.
The MSVC pragma is the standard idiom and only narrows two specific
warnings on a single target.

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 2 times, most recently from e030c86 to 258c0cc Compare May 8, 2026 21:00
@christianparpart christianparpart changed the title Add dbtool GUI plus diff/exec/fold/migrate-to-release and broader migration tooling Add dbtool GUI plus migrate-to-release, hard-reset, exec and broader migration tooling May 8, 2026
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 3 times, most recently from 973202e to 594ce8a Compare May 9, 2026 14:44
@christianparpart christianparpart changed the title Add dbtool GUI plus migrate-to-release, hard-reset, exec and broader migration tooling Qt 6 GUI for SQL migrations on top of dbtool May 9, 2026
@christianparpart christianparpart self-assigned this May 9, 2026
@christianparpart christianparpart marked this pull request as ready for review May 9, 2026 14:47
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 6 times, most recently from 1b4b0de to 815e84d Compare May 10, 2026 08:47
christianparpart added a commit that referenced this pull request May 10, 2026
…hang

Root cause of the "Windows Tests (MS SQL Server (LocalDB))" CI hang on
PR #474:

`ParseConnectionString` does

    connectionString.value
        | std::views::split(';')
        | std::views::transform([](auto pair_view) {
              return std::string_view(&*pair_view.begin(),
                                      static_cast<size_t>(std::ranges::distance(pair_view)));
          });

When the connection string ends in `;` (the CI's LocalDB string is
`"…;Trusted_Connection=Yes;"`), `views::split` yields a trailing empty
fragment. `&*pair_view.begin()` dereferences `end()` on an empty
subrange, which is UB. On clang-cl Debug builds with /RTC1 the UB
manifests as the *next* operation hanging — specifically inside
`EnsureSqliteDatabaseFileExists`, which calls `ParseConnectionString`
on every dbtool invocation regardless of dialect.

Linux MSSQL Docker tests in CI use a connection string *without* a
trailing semicolon, so they never hit the bug. Same for PostgreSQL on
Windows. Only the Windows + LocalDB combination triggered it, hence
the platform-pair-specific failure.

Reproduced locally on Windows + clang-cl Debug:
    dbtool ... --connection-string "...;Trusted_Connection=Yes;"
hangs at EnsureSqliteDatabaseFileExists. Removing the trailing `;`
makes it pass. Adding the empty-fragment filter makes both forms work.

Fix: filter out empty subranges before the transform. Equivalent to
the existing for-loop's `pair.find('=') != npos` check, but moves the
filter ahead of the UB-prone string_view construction.

Regression coverage added to `src/tests/CoreTests.cpp`:
`ParseConnectionString tolerates empty fragments` exercises trailing,
leading, and double-semicolon plus only-semicolons and entirely-empty
connection strings — all of which used to trigger the dereference.

This commit also keeps the `DBTOOL_TRACE` breadcrumb scaffolding from
the previous diagnostic commit (gated on the env var, free in normal
runs) plus a finer-grained set inside `SetupConnectionString` itself.
That's how we localised the hang to `EnsureSqliteDatabaseFileExists`
in the first place; keeping the breadcrumbs costs nothing in normal
usage and means the next time something wedges on a CI runner we get
a precise failure point in minutes instead of guessing.

Risk: extremely low. The filter only changes behavior for empty
fragments which were already silently dropped by the for-loop's
find('=') check anyway. No connection string that previously parsed
correctly will parse differently now.

Local verification:
- Windows + clang-cl Debug: full SQLite suite (593/593) green;
  `test_dbtool.py --test-env=sqlite3` SUCCESS;
  `dbtool ... list-pending` against LocalDB now returns instead of
  hanging.

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart christianparpart force-pushed the feature/migrations-gui branch 2 times, most recently from 96174e5 to 0b80aab Compare May 10, 2026 16:45
Copy link
Copy Markdown
Member

@Yaraslaut Yaraslaut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some comments, i think that we need to have parallel execution of the backup/restore to make it fast

Comment thread src/tools/dbtool-gui/Models/ReleaseListModel.hpp
Comment thread src/tools/dbtool-gui/Models/MigrationListModel.hpp
Comment thread src/tools/dbtool-gui/ThemeController.hpp
Comment thread src/tools/dbtool-gui/SqlQueryRunner.hpp
Comment thread src/tools/dbtool-gui/QtKeychainBackend.hpp Outdated
Comment thread src/tools/dbtool-gui/BackupRunner.cpp
@christianparpart christianparpart force-pushed the feature/migrations-gui branch from 0b80aab to 1722417 Compare May 11, 2026 07:31
…ons, lup2dbtool

Bundles all non-GUI work from the migrations-gui branch (minus
unicode-upgrade-tables, which lives in its own follow-up commit).

Library
  - Config::ProfileStore: yaml-backed connection profile store
  - Secrets: env / file / stdin backends + SecretResolver
  - SqlMigration: structured errors, FoldRegisteredMigrations,
    bounded ApplyUpToTimestamp/PreviewUpToTimestamp, batched HardReset,
    cross-engine preserved-tables compare, per-migration CompatPolicy
    + lup-truncate renderer
  - SqlSchema: cross-engine introspection gap fixes
  - QueryFormatter: MSSQL StringLiteral UTF-8 via NCHAR concatenation;
    SQLite formatter touch-ups
  - CodeGen::SplitFileWriter shared codegen helper

dbtool / lup2dbtool / LupMigrationsPlugin
  - Migrate dbtool profile/secrets onto Lightweight; route SqlLogger to StandardLogger
  - exec <QUERY> for ad-hoc SQL
  - hard-reset, rewrite-checksums admin commands
  - migrate-to-release; status alignment + latest-available-release
  - --show-examples; skip profile defaults when only --connection-string set
  - CollectMigrations: propagate per-plugin CompatPolicy
  - LUP SQL -> C++ migration converter, WhereClauseParser, StringUtils
  - Per-file encoding detection + strict explicit modes
  - Default --force-unicode on; --no-force-unicode opt-out
  - CodeGenerator release-marker comment update
  - lup2dbtool: --manifest writes a newline-separated list of every emitted
    source file (consumed by the plugin's CMake at configure time)
  - LupMigrationsPlugin: cold-start bootstrap, manifest-driven source list
    (one `lup_<ver>.cpp` per emitted output, declared as add_custom_command
    OUTPUTs so ninja owns each file), side-build lup2dbtool;
    stale-manifest refresh during configure when any SQL file is newer
    than the manifest (so adding a brand-new SQL release is picked up by
    a single `cmake --preset` reconfigure); lup-truncate compat policy
    by timestamp

Tests / docs / infra
  - MigrationTests: portable live-schema introspection via ReadAllTables
  - ConfigProfileStore, SecretResolver, Lup2Dbtool tests
  - QueryBuilder/MigrationTests coverage for new features
  - test_dbtool.py additions
  - docs/dbtool.md updates
  - scripts/prepare-test-env.py + .github/prepare-test-run.sh additions
  - scripts/tests/docker-databases.py: --pull, streamed output

Build
  - tools_shared (STATIC, in src/tools/shared/) hosts Config / Secrets /
    SplitFileWriter, keeping them off the public Lightweight API. yaml-cpp
    is linked PRIVATE on tools_shared, so it no longer propagates to the
    installed Lightweight package.
  - MSVC C4251/C4275 suppression for shared builds.

Signed-off-by: Christian Parpart <christian@parpart.family>
Rewrites legacy `VARCHAR/CHAR` columns to `NVARCHAR/NCHAR` where the
registered migrations now declare wide types. Drops + re-adds touched
FKs, with a SQLite-specific path via `RebuildSqliteTable` for in-place
column-type rewrite.

Compares the folded plan's intended column types against
`SqlSchema::ReadAllTables` output; an upgrade is triggered iff
intended is `NVarchar`/`NChar` AND live is `Varchar`/`Char` with the
same `size`. Foreign keys touching any upgrade column are dropped
before the alter and re-added afterwards. Cross-backend.

Wired into dbtool through the shared `RunAdminCommand<Result>`
template — dry-run prints the diff, `--yes` confirms the destructive
action.

Tests: SQLite coverage for dry-run drift reporting plus an idempotent
roundtrip (running unicode-upgrade-tables twice in a row produces no
second-run drift).

Signed-off-by: Christian Parpart <christian@parpart.family>
…calDB hang fix

Adds a generic distributed-locking primitive that the migration system
*and* user code can use, replaces the migration-only `MigrationLock`
with it, and fixes the Windows + MS SQL LocalDB CI hang along the way.

- `Lightweight::SqlScopedLock` — RAII cross-process advisory lock.
  Throwing constructor for ergonomic use; non-throwing `TryConstruct`
  factory returning `std::expected<SqlScopedLock, SqlLockError>` for
  structured error handling. Move-only.
  - `IsLocked()`, `Name()`, explicit `Release()`.
  - User code can lock any string ("cron-leader", "queue-worker-N", …);
    two processes that pass the same string serialise on it.
- `Lightweight::SqlAdvisoryLockHandler` — abstract per-dialect
  interface returned by `SqlQueryFormatter::AdvisoryLockOps()`.
  - Three concrete handlers next to their formatters: SQL Server
    (`sp_getapplock` / `sp_releaseapplock`), PostgreSQL
    (`pg_advisory_lock` / `pg_advisory_unlock`), SQLite
    (`_lightweight_locks` table guarded by a unique constraint).
  - `BookkeepingTableNames()` lets tooling distinguish lock
    infrastructure from user data — see hard-reset hardening below.
- `Lightweight::SqlLockError` / `SqlLockFailureReason` — structured
  failure type so callers branch on Timeout / Deadlock / Cancelled /
  ParameterError / DriverError without parsing exception messages.

The dialect-specific code lives entirely in formatter overrides and
free helpers — `SqlScopedLock` itself contains zero `switch
(ServerType)`. Adding a new dialect is one new
`SqlAdvisoryLockHandler` subclass and one formatter override.

`SqlMigration::MigrationLock` and `MigrationLockHandler` are deleted;
dbtool's `OptionalMigrationLock` is now `OptionalScopedLock` wrapping
`SqlScopedLock { conn, "lightweight_migration" }`. The lock-table
rename `_migration_locks` → `_lightweight_locks` reflects the more
general use.

`MigrationManager::HardReset` now consults
`formatter.AdvisoryLockOps().BookkeepingTableNames()`. Lock
bookkeeping is skipped in `preservedTables` (so it's never reported as
user data) and explicitly dropped after `schema_migrations`. The
SQLite handler's `Release` honours the idempotent contract: a "no
such table" error after hard-reset drops the lock table is treated as
"lock is already gone" → success, not warning.

The "Windows Tests (MS SQL Server (LocalDB))" CI job was hanging for
the GitHub Actions 6-hour timeout on every run. Investigated end to
end and reproduced locally on Windows + clang-cl Debug:

`Lightweight::ParseConnectionString` does
`connectionString.value | std::views::split(';') | …
    std::string_view(&*pair_view.begin(), …)`.
For a connection string ending in `;` (the CI's LocalDB string is
`"…;Trusted_Connection=Yes;"`), `views::split` yields a trailing empty
fragment. `&*pair_view.begin()` dereferences `end()` on an empty
subrange — UB. On clang-cl Debug builds with /RTC1 the UB manifests
as the next operation hanging, specifically inside
`EnsureSqliteDatabaseFileExists` (which calls `ParseConnectionString`
on every dbtool invocation regardless of dialect). Linux MSSQL Docker
strings have no trailing `;`, so they never tripped it; same for
PostgreSQL on Windows. Only the Windows + LocalDB combination
triggered it.

Fix: filter out empty subranges before constructing the `string_view`
in `ParseConnectionString`. Regression coverage in
`src/tests/CoreTests.cpp` exercises trailing/leading/double-semicolon
plus only-semicolons and entirely-empty connection strings.

- `.github/workflows/build.yml`:
    - `timeout-minutes: 30` on the `windows_dbms_test_matrix` job so a
      hung dbtool invocation can never starve a runner for 6 hours.
    - `PYTHONUNBUFFERED: "1"` and `DBTOOL_TRACE: "1"` on the
      `Run dbtool tests` step so per-step markers and dbtool startup
      breadcrumbs surface in real time.
    - On-failure `sys.dm_exec_requests` / `dm_tran_locks` / `sp_who2`
      capture step gated on `matrix.test_env == 'mssql'`.
- `src/tests/test_dbtool.py`:
    - Per-command `timeout=180` and `stdin=subprocess.DEVNULL` on
      every dbtool invocation. On `TimeoutExpired`, capture hung
      command, partial stdout/stderr, `tasklist` snapshot, and a
      `sqlcmd` LocalDB DMV dump.
    - New step 10 exercises `dbtool hard-reset --yes` followed by
      `migrate`, regression-guarding the "drop lock table, then
      re-create on the next migrate" loop end to end.
- `src/tools/dbtool/main.cpp`: `DBTOOL_TRACE=1`-gated startup
  breadcrumbs at every notable boundary — main / parsing / profile /
  Windows-console / connection-string / dispatch / plugin-load /
  collect-migrations / create-history / list-pending. Free in normal
  runs; invaluable when correlated with the `dm_exec_requests`
  capture next time something wedges.

- `src/tests/MigrationTests.cpp`: replaces the old `MigrationLock`
  tests with `[SqlScopedLock]`-tagged equivalents (singleton
  stability, throwing + non-throwing acquire, idempotent Release) plus
  a `[HardReset]` test asserting bookkeeping tables don't end up in
  `preservedTables`.
- `src/tests/CoreTests.cpp`: `[ConnectionString]` regression coverage
  for the empty-fragment UB.
- `docs/sql-migrations.md`: rewritten Concurrency Control section to
  describe `SqlScopedLock` as the generic primitive (not migration-
  specific), including the lock-table rename note.
- `AGENT.md`, `.agent/architecture.md`: file map updated.

Full SQLite suite (594/595 — 1 pre-existing skip);
`test_dbtool.py --test-env=sqlite3` SUCCESS including the new
hard-reset step; `dbtool migrate / hard-reset / migrate` clean with
no spurious release-time warnings. CI: all 21 jobs green
(https://github.com/LASTRADA-Software/Lightweight/actions/runs/25632752036),
including the previously-hanging
`Windows Tests (MS SQL Server (LocalDB))` finishing in 2 m 03 s —
parity with master's 1 m 58 s.

API rename, no compat shim. Migration is mechanical
(`MigrationLock` → `SqlScopedLock { conn, "lightweight_migration" }`).
The vtable-export shape is preserved (formatter `MigrationLockOps()`
overrides delegate inline to free `*AdvisoryLockOps()` helpers, so
each formatter's vtable stays weak — no `LIGHTWEIGHT_API` retrofit on
the concrete formatter classes).

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart christianparpart force-pushed the feature/migrations-gui branch from 1722417 to 28321b2 Compare May 11, 2026 07:52
Adds dbtool-gui, a Qt 6 / QML companion to the dbtool CLI for browsing,
previewing, and applying migrations against a profile-managed connection.

  - src/tools/dbtool-gui/: AppController, MigrationRunner, BackupRunner,
    SqlQueryRunner, ThemeController, QmlProgressManager,
    SqlSyntaxHighlighter, QtKeychainBackend
  - List models: Migration, Profile, Release, OdbcDataSource, SqlResult
  - QML views: Main, ConnectionPanel, MigrationView, SimpleView,
    ExpertView, ActionsPanel, BackupRestoreDialog, SqlQueryPanel,
    SqlPreviewDialog, LogPanel, BottomPanel, ToolBar, Theme,
    Card / StatusCard / StatusPill / FilterTabs / ReleaseGroup /
    ReleasesSummary / MigrationRow / KineticListView /
    WheelScrollAmplifier / TimestampAutocomplete / PluginsDirField /
    BulkControls
  - tools_shared/Odbc/DataSourceEnumerator: enumerate installed ODBC DSNs
    (consumed by the GUI's connection panel; lives in the tools-private
    static lib introduced by the previous commit)
  - cmake/FindQt.cmake: probe common Qt 6 install locations
  - LIGHTWEIGHT_BUILD_GUI option in top-level CMakeLists.txt with
    graceful downgrade when Qt is unavailable
  - docs/migrations-gui-plan.md, docs/migrations-gui-mockup.html

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart christianparpart force-pushed the feature/migrations-gui branch from 28321b2 to 804864e Compare May 11, 2026 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI pipeline CLI command line interface tools CMake Core API documentation Improvements or additions to documentation Query Builder Query Formatter SQL dialect implementations tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants