Provide prebuilt native libraries by NuGet Packages. by ha-ves · Pull Request #257 · NetCordDev/NetCord

ha-ves · 2026-02-09T21:51:02Z

Summary

Continues from #245. Implements CI-driven distribution of prebuilt native binary dependencies.

Download size: ~112 MB
Extracted size: ~429 MB

Supported Platforms & Build Variants

Platform	RID	NativeAOT Support	Built	CI Test
Windows x64	`win-x64`	✓ (Static CRT /MT)	☑️	✅
Windows ARM64	`win-arm64`	✓ (Static CRT /MT)	☑️	✅
Linux x64	`linux-x64`	✓ (Dynamic CRT)	☑️	✅
Linux ARM64	`linux-arm64`	✓ (Dynamic CRT)	☑️	✅
macOS x64	`osx-x64`	✓ (Dynamic CRT)	☑️	⬜
macOS ARM64	`osx-arm64`	✓ (Dynamic CRT)	☑️	⬜

Need to Follow-Up

DOCS: instructions for referencing prebuilt packages in projects
DOCS: fallback build guide for unsupported platforms or custom RIDs
FIX MACOS CI TEST:

dotnet test Tests/NetCord.Natives.Tests/ -tl:off -clp:NoSummary,Verbosity=normal -v normal -bl

and then use https://github.com/JanKrivanek/MSBuildStructuredLog

Overview

This PR provides prebuilt native libraries via NuGet as per-RID packages. Each package bundles:

Runtime binaries for standard .NET consumption
Static libraries for NativeAOT compilation
MSBuild integration to transparently resolve native paths based on target platform
Bundled licenses for all included dependencies

Packages are published automatically on successful CI builds to the configured NuGet feed.

Structure & Packaging Model

Per-RID Distribution Model:

Packages are produced per target RID (local or CI builds)
Package ID format: NetCord.Natives.{win|linux|osx}-{x64|arm64}
Example: NetCord.Natives.linux-x64, NetCord.Natives.win-arm64

Package Contents (per-RID package):

runtimes/{rid}/native/              ← Runtime binaries for standard .NET
staticlibs/{rid}/                   ← Static libraries for NativeAOT builds
build/NetCord.Natives.{rid}.*       ← MSBuild props/targets per platform
licenses/                           ← Copyright files from vcpkg ports

What Changed

CI Pruning:

.github/workflows/build.yml & .github/workflows/build-and-publish.yml: Remove redundancies and make lean
.github/workflows/build-natives.yml: Parallel multi-platform CI matrix (6 RIDs), artifact caching, NuGet publishing pipeline

Core Build & Packaging

NetCord.Natives.csproj: VCPKG build and use integration project
Custom vcpkg ports (natives-ports/):
- libdave/portfile.cmake — discord/libdave with MSVC ARM64 patch
- mlspp/portfile.cmake — cisco/mlspp, libdave's dependency

NuGet Package build files

NetCord.Natives.props: dynamic library exclusion list & staticlib directory resolution via $(RuntimeIdentifier)
NetCord.Natives.targets: NativeAOT static library linking, dynamic library exclusion

Accessible Metadata

NativesHelper.cs: NativeLibraryVersionAttribute for runtime version discovery of bundled native libs

How to Use

CLI:

dotnet add package NetCord.Natives.win-arm64 --version 1.0.0

Reference the package for your target platform:

<ItemGroup>
  <PackageReference Include="NetCord.Natives.win-x64" Version="1.0.0" />
</ItemGroup>

NativeAOT with Exclusion

When populating <DirectPInvoke> for <PublishAot>, static libraries are automatically linked.

You can also exclude some library if you use another external build:

<ItemGroup>
  <PackageReference Include="NetCord.Natives.linux-x64" Version="1.0.0" />
  <DirectPInvoke Include="libdave" />
</ItemGroup>
<PropertyGroup>
  <PublishAot>true</PublishAot>
  <RuntimeIdentifier>linux-x64</RuntimeIdentifier>

  <NetCordExcludeNatives>opus</NetCordExcludeNatives>
</PropertyGroup>

Multi-Target Build with Per-RID Packages

For projects targeting multiple platforms with minimal per-platform size:
(Condition is arbitrary, each package will not conflict)

<ItemGroup Condition="'$(RuntimeIdentifier)' == 'win-x64'">
  <PackageReference Include="NetCord.Natives.win-x64" Version="1.0.0" />
</ItemGroup>
<ItemGroup Condition="'$(RuntimeIdentifier)' == 'linux-x64'">
  <PackageReference Include="NetCord.Natives.linux-x64" Version="1.0.0" />
</ItemGroup>

Maintainer Notes: `vcpkg` Baseline & Dependency Pinning

All native dependencies are pinned to a specific vcpkg baseline (vcpkg.json) to ensure reproducible builds.

Current constraint: openssl==3.0.7

When upgrading native library versions:

Update version constraints in vcpkg.json
Run local vcpkg build to validate against all RID triplets
Verify if licenses are still viable

Resource requirements:

Build Runner Availability: On github hosted runner, they run for up to ~30 min

Testing

Comprehensive native library validation via Tests/NetCord.Natives.Tests/:

Unit Tests (NativesBuildTests.cs)

Framework: MSTest with method-level parallelization
Target: .NET 10.0

Should always pass-in <RuntimeIdentifier> so cross-compiling dotnet could be done

`NativeLoaded` — Runtime Library Resolution

[TestMethod]
[DataRow("libdave")]
[DataRow("libsodium")]
[DataRow("opus")]
[DataRow("zstd")]
public void NativeLoaded(string libName)

Validates each of the four core native libraries can be successfully loaded via NativeLibrary.Load()
Catches load failures, missing binaries, or corrupted artifacts

`AllLibraryImportsExistInBinary` — P/Invoke Symbol Verification

[TestMethod]
[DataRow("libdave", "NetCord.Gateway.Voice.Dave")]
[DataRow("libsodium", "NetCord.Gateway.Voice.Encryption.XChaCha20Poly1305")]
[DataRow("opus", "NetCord.Gateway.Voice.Opus")]
[DataRow("zstd", "NetCord.Gateway.Compression.Zstandard")]
public void AllLibraryImportsExistInBinary(string libName, string className)

Reflects over managed P/Invoke wrapper classes to extract all [LibraryImport] entry points
Loads each native binary and validates every exported symbol exists via NativeLibrary.TryGetExport()
Catches missing entry points or mismatched P/Invoke declarations

`NativeAotStaticLinking` — NativeAOT Full Integration

[TestMethod]
[DataRow("libdave;libsodium;opus;zstd")]
public void NativeAotStaticLinking(string libName)

Build NativeAot: Invokes dotnet publish -c Release on NativeAotApp with <PublishAot> enabled:
- Passes current RID (e.g., win-x64, linux-x64, osx-arm64)
- Links static libraries via <DirectPInvoke>
Runtime validation: Executes the published AOT app:
- All native calls are valid
- Exit code is 0
- Logs build and run output for debugging

Included in This PR

✓ CI multi-platform native package build and publish
✓ Per-RID targeted package distribution
✓ Direct integration usage
✓ Custom vcpkg overlay ports for libdave and mlspp &
✓ Dependencies pinning
✓ Basic Unit & Integration tests

github-actions · 2026-05-05T22:22:38Z

The documentation preview is available at https://preview.netcord.dev/257.

AraHaan · 2026-05-06T01:21:01Z

I feel like the best option is to do something like:
NetCord.Natives.<rid> that way only the target RID's static libraries are downloaded and used for NAOT and does not slow down the build with extracting unused native static libraries.

* put into nuget package

* Add native ports for libdave and mlspp * Update build workflows for native packaging * Add local targets and CI helpers for natives * Update README

- GitHub Actions tag triggers, NuGet publish job - NativesHelper, NativeLibraryVersionAttribute - vcpkg integration, static linking - zstd support - native library tests, import checks - Native AOT test app - NetCordNativesDir assembly metadata

* Add MSVC builtin add overflow patch for libdave * Add vcpkg-non-windows.targets for cross-platform support * Update libdave and mlspp portfiles with build fixes * Update build-natives workflow and vcpkg.json versions * Fix vcpkg dependency resolution for multi-platform builds

* Add dynamic library exclusion for AOT builds * Add NativeAotApp test project

* natives conditionals fix * fix nativeaot test structure * add natives tests to build workflow

- add reusable vcpkg setup - pack per-RID native nupkgs - streamline NativeAOT test logging - tighten native linking targets

ha-ves · 2026-05-08T19:53:11Z

@AraHaan by default CI workflow will provide per-RID packages now.

But by .csproj design it will also work If anyone for some reason still needs to package all in one.

KubaZ2

I haven't went through all the MSBuild stuff. I kind of hate that it takes so much and I am afraid it may require changes when new versions of binaries get released. I guess there is no better way to handle that? I think most binaries just require a single make or 2 cmake commands to build, but I think most of the MSBuild stuff is just for copying the binaries in the correct place and licensing?

KubaZ2 · 2026-05-08T22:57:02Z

+    [DoNotParallelize]
+    [TestMethod]
+    [DataRow("libdave;libsodium;opus;zstd")]
+    public void NativeAotStaticLinking(string libName)


Wouldn't it be better to make tests that are built with static linking instead of building a static linking app in the tests?

~~yeah, NativeAotApp can be pulled to a separate static-linked test .csproj~~

I can't see how it'll work without putting more stuff than the test method,
including the new test platform is early preview.

KubaZ2 · 2026-05-08T23:02:37Z

+diff --git a/cpp/src/frame_processors.cpp b/cpp/src/frame_processors.cpp
+index 37c8d70..5f0460c 100644
+--- a/cpp/src/frame_processors.cpp
+++ b/cpp/src/frame_processors.cpp
+@@ -13,6 +13,9 @@
+
+ #if defined(_MSC_VER)
+ #include <intrin.h>
+#if defined(_M_ARM64)
+#include <arm64intr.h>
+#endif
+ #endif
+
+ namespace discord {
+@@ -25,6 +28,9 @@ std::pair<bool, size_t> OverflowAdd(size_t a, size_t b)
+     bool didOverflow = _addcarry_u64(0, a, b, &res);
+ #elif defined(_MSC_VER) && defined(_M_IX86)
+     bool didOverflow = _addcarry_u32(0, a, b, &res);
+#elif defined(_MSC_VER) && defined(_M_ARM64)
+    res = a + b;
+    bool didOverflow = res < a;
+ #else
+     bool didOverflow = __builtin_add_overflow(a, b, &res);
+ #endif


Was it reported to libdave maintainers? A patch may require additional maintenance in the future.

this is an out-of-support patch for the current pinned libdave version,

No one asked them for win-arm64 yet,
in the meantime you could provide this runtimeId unsupported.

KubaZ2 · 2026-05-08T23:04:35Z

+vcpkg_from_github(
+    OUT_SOURCE_PATH SOURCE_PATH
+    REPO cisco/mlspp
+    REF "${VERSION}"
+    SHA512 5d37631e2c47daae1133ef074e60cc09ca2d395f9e11c416f829060e374051cf219d2d7fe98dae49d1d045292e07d6a09f4814a5f16e6cc05e67e7cd96f146c4
+)
+
+if(VCPKG_TARGET_IS_OSX AND EXISTS "/usr/local/include/openssl/")
+    set(VCPKG_INCLUDE_OVERRIDE "-DCMAKE_CXX_FLAGS=-I${CURRENT_INSTALLED_DIR}/include")
+endif()
+
+set(VCPKG_LIBRARY_LINKAGE static)
+
+vcpkg_cmake_configure(
+    SOURCE_PATH "${SOURCE_PATH}"
+    OPTIONS
+        ${VCPKG_INCLUDE_OVERRIDE}
+        -DDISABLE_GREASE=ON
+        -DTESTING=OFF
+        -DBUILD_TESTING=OFF
+        -DMLS_CXX_NAMESPACE="mlspp"
+    MAYBE_UNUSED_VARIABLES
+        BUILD_TESTING
+)
+
+vcpkg_cmake_install()
+vcpkg_copy_pdbs()
+
+vcpkg_cmake_config_fixup(PACKAGE_NAME "MLSPP" CONFIG_PATH "share/MLSPP")
+
+# Remove redundant debug directories to comply with vcpkg policy
+file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/debug/include")
+file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/debug/share")


Why is a port needed for mlspp?

cisco doesn't "port" (submit) it to vcpkg registry, this is taken from libdave's port of it.

KubaZ2 · 2026-05-08T23:07:24Z

+using System;
+using System.Collections.Generic;
+using System.Reflection;
+
+namespace NetCord.Natives;
+
+[AttributeUsage(AttributeTargets.Assembly, AllowMultiple = true)]
+public class NativeLibraryVersionAttribute(string name, string version) : Attribute
+{
+    public string Name { get; } = name;
+    public string Version { get; } = version;
+}
+
+public static class NativesHelper
+{
+    public static IEnumerable<NativeLibraryVersionAttribute> GetNativeLibraryVersions()
+    {
+        return typeof(NativesHelper).Assembly.GetCustomAttributes<NativeLibraryVersionAttribute>();
+    }
+}


I don't think I like this. I think a native package shouldn't come with any managed code.

true, but you'd like to know which nativelib version it was when issues arise.
Probably put the versions in the package description instead

KubaZ2 · 2026-05-08T23:10:35Z

+<Solution>
+  <Folder Name="/Solution Items/">
+    <File Path=".editorconfig" />
+    <File Path="Directory.Build.props" />
+    <File Path="Directory.Build.targets" />
+    <File Path="Directory.Packages.props" />
+    <File Path="global.json" />
+  </Folder>
+  <Folder Name="/SourceGenerators/">
+    <File Path="SourceGenerators/Directory.Build.props" />
+    <File Path="SourceGenerators/Directory.Build.targets" />
+    <Project Path="SourceGenerators/MethodsForPropertiesGenerator/MethodsForPropertiesGenerator.csproj" />
+    <Project Path="SourceGenerators/RestClientMethodAliasesGenerator/RestClientMethodAliasesGenerator.csproj" />
+    <Project Path="SourceGenerators/ShardedGatewayClientEventsGenerator/ShardedGatewayClientEventsGenerator.csproj" />
+    <Project Path="SourceGenerators/Shared/Shared.csproj" />
+    <Project Path="SourceGenerators/UserAgentHeaderGenerator/UserAgentHeaderGenerator.csproj" />
+    <Project Path="SourceGenerators/WebSocketClientEventsGenerator/WebSocketClientEventsGenerator.csproj" />
+  </Folder>
+  <Folder Name="/Tests/">
+    <File Path="Tests/Directory.Build.props" />
+    <File Path="Tests/Directory.Build.targets" />
+    <Project Path="Tests/NetCord.Natives.Tests/NetCord.Natives.Tests.csproj" />
+    <Project Path="Tests/NetCord.Natives.Tests/Assets/NativeAotApp/NativeAotApp.csproj" />
+  </Folder>
+  <Project Path="NetCord.Natives/NetCord.Natives.csproj" />
+  <Project Path="NetCord/NetCord.csproj" />
+</Solution>


What is the idea behind a separate solution? Is it build times? Also I doubt this solution needs these source generators.

the source generators used by main NetCord, which gets referenced by NetCord.Natives.Tests.

I haven't finished wiring up the other tests how to get the nativelibs, in CI workflow should be pretty straight forward, restore cache, NetCord.slnx build will be quick.

local build from source will be very long on first-time NetCord.slnx build unless they skip the natives and tests requiring it.

KubaZ2 · 2026-05-08T23:13:09Z

+    <PackageReference Include="GitHubActionsTestLogger" />
+    <PackageVersion Include="GitHubActionsTestLogger" Version="3.0.4" />


I think versions should be managed by Directory.Packages.props like all other packages. Also I am not sure this is needed here. Maybe this package could be added to all tests at once, ideally in a separate PR.

it was testing artifact, though this package is very good for CI.

since this PR also (/will) modify the build workflow,
yeah I think it will still be pretty related to be included in Directory.Packages.props

KubaZ2 · 2026-05-08T23:25:19Z

+      - name: Upload NuGet Package Artifacts
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
-          name: Build Artifacts
+          name: NuGet Packages
          path: |
-            NetCord/bin/Release
-            NetCord.Services/bin/Release
-            Hosting/NetCord.Hosting/bin/Release
-            Hosting/NetCord.Hosting.Services/bin/Release
-            Hosting/NetCord.Hosting.AspNetCore/bin/Release
+            NetCord/bin/Release/*.nupkg
+            NetCord.Services/bin/Release/*.nupkg
+            Hosting/NetCord.Hosting/bin/Release/*.nupkg
+            Hosting/NetCord.Hosting.Services/bin/Release/*.nupkg
+            Hosting/NetCord.Hosting.AspNetCore/bin/Release/*.nupkg


I think snupkgs should also be uploaded there.

ha-ves · 2026-05-09T03:35:52Z

@KubaZ2
I haven't went through all the MSBuild stuff. I kind of hate that it takes so much and I am afraid it may require changes when new versions of binaries get released. I guess there is no better way to handle that? I think most binaries just require a single make or 2 cmake commands to build, but I think most of the MSBuild stuff is just for copying the binaries in the correct place and licensing?

Vcpkg is the least time-consuming dependency manager, it can build and put it all together in a convenient path.

The previous version of this PR was just using CMake, but it won't get the dependencies for your NativeAOT.
And it gets many times more complicated than this vcpkg version.

you can refer to this when there's new version:

When upgrading native library versions:

Update version constraints in vcpkg.json (or natives-port// and target specific version and patch)

Run local vcpkg build to validate against all RID triplets

Verify if licenses are still viable

ha-ves force-pushed the feature/prebuilt-natives branch 2 times, most recently from 1c05c69 to 6565a35 Compare May 5, 2026 12:47

ha-ves added 9 commits May 9, 2026 01:32

Pivot to use vcpkg to build natives

4b3d111

* put into nuget package

* Add NetCord.Natives project and vcpkg support

48bfd5b

* Add native ports for libdave and mlspp * Update build workflows for native packaging * Add local targets and CI helpers for natives * Update README

feat: native library support and testing

f0030a5

- GitHub Actions tag triggers, NuGet publish job - NativesHelper, NativeLibraryVersionAttribute - vcpkg integration, static linking - zstd support - native library tests, import checks - Native AOT test app - NetCordNativesDir assembly metadata

* Refactor native library handling and packaging

aa317d0

* Add dynamic library exclusion for AOT builds * Add NativeAotApp test project

CI finished & fix typos & guarding

da37fff

solution build fix

a883480

separate solution for natives

498226a

* natives conditionals fix * fix nativeaot test structure * add natives tests to build workflow

refactor native build/test pipeline

ee00a64

- add reusable vcpkg setup - pack per-RID native nupkgs - streamline NativeAOT test logging - tighten native linking targets

ha-ves force-pushed the feature/prebuilt-natives branch from 66a1e8d to ee00a64 Compare May 8, 2026 18:10

KubaZ2 reviewed May 8, 2026

View reviewed changes

		<PackageReference Include="GitHubActionsTestLogger" />
		<PackageVersion Include="GitHubActionsTestLogger" Version="3.0.4" />

Uh oh!

Conversation

ha-ves commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Supported Platforms & Build Variants

Need to Follow-Up

Overview

Structure & Packaging Model

What Changed

CI Pruning:

Core Build & Packaging

NuGet Package build files

Accessible Metadata

How to Use

NativeAOT with Exclusion

Multi-Target Build with Per-RID Packages

Maintainer Notes: vcpkg Baseline & Dependency Pinning

When upgrading native library versions:

Resource requirements:

Testing

Unit Tests (NativesBuildTests.cs)

Should always pass-in <RuntimeIdentifier> so cross-compiling dotnet could be done

NativeLoaded — Runtime Library Resolution

AllLibraryImportsExistInBinary — P/Invoke Symbol Verification

NativeAotStaticLinking — NativeAOT Full Integration

Included in This PR

Uh oh!

github-actions Bot commented May 5, 2026

Uh oh!

AraHaan commented May 6, 2026

Uh oh!

ha-ves commented May 8, 2026

Uh oh!

KubaZ2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ha-ves May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ha-ves commented May 9, 2026

When upgrading native library versions:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ha-ves commented Feb 9, 2026 •

edited

Loading

Maintainer Notes: `vcpkg` Baseline & Dependency Pinning

Should always pass-in `<RuntimeIdentifier>` so cross-compiling dotnet could be done

`NativeLoaded` — Runtime Library Resolution

`AllLibraryImportsExistInBinary` — P/Invoke Symbol Verification

`NativeAotStaticLinking` — NativeAOT Full Integration

ha-ves May 9, 2026 •

edited

Loading