Lightning Stream

User documentation can be found here

Lightning Stream is a tool to sync changes between a local LMDB (Lightning Memory-Mapped Database) and an S3 bucket in near real-time. If the application schema is compatible, this can be used in a multi-writer setup where any instance can update any data, with a global eventually-consistent view of the data in seconds.

Our main target application is the sync of LMDB databases in the PowerDNS Authoritative Nameserver (PDNS Auth). We are excited about how Lightning Stream simplifies running multiple distributed PowerDNS Authoritative servers, with full support for keeping DNSSEC keys in sync. Check the Getting Started section to understand how you can use Lightning Stream together with the PowerDNS Authoritative server.

Its use is not limited to the PowerDNS Authoritative server, however. Lightning Stream does not make any assumptions about the contents of the LMDB, and can be used to sync LMDBs for other applications, as long as the data is stored using a compatible schema.

Basic Operation

Lightning Stream is deployed next to an application that uses an LMDB for its data storage:

Its operation boils down to the following:

• Whenever it detects that the LMDB has changed, it writes a snapshot of the data to an S3 bucket.
• Whenever it sees a new snapshot written by a different instance in the S3 bucket, it downloads the snapshot and merges the data into the local LMDB.

The merge of a key is performed based on a per-record last-modified timestamp: the most recent version of the entry wins. Deleted entries are cleared and marked as deleted, together with their deletion timestamp. This allows Lightning Stream to provide Eventual Consistency across all nodes.

If the application uses a carefully designed data schema, this approach can be used to support multiple simultaneously active writers. In other instances, it can often be used to sync data from one writer to multiple read-only receivers. Or it can simply create a near real-time backup of a single instance.

Building

At the moment of writing, this project requires Go 1.19. Please check the go.mod file for the current version.

To install the binary in a given location, simply run:

GOBIN=$HOME/bin go install ./cmd/lightningstream/

Or run ./build.sh to install it in a bin/ subdirectory of this repo.

Easy cross compiling is not supported, because the LMDB bindings require CGo.

Example in Docker Compose

This repo includes an example of syncing the PowerDNS Authoritative Nameserver LMDB. It runs two DNS servers, each with their own syncer, syncing to a bucket in a MinIO server.

The Lightning Stream config used can be found in docker/pdns/lightningstream.yaml. Note that the config file contents can reference environment variables.

To get it up and running:

docker-compose up -d

You may need to rerun this command once, because of a race condition creating the LMDBs.

To see the services:

docker-compose ps

This should show output like:

         Name                        Command               State                                    Ports
-------------------------------------------------------------------------------------------------------------------------------------------
lightningstream_auth1_1   /run.sh                          Up      127.0.0.1:4751->53/tcp, 127.0.0.1:4751->53/udp, 127.0.0.1:4781->8081/tcp
lightningstream_auth2_1   /run.sh                          Up      127.0.0.1:4752->53/tcp, 127.0.0.1:4752->53/udp, 127.0.0.1:4782->8081/tcp
lightningstream_minio_1   /usr/bin/docker-entrypoint ...   Up      127.0.0.1:4730->9000/tcp, 127.0.0.1:4731->9001/tcp
lightningstream_sync1_1   /usr/local/bin/lightningst ...   Up      127.0.0.1:4791->8500/tcp
lightningstream_sync2_1   /usr/local/bin/lightningst ...   Up      127.0.0.1:4792->8500/tcp

Open one terminal with all the logs:

docker-compose logs

Then in another terminal call these convenience scripts, with a delay between them to allow for syncing:

docker/pdns/pdnsutil -i 1 create-zone example.org
docker/pdns/pdnsutil -i 1 secure-zone example.org
docker/pdns/pdnsutil -i 1 set-meta example.org foo bar
docker/pdns/pdnsutil -i 2 generate-tsig-key example123 hmac-sha512

sleep 2

docker/pdns/curl-api -i 2 /api/v1/servers/localhost/zones/example.org
docker/pdns/curl-api -i 2 /api/v1/servers/localhost/zones/example.org/metadata
docker/pdns/curl-api -i 1 /api/v1/servers/localhost/tsigkeys

To view a dump of the LMDB contents:

docker/pdns/dump-lmdb -i 1
docker/pdns/dump-lmdb -i 2

You can browse the snapshots in MinIO at http://localhost:4731/buckets/lightningstream/browse (login with minioadmin / minioadmin).

Using Azure Blob storage

Lightning Stream supports Azure Blob Storage as a backend for storing snapshots. You can configure it to use either static credentials (account name and key) or Azure service principal authentication.

Basic configuration with static credentials

storage:
  type: azure
  options:
    account_name: myaccountname
    account_key: myaccountkey
    use_shared_key: true
    container: lightningstream
    create_container: true

Configuration with Azure service principal (recommended for production)

When account_key is not set, the backend uses DefaultAzureCredential, which automatically picks up service principal credentials from environment variables:

• AZURE_CLIENT_ID
• AZURE_TENANT_ID
• AZURE_CLIENT_SECRET

storage:
  type: azure
  options:
    container: lightningstream
    endpoint_url: https://myaccount.blob.core.windows.net/
    create_container: true

Available options

Option	Type	Summary
account_name	string	Azure storage account name (required for shared key auth)
account_key	string	Azure storage account key
use_shared_key	bool	Use shared key (account name + key) authentication; if false, DefaultAzureCredential is used
container	string	Azure blob container name (required)
create_container	bool	Create container if it does not exist
endpoint_url	string	Custom endpoint URL (defaults to https://<account_name>.blob.core.windows.net/)
global_prefix	string	Transparently apply a global prefix to all blob names
disable_send_content_md5	bool	Disable sending the Content-MD5 header
tls	tlsconfig.Config	TLS configuration
init_timeout	duration	Time allowed for initialisation (default: "20s")
use_update_marker	bool	Reduce LIST commands by using an update marker (see below)
update_marker_force_list_interval	duration	Force full LIST sync at this interval (default: "5m")
concurrency	int	Max number of concurrent uploads (default: 1)

The use_update_marker option can reduce Azure costs by replacing LIST operations (which are more expensive) with GET operations. However, it cannot be used if the container itself is replicated in an active-active fashion between data centers.

You can see a working example in the docker-compose setup, which uses Azurite (Azure Storage Emulator):

docker-compose up

For all available Azure backend options with full descriptions, see Simpleblob's Azure backend Options struct.

Open Source

This is the documentation for the Open Source edition of Lightning Stream. For more information on how we provide support for Open Source products, please read our blog post on this topic.

PowerDNS also offers an Enterprise edition of Lightning Stream that includes professional support, advanced features, deployment tooling for large deployments, Kubernetes integration, and more.