Skip to content

stalwart-migrator

This page helps you upgrade a Stalwart mail server from 0.15.5 to 0.16, the version ihasmail needs. It covers what to fix first, how to try it safely, how to run it, and what to check afterwards.

stalwart-migrator is a command-line tool that does the upgrade in place. It checks your server before it starts, saves its progress at every step so an interrupted run can carry on, and tests the server when it is done.

The upgrade cannot be undone. Take a snapshot first.

Upgrading changes how Stalwart stores your data, and there is no way back except restoring a snapshot or backup you took beforehand. stalwart-migrator does not take one, cannot check yours, and has no rollback command.

Before you run it, take a snapshot you know how to restore — a ZFS, LVM or btrfs snapshot, a virtual machine snapshot, or a backup you have restored before. The tool refuses to start until you say you have one.

Already on 0.16? You don't need this page. Go back to Installing ihasmail.

What it does

A migration runs these steps in order, and saves a checkpoint after each one:

  1. Preflight: read-only checks, and a plan of what will happen.
  2. Stage: fetches the new Stalwart and Stalwart's own conversion script.
  3. Dump: saves your settings and accounts from the running server.
  4. Stop the server.
  5. Convert your settings to the 0.16 format.
  6. Recovery mode: starts 0.16 once in a special mode and loads your settings into it.
  7. Cutover: points your service at the new version and starts it.
  8. Validate: checks that every account that existed before still exists.

In real migrations the server was down for 6 to 8 seconds. Plan your maintenance window around checking the result, not around the size of your mail.

Why not just follow Stalwart's upgrade notes by hand?

0.15 to 0.16 is not a simple swap of one program for another. Settings move from files into the data store, the data has to be converted rather than copied, and the conversion needs two programs running at once in a particular order.

On its first normal start, 0.16 also permanently deletes the old account and settings records it no longer understands. If they were not captured first, they are gone. A half-finished migration with no way back is the failure that matters, so checkpoints and the snapshot question are the center of the tool, not extras.

Has this been used for real?

Yes. It took a production server with nine domains, six accounts and a 2.4 GB store from 0.15.5 to 0.16.19 with 8 seconds of downtime, after a full rehearsal on a copy of the same server.

Another operator reported three successful 0.15.5 to 0.16.19 migrations: two servers with 16 domains, 55 accounts and about 221 GB of mail each, and a small arm64 home server. They did the cutover step by hand, because they used a version from before it was automated. Their report, including the problems below, is issue #1.

Docker containers are less proven. The container path has never completed a migration against a real Stalwart image. See Docker deployments.

Before you start

What you need

  • Stalwart 0.15.5, running as a systemd service or a single Docker container.
  • Root access on the mail server. The tool records its progress in /var/lib/stalwart-migrator/runs unless you pass --state-dir.
  • python3, which Stalwart's conversion script needs.
  • stalwart-cli version 1.0.2 or later. It is a separate download from the server itself.
  • An administrator account in Stalwart's directory, and the address of its admin API, such as https://mail.example.com. See below: the built-in admin from config.toml will not do.
  • A snapshot or backup you have checked you can restore.

Install the tool

Download the tool for your server's architecture, check it, and put it on your path:

ARCH=amd64   # or arm64
curl -fsSLO https://github.com/Coffey-Labs/stalwart-migrator/releases/latest/download/stalwart-migrate-linux-$ARCH.tar.gz
curl -fsSLO https://github.com/Coffey-Labs/stalwart-migrator/releases/latest/download/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar -xzf stalwart-migrate-linux-$ARCH.tar.gz
sudo install -m 0755 stalwart-migrate /usr/local/bin/
stalwart-migrate version

sha256sum should print OK for your file. If it doesn't, don't use the download.

Every command below asks for the admin password. Give it with --admin-password, or set STALWART_MIGRATE_ADMIN_PASSWORD so it stays out of your shell history.

Fix two things on the server first

Both of these stop a migration, and neither is something the tool can fix for you. preflight checks for both while your server is still running, but fixing them means changing your accounts, so find out before you book a maintenance window.

1. Untangle shared tenants. A tenant is a separate organization on the same server. 0.16 requires an account in a tenant to use only domains that belong to that tenant. 0.15 had no such rule. See who owns what:

stalwart-migrate tenants --admin-url https://mail.example.com --admin-user [email protected]
  • If a domain has no tenant of its own and only one tenant's accounts use it, the conversion fixes it for you.
  • If two tenants really share a domain, nothing can fix it. Sort it out on 0.15 first: give each tenant its own domains, move the accounts into one tenant, or remove the tenants.

2. Use a real administrator account. If your only administrator is the built-in [authentication.fallback-admin] from config.toml — which is what stalwart --init sets up — you will be locked out after the migration. Create an account in Stalwart's directory, give it the admin role, and check you can sign in with it before you migrate. Then make sure that:

  • its name is unique. If [email protected] and [email protected] both exist, the tool will not guess which one to make an administrator. You will have to grant the role by hand afterwards.
  • its admin rights don't come only from tenant-admin. That role has no 0.16 equivalent and is not carried over.

preflight refuses to go on if the account you give it is not in the directory.

Why the built-in admin disappears
  • 0.16 keeps its settings in the data store, not in a file. After the migration, the server's config file only says where the data is. The old config.toml, and the admin defined in it, is never read again. The migration itself still succeeds; you just can't check it or manage the server afterwards.
  • Stalwart's converter gives every account the User role, whatever it had before. rehearse works out how to give your administrator its role back, but only for an account that exists in the directory.
  • Preflight cannot know which roles the converter will keep, so check on a copy of your server, or straight after the migration, that the account can still manage the server.

Step 1: Check the server

preflight only reads from your server. It is safe to run any time.

sudo stalwart-migrate preflight --admin-url https://mail.example.com --admin-user [email protected]

It prints every problem it finds and a plan of what a migration would do. It does need write access to save its checkpoint, so run it as root.

Step 2: Rehearse

rehearse shows what the upgrade will and won't carry over, without changing anything. It copies no data, starts no server and never writes to the store, so you can run it on your live server as often as you like.

sudo stalwart-migrate rehearse --admin-url https://mail.example.com \
    --admin-user [email protected] --target 0.16.14

--target is the Stalwart version to move to. Leave it out for the newest release.

It reports two things:

  • The plan: the settings that will carry over.
  • The worklist: the settings that won't. It is long, but mostly not work, and rehearse says which is which.

It also writes a supplemental plan for what it can rebuild for you — today, your network listeners, without which a migrated server answers on no ports at all. run builds and loads the same supplement for you. The outputs are kept in /var/lib/stalwart-migrator/runs/<run-id>/ as export.json, supplement.json and unmigrated.txt.

How much of the worklist needs you

On a real production server, Stalwart's converter carried 219 of 12,401 settings. Of the 12,182 it left:

  • 8,547 are automatic IP-ban records that rebuild themselves.
  • 3,337 are stock spam-filter and lookup data that 0.16 ships newer copies of.
  • About 224 were already carried another way, DKIM signatures included.
  • About 293 genuinely need a person to look at them.

The supplemental plan covered 24 of 3,505 keys on a test server, and the tool says exactly that rather than implying more.

Step 3: Rehearse on a copy of your server

rehearse converts your settings but never loads them, and loading them is where real migrations fail. A copy of your server closes that gap. This is the step we most recommend. On a 2.4 GB store, making the copy cost about six seconds of downtime.

  1. Copy the data directory with Stalwart stopped. A copy of a running store can be broken in ways your real one isn't. Stop the service, copy the data directory with cp -a to the same disk, start the service again.
  2. Cut the copy off from the internet. Otherwise a copy of a live mail server renews certificates for your real domains and sends whatever mail is waiting to go out. Check from inside the copy that it really has no route out.
  3. Run the real migration on the copy: preflight, then run. Because the copy can't download anything, point --target-binary, --stalwart-cli and --migration-script at copies you put there first.
  4. Snapshot the copy while it is shut down, so a failed attempt takes seconds to reset.

On a copy, two things will look broken and aren't: the 0.16 web interface at /account/ returns 404, because it can't be downloaded, and certificate renewal can't be tested.

More detail, including what rehearsing on a copy has caught: rehearsal notes.

Step 4: Take your snapshot

Take it with Stalwart stopped if you can. A snapshot of a running server usually recovers, but "usually" is not what you want for your mail.

Restoring any snapshot loses the mail that arrived after you took it. If that gap matters, consider holding incoming mail at a backup mail server for the length of the migration.

Step 5: Run the migration

sudo stalwart-migrate run --admin-url https://mail.example.com \
    --admin-user [email protected] --recovery-point-confirmed --yes
  • --recovery-point-confirmed says you have a snapshot you have checked you can restore. The tool cannot check it; the flag only makes sure nobody migrates a mail server without being asked.
  • --yes says you really mean it.

If the run is interrupted — a dropped SSH session, a crash — it carries on from the last completed step. sudo stalwart-migrate status <run-id> shows where it stopped, and run --resume <run-id> with the same flags continues it.

Docker deployments

A Docker container can be migrated, but this path has never completed a migration against a real Stalwart image. Its checks and commands are tested, and two problems have been found and fixed, but it is not proven. Rehearse on a copy first; that advice counts double here.

Add these to run:

  • --container-path-unproven, to say you know the above. run refuses a container without it.
  • --target-image, the new image named in full, such as stalwartlabs/stalwart:v0.16.14. The tool never guesses it from your current container.
  • --container, if your container isn't called stalwart.
  • --data-dir, the data path inside the container.

It refuses a container it can't rebuild faithfully, and tells you why during preflight:

  • one with settings it doesn't understand, such as extra capabilities, a custom network or device mappings;
  • one whose data is not on a volume;
  • one managed by Docker Compose. For Compose, change the image tag in your compose file and run docker compose up -d instead.

Your old container is renamed, not removed, and the old image is kept.

More detail: Docker deployments.

After the migration

run checks that every account that existed before still exists, and fails if one is missing. A domain that no longer appears is only a warning, because the two versions count domains differently. Print the result again any time with sudo stalwart-migrate report <run-id>.

Then check these yourself.

Check the certificate on the mail ports

After the upgrade, your certificate may work for the website on port 443 but not for IMAP and SMTP on ports 993 and 465, which keep a self-signed certificate. Connect to 993 or 465 and look at which certificate you get.

If it is the wrong one, set defaultCertificateId on SystemSettings to your certificate, and restart Stalwart. The tool doesn't do this for you yet.

Set up certificate renewal again

Stalwart's converter drops your ACME settings without saying so. ACME is how your certificate renews itself. Your current certificate carries over, so everything works — until it expires, about ninety days later.

Before migrating, look for acme.* settings in the dump. If there were any, create an AcmeProvider in 0.16 afterwards. Stalwart registers a new ACME account when it next issues a certificate; the old one can't be carried over.

If loading settings or the next start fails

Across three real migrations, two failures appeared that one more recovery-mode boot fixed. The tool does not do that extra boot yet, so it is a manual step:

  • The settings load fails on its first object with primaryKeyViolation. Starting 0.16 in recovery mode again and loading the same settings went straight through.
  • The next normal start stops with "Upgrading to version 0.16 is a multi-step process". Starting recovery mode once more, letting it come up, and stopping it cleanly was enough.

This is the best reason to rehearse on a copy: you find out whether your server needs it before it matters.

Only straight after the migration

This extra boot belongs to the migration itself. Once your server has migrated and started normally, never boot recovery mode again — see below.

Check your administrator can still manage the server

Sign in as your directory administrator and make a harmless change. If it is refused, the role was not carried over; grant it by hand.

If something goes wrong

Your snapshot is how you go back. A failed run stops and tells you what happened; a person decides what to restore. To help, the tool keeps:

  • the old Stalwart program, next to the new one as <binary>.v<old-version>;
  • your original service file, as <unit>.pre-<run-id>;
  • your settings and accounts dumps, and the plan that was loaded, in /var/lib/stalwart-migrator/runs/<run-id>/, whether the run succeeded or not;
  • a record of every step, which status <run-id> prints.

Restoring loses mail delivered since the snapshot.

Never boot recovery mode on a migrated server

Recovery mode is part of the migration, not a maintenance mode. An operator who started it again on servers that had migrated successfully days earlier found their domains and accounts gone on the next normal start. It happened twice, on two different servers.

What brought them back both times was loading that run's export.json and supplement.json again into a fresh recovery boot — which is why the tool keeps those files. To change something after a migration, use the admin API or stalwart-cli against the running server.

Why the tool doesn't restore for you

An earlier version did. It restored files, checked every one against a list, and reported success — but it did not restore file ownership. Run as root, it would have produced a perfect copy that Stalwart, running as its own user, could not open. A filesystem snapshot can't make that mistake, because it never lost the ownership in the first place.

Learn more

Then go back to Installing ihasmail.