Skip to content

Migrating data from Kopano Groupware

Attention

This feature is disabled by default and requires configuration by the system administrator.

A migration tool is available to import data from a Kopano Groupware installation. The import can only be performed using trusted sources, verified by their IP address.

Requirements

Attention

The migration tool does not support partial or incremental migrations. Repeated migrations are also not recommended, as this will lead to duplicate items

The migration tool has the following requirements:

  1. Target users must exist
    1. Can either be
      1. Manually created
      2. Synchronised via ad-connect
    2. But must be set inactive to ensure the mailbox is not in use while importing
  2. Source users must either be
    1. Fully qualified user principal names (i.e. have a domain)
    2. Or the mapping configuration on import must provide the extra information
  3. Read access to the Kopano Groupware database (MySQL or MariaDB)
  4. Read access to the Kopano Groupware attachment storage directory (either directly or via network mount)
  5. Access to the Exchange4all server importer endpoint (source IP must be whitelisted)
  6. `kgdump tool supports Kopano Groupware database schema database revision 63 or higher
    1. This corresponds to the first public release of Kopano Groupware (8.0) and also includes Zarafa Groupware 7.1
  7. kgdump tool directly operates on the database level and will therefore process messages marked as deleted, but not yet purged (so called soft deleted messages)
    1. It is recommended to purge soft deleted messages before starting the export
      1. e.g. by running kopano-admin --purge-softdelete 0
Tip

It is recommended that you have a fast dedicated host, on which you can run both the dump and the migration. This in order to achieve the best performance when migrating large datasets, as this avoids using CPU resources on both the source or target systems.

A suitable migration system should at least have:

  1. 8 CPU cores

  2. 16 GiB of RAM

The disk space requirements depends on the amount of data being transferred. If dumping with concurrency is set to 1, the maximum amount of temporary disk space is used, for the dump.

Server side configuration

To add an IP to the import whitelist, simply add it to list stored in /storage/config.yaml. The format is as follows:

e4a:
  adminrpc:
    importer:
      ip_white_lists:
        - 198.51.100.5
        - 203.0.113.19

Downloading the migration tool

The latest version of the tool is always bundled with the container. It can be downloaded by navigating to https://your-server/download/service/kg-migration/e4a-kg-migration-latest-linux-amd64.tar.gz. Additional documentation and example configurations can be found inside the archive.

Examples for running the import tools

The migration tool consists of two components, kg-dump and migrate.

  1. The kg-dump tool connects to the Kopano database and its attachments, and generates an export stream.
  2. The migrate tool opens the generated export stream, and maps it to the corresponding user, forwarding the data to the new server (using a TLS connection).

Since the dump and import can take a while, it is recommended to run it inside a tool like screen or tmux. These tools take care of logging, and keep the migration running even if the terminal session unexpectedly closes.

Important

The importer operates in dry run mode by default, meaning no changes are made to your data. To execute an actual migration, set .settings.dry_run to false in your config.yml file.

Setup environment

It is recommended to set up the import environment using an .env file in a dedicated directory.

An example of this is shown below:

FOLDER=migrate-$(date -I)
mkdir "$FOLDER" \
&& cat <<EOF > "$FOLDER/.env"
export LOG_LEVEL=info
export DUMP_MYSQL_DSN=mysql://localhost/kopano?socket=/tmp/mysql.sock
export DUMP_ATTACHMENT_STORAGE=files_v1-10-20
export DUMP_ATTACHMENT_STORAGE_URI=file:///kopano/attachments
export IMPORTER_CONFIG=migrate-import-config.yaml
export IMPORTER_IMPORT_URL=https://exchange4all.local/e4a.importer_d1.v1.importer/import
EOF
cd "$FOLDER" && source ./.env

Different scenarios for exporting data

There are three scenarios in which you can dump data. While there are multiple scenarios, each come with their own advantages, and disadvantages.

Important note

When checking which situations fits to yours, please ensure you have read through the examples carefully and understand what is being done. Migrating data is a process in which data can be lost and not recovered. While the tool prevents this from happening, inputting or changing the wrong command could lead to data loss.

Dump and directly pipe into import

The example below assumes that the system on which the migration tools are running, is able to reach both the source database and the target endpoint. The example below also assumes that the migration tools are running all on the same host.

screen -L -Logfile kg-e4a-migrate-$(date -I'second').log -dmS kg-e4a-migrate sh -c "../bin/kgdump dump --stdout | ../bin/migrate import --stdin"

Dump and directly pipe into remote import through SSH

The example below is the recommended way to dump and pipe the data, if there is no machine that is able to access both the source and the target. For this example to work, the migrate tools and its configuration YAML file must be available on the SSH_TARGET host.

SSH_TARGET=target.url
screen -L -Logfile kg-e4a-migrate-$(date -I'second').log -dmS kg-e4a-migrate sh -c "../bin/kgdump dump --stdout | ssh ${SSH_TARGET} env DEBUG_LEVEL=${DEBUG_LEVEL} IMPORTER_CONFIG=${IMPORTER_CONFIG} IMPORTER_IMPORT_URL=${IMPORTER_IMPORT_URL} ../bin/migrate import --stdin"

Dump into file, transfer file, import from file

The example below is not recommended for large datasets, as it will take twice as long, while requiring a lot of disk space to store the dumped data. In some cases it may be possible to transfer the dumped data onto a removable storage device. This would require the user to plug the storage device into the source server, and follow the 4-step process of dumping and unplugging between the source and target servers.

First dump the data into a file: 

DUMPFILE=kgdump-yourserver-$(date -I'seconds').d1
../bin/kgdump dump --output="${DUMPFILE}"

Then transfer the resulting file to the target system. After the data has been transferred, run the importer locally:

../bin/migrate import --input="${DUMPFILE}"

Migration aftercare

Once the data has been imported, the user can be switched back to active, which activating the mailbox.

There are certain MAPI properties, which will lead to issues with Outlooks cached mode. To clean up these properties the admin script clear-unregistered-named-properties can be used:

e4a service admin clear-unregistered-named-properties


    [ConsoleKit\ConsoleException]                                                                                                
    Usage: clean-unregistered-named-properties [--all|<username>] [--organization-id=<organization ID>] [--maildir] [--dry-run]

Errata

Known issues and improvements:

  • #39 Export filters from Kopano Groupware can be confusing when exporting. Stores can be selected via user IDs. While importing the store id is used for all filtering and selecting.
  • #46 Migration of "rules" is not implemented.
  • #47 Migration of "ACLs" and "permissions" is not implemented.
  • #48 Migration of "Out of Office settings" is not implemented.
  • #49 Migration of "Email Signatures" is not implemented.
  • #64, #67, #69 References to users from the global addressbook are converted to SMTP Email address references.
  • #66 Some old Meeting requests in the "Sent" folder cannot be opened.
  • #71 The only implemented way to import attachment data is by including them in the export directly.
  • #72 The importer is not limited to the user mailbox quota, means the mailbox might be full and the quota needs to be increased by an admin to make such a mailbox fully usable.
  • #73 After importing the log output "size" field does not include attachment size and the resulting mailbox size might be way larger because of its attachments.
  • #75 Imported message size is taken as is from the source system and can change slightly whenever the imported message is changed as that computes the message size.
  • #479 The memory consumption while importing is relational to the total numer of attachments. Millions of attachments in the imported data can require multiple gigabytes of RAM until the import is finished.
  • #480 The importer has a message class white list and logs "message class not supported yet" in the server's log if a message is skipped.