Skip to content

Migrate data between different backends

Seafile supports data migration between filesystem, s3, ceph, swift and Alibaba oss by a built-in script. Before migration, you have to ensure that both S3 hosts can be accessed normally.

Migration to or from S3

Since version 11, when you migrate from S3 to other storage servers or from other storage servers to S3, you have to use V4 authentication protocol. This is because version 11 upgrades to Boto3 library, which fails to list objects from S3 when it's configured to use V2 authentication protocol.

Copy seafile.conf and use new S3 configurations

During the migration process, Seafile needs to know where the data will be migrated to. The easiest way is to copy the original seafile.conf to a new path, and then use the new S3 configurations in this file.

Warning

For deployment with Docker, the new seafile.conf has to be put in the persistent directory (e.g., /opt/seafile-data/seafile.conf) used by Seafile service. Otherwise the script cannot locate the new configurations file.

cp /opt/seafile-data/seafile/conf/seafile.conf /opt/seafile-data/seafile.conf

nano /opt/seafile-data/seafile.conf

cp /opt/seafile/conf/seafile.conf /opt/seafile.conf

nano /opt/seafile.conf

Then you can follow here to use the new S3 configurations in the new seafile.conf. By the way, if you want to migrate to a local file system, the new seafile.conf configurations for S3 example is as follows:

# ... other configurations

[commit_object_backend]
name = fs
dir = /var/data_backup

[fs_object_backend]
name = fs
dir = /var/data_backup

[block_backend]
name = fs
dir = /var/data_backup

Stop Seafile Server

Since the data migration process will not affect the operation of the Seafile service, if the original S3 data is operated during this process, the data may not be synchronized with the migrated data. Therefore, we recommend that you stop the Seafile service before executing the migration procedure.

docker exec -it seafile bash
cd /opt/seafile/seafile-server-latest
./seahub.sh stop
./seafile.sh stop

cd /opt/seafile/seafile-server-latest
./seahub.sh stop
./seafile.sh stop

Run migrate.sh to initially migrate objects

This step will migrate most of objects from the source storage to the destination storage. You don't need to stop Seafile service at this stage as it may take quite long time to finish. Since the service is not stopped, some new objects may be added to the source storage during migration. Those objects will be handled in the next step:

Speed-up migrating large number of objects

If you have millions of objects in the storage (especially the fs objects), it may take quite long time to migrate all objects and more than half is using to check whether an object exists in the destination storage. In this situation, you can modify the nworker and maxsize variables in the migrate.py:

class ThreadPool(object):
    def __init__(self, do_work, nworker=20):
            self.do_work = do_work
            self.nworker = nworker
            self.task_queue = Queue.Queue(maxsize = 2000)

However, if the two values (i.e., nworker and maxsize) ​​are too large, the improvement in data migration speed may not be obvious because the disk I/O bottleneck has been reached.

Encrypted storage backend data (deprecated)

If you have an encrypted storage backend, you can use this script to migrate and decrypt the data from that backend to a new one. You can add the --decrypt option in calling the script, which will decrypt the data while reading it, and then write the unencrypted data to the new backend:

./migrate.sh /opt --decrypt

# make sure you are in the container and in directory `/opt/seafile/seafile-server-latest`
./migrate.sh /shared

# exit container and stop it
exit
docker compose down

# make sure you are in the directory `/opt/seafile/seafile-server-latest`
./migrate.sh /opt

Success

You can see the following message if the migration process is done:

2025-01-15 05:49:39,408 Start to fetch [commits] object from destination
2025-01-15 05:49:39,422 Start to fetch [fs] object from destination
2025-01-15 05:49:39,442 Start to fetch [blocks] object from destination
2025-01-15 05:49:39,677 [commits] [0] objects exist in destination
2025-01-15 05:49:39,677 Start to migrate [commits] object
2025-01-15 05:49:39,749 [blocks] [0] objects exist in destination
2025-01-15 05:49:39,755 Start to migrate [blocks] object
2025-01-15 05:49:39,752 [fs] [0] objects exist in destination
2025-01-15 05:49:39,762 Start to migrate [fs] object
2025-01-15 05:49:40,602 Complete migrate [commits] object
2025-01-15 05:49:40,626 Complete migrate [blocks] object
2025-01-15 05:49:40,790 Complete migrate [fs] object
Done.

Replace the original seafile.conf and start Seafile

After running the script, we recommend that you check whether your data already exists on the new S3 storage backend server (i.e., the migration is successful, and the number and size of files should be the same). Then you can remove the file from the old S3 storage backend and replace the original seafile.conf from the new one:

mv /opt/seafile-data/seafile.conf /opt/seafile-data/seafile/conf/seafile.conf

mv /opt/seafile.conf /opt/seafile/conf/seafile.conf

Finally, you can start Seafile server:

docker compose up -d

# make sure you are in the directory `/opt/seafile/seafile-server-latest`
./seahub.sh start
./seafile.sh start