Synchronize Files Between Multiple Systems With Syncthing

In this step by step tutorial, we are going to learn what is Syncthing, how to install Syncthing on Linux, how to synchronize files between multiple systems with Syncthing in real time, and finally how to troubleshoot common Syncthing problems.

1. Introduction to Syncthing

Syncthing is a free, opensource, peer-to-peer (P2P) file synchronization program that allows you to securely synchronize files between two or more devices within LAN or remote devices over Internet.

The device could be a desktop system, server, or a mobile phone. The changes made in one device (i.e. file creation, modification, and deletion etc.) will automatically be replicated to other devices.

Syncthing doesn't need any configuration! It just works out of the box. Each device is identified by a ID. Just give the device ID to exchange files.

Since it is a P2P file sync program, your data will not be uploaded to any public cloud or central server. Syncthing simply exchanges the data between your devices when they are online at the same time.

Syncthing is a platform-independent application, so you can sync files between multiple computers that runs with same or different OS, or sync files between a computer and a mobile device and vice versa. It is also possible to sync a single folder to multiple locations on different devices.

Syncthing file synchronization tool is written in Go programming language. It works under GNU/Linux, Mac OS, Solaris and several versions of BSD.

There is also an official Syncthing Android app available for those who wants to sync files between mobile devices and computers.

Heads Up: Please note that Syncthing doesn't support local synchronization. Meaning - you can't sync files between folders on the same system. If you want to perform local sync, consider using either rsync or unison.

1.1. How Syncthing works?

Syncthing divides files into multiples pieces called Blocks and transfer them from one device to another. Therefore, multiple devices can share the synchronization load, similar to a torrent protocol. So, the data transfer rate is usually fast. The more devices you have online, the faster the data transfer rate.

If the content of the files are changed simultaneously and are different on local and remote devices, it causes a conflict. Syncthing handles the file conflicts efficiently by renaming one of the files to <filename>.sync-conflict-<date>-<time>-<modifiedBy>.<ext>.

Usually, the file with the older modification time will be marked as the conflicting file and then it will be renamed. If the modification times are equal, the file originating from the device which has the larger value of the first 63 bits for its device ID will be marked as the conflicting file. 

If the conflict is between a modification and a deletion of the file, the modified file will be restored without renaming on the device where it was deleted.

Another notable thing about Syncthing is it never writes directly to a destination file. It makes a temporary copy of the data before overwriting the old version. Once the file transfer is completed successfully, the temporary files will be deleted.

If there are any errors during file synchronization, the temporary file is kept for 24 hours by default. You can, of course, change this setting as per your convenient. The temporary files are named .syncthing.original-filename.ext.tmp and they are hidden by default.

As stated already, Syncthing directly transfers files between two devices. When it is not possible to establish a direct connection between the two devices, Syncthing connects them via a relay.

The relayed connections are encrypted using TLS encryption method to securely transfer data between the devices. It uses its own free protocol, named Block Exchange Protocol to establish communication with all devices.

2. Syncthing features

The core features of Syncthing are given below:

• Private - The data is never replicated anywhere else other than on the devices configured.
• Secure - All communication between devices are encrypted with TLS.
• Authentication - Every device is identified by a unique device ID. The device ID is used for address resolution, authentication and authorization. Only the devices you have explicitly allowed can connect to your cluster.
• Transparency - Syncthing is an open source project. So anyone can inspect its code and understand how Syncthing finds other active devices to sync with, how it transfers data from node to node, and how it synchronizes data between different devices.
• Cost - Syncthing is a cloudless, decentralised file sharing application. There is no central server or cloud service to pay for. So, it is completely free!
• Limitation - There is no limit. You can share unlimited number of files of any size as long as you have enough free disk space on your devices.
• Cross-platform and portable - Syncthing is platform-agnostic. It works on many operating systems, including Linux, Windows, Solaris, BSDs and Android.

3. Install Syncthing On Linux

You must install Syncthing on all systems that you want to keep in sync.

Syncthing is packaged for many Linux distributions and is available in the official repositories.

To install Syncthing on Arch Linux and its variants such as EndeavourOS and Manjaro Linux, run:

$ sudo pacman -S syncthing

On Debian, Ubuntu:

$ sudo apt install syncthing

Please note that in Ubuntu, you must enable [universe] repository to install Syncthing.

On RHEL 8, CentOS 8, AlmaLinux 8, Rocky Linux 8, Fedora 31 and newer versions:

Enable [EPEL] repository:

$ sudo dnf install epel-release

And then install syncthing using the following command:

$ sudo dnf install syncthing

On CentOS 7 and 6 editions, you can install Syncthing from pre-compiled binaries as described in the next section.

openSUSE:

$ sudo zypper install syncthing

3.1. Install Syncthing from binaries

If Syncthing is not available in the default repositories, you can download the pre-compiled binaries from the releases page, and move it to your $PATH as shown below.

$ wget https://github.com/syncthing/syncthing/releases/download/v1.11.1/syncthing-linux-amd64-v1.11.1.tar.gz

Extract it:

$ tar xvf syncthing-linux-amd64-v1.11.1.tar.gz

And move it to your $PATH, for example /usr/local/bin/.

$ sudo cp syncthing-linux-amd64-v1.11.1/syncthing /usr/local/bin/

To check if it is installed correctly, try to display its version using command:

$ syncthing --version

If you see an output like below, congratulations! You have successfully installed Syncthing!!

syncthing v1.11.1 "Fermium Flea" (go1.15.3 linux-amd64) teamcity@build.syncthing.net 2020-11-03 11:29:54 UTC

3.2. Install syncthing-gtk client

Syncthing also has a graphical user interface named syncthing-gtk. If you want desktop notifications, and integration with your file managers such as Nautilus, Nemo, Caja, install the syncthing GTK graphical interface.

Syncthing-gtk is available in the official repositories, so you can install it using your Linux distribution's default package manager as well. For instance, to install syncthing-gtk on Debian-based systems, run:

$ sudo apt install syncthing-gtk

If Syncthing-gtk is not available in the official repositories, get the flatpak app instead. Make sure you have installed flatpak on your Linux machine and then run the following command to install syncthing-gtk:

$ flatpak install flathub me.kozec.syncthingtk

4. Initialize Syncthing on all devices

For the purpose of this guide, I will be using the following two systems.

System1:

Operating system   - Ubuntu 20.04 LTS desktop system
Hostname           - ostechnix
IP Address         - 192.168.225.37/24

System2:

Operating system   - Fedora 32 silverblue desktop
Hostname           - toolbox
IP Address         - 192.168.225.46/24

You must do the followings steps on all systems that you want to keep in sync.

Open your Terminal and run Syncthing to initialise it:

$ syncthing

If Syncthing is installed using flatpak, launch it by running this command:

$ flatpak run me.kozec.syncthingtk

When you run the Syncthing for the first time, it will generate the necessary keys, create default configuration files and setup a default synchronization folder named "Sync" in your $HOME directory.

Here is a sample output from my Ubuntu desktop system when initializing Syncthing at first time:

[monitor] 17:23:06 INFO: Starting syncthing
[start] 17:23:06 INFO: Generating ECDSA key and certificate for syncthing...
[R7FIO] 17:23:06 INFO: syncthing unknown-dev "Erbium Earthworm" (go1.13.5 linux-amd64) unknown@unknown 1970-01-01 00:00:00 UTC
[R7FIO] 17:23:06 INFO: My ID: R7FIOSA-3ZO6FLJ-LFBAVWM-LUEDOR2-RPDXXH7-4P2HFWX-QCWBN4D-ENEGIAY
[R7FIO] 17:23:07 INFO: Single thread SHA256 performance is 118 MB/s using minio/sha256-simd (88 MB/s using crypto/sha256).
[R7FIO] 17:23:07 INFO: Default folder created and/or linked to new config
[R7FIO] 17:23:07 INFO: Default config saved. Edit /home/sk/.config/syncthing/config.xml to taste (with Syncthing stopped) or use the GUI
[R7FIO] 17:23:07 INFO: Hashing performance is 103.85 MB/s
[R7FIO] 17:23:07 INFO: Starting deadlock detector with 20m0s timeout
[R7FIO] 17:23:07 INFO: No stored folder metadata for "default": recalculating
[R7FIO] 17:23:07 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[R7FIO] 17:23:07 INFO: Overall send rate is unlimited, receive rate is unlimited
[R7FIO] 17:23:07 INFO: Using discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[R7FIO] 17:23:07 INFO: TCP listener ([::]:22000) starting
[R7FIO] 17:23:07 INFO: Using discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[R7FIO] 17:23:07 INFO: Using discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[R7FIO] 17:23:07 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[R7FIO] 17:23:08 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[R7FIO] 17:23:08 INFO: Loading HTTPS certificate: open /home/sk/.config/syncthing/https-cert.pem: no such file or directory
[R7FIO] 17:23:08 INFO: Creating new HTTPS certificate
[R7FIO] 17:23:08 INFO: GUI and API listening on 127.0.0.1:8384
[R7FIO] 17:23:08 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[R7FIO] 17:23:08 INFO: My name is "ostechnix"
[R7FIO] 17:23:18 INFO: Detected 0 NAT services
[R7FIO] 17:24:42 INFO: Joined relay relay://210.23.25.77:22067

Once the initialization is completed, the Syncthing web GUI will automatically open in your default browser:

As you can see in the above screenshot, Syncthing's web interface is very easy to understand and use. Most of the options are self-explanatory.

The web interface has three sections namely,

1. Folders,
2. This Device,
3. and Remote Devices.

The "Folders" section shows the list of shared folders. By default, the default shared directory ~/Sync is listed here.

In the "This Device" section, you will find the details such as the name of the computer, download/upload rate, RAM/CPU utilisation, total up time and a few more.

Below the "The Device" section, there is another section named "Remote Devices" section where you can add all remote devices to sync with. Since we didn't add any devices yet, it shows empty.

On the top of the Syncthing web interface, you can choose your preferred language, open help section and do various administrative tasks under Actions tab.

The list of things you can configure from the "Actions" tab is given below:

• change device name,
• generate API key,
• minimum required free disk space in the $HOME directory,
• enable or disable anonymous usage report ,
• change default folder path,
• change GUI listen address,
• set GUI admin username and password,
• enable HTTPS,
• set incoming and outgoing rate limit,
• add devices or folders to ignore from synchronization,
• and more.

Most options in the Actions tab should be left untouched. The default configuration is just enough. If you're familiar with all options, change them as per your wish.

Now, go to the other systems and initialize Syncthing to generate necessary config files and create default synchronization folder.

You can view the currently configured syncthing paths using command:

$ syncthing --paths

Sample output:

Configuration file:
	/home/sk/.config/syncthing/config.xml

Database directory:
	/home/sk/.config/syncthing/index-v0.14.0.db

Device private key & certificate files:
	/home/sk/.config/syncthing/key.pem
	/home/sk/.config/syncthing/cert.pem

HTTPS private key & certificate files:
	/home/sk/.config/syncthing/https-key.pem
	/home/sk/.config/syncthing/https-cert.pem

Log file:
	-

GUI override directory:
	/home/sk/.config/syncthing/gui

Default sync folder directory:
	/home/sk/Sync

4.1. Set administrative username and password for Syncthing web GUI

By default, you can access the Syncthing web GUI without any username and password. To make your Syncthing instance little bit secure, set an administrative user and password.

Click the "Actions" tab on the top right corner and choose "Settings" from the drop-down menu.

Under the "GUI" tab, enter your administrative username and its password. Make sure the password is strong and is not easier to guess. Also enable "Use HTTPS for GUI" option.

Click Save button to create the admin user account.

Now refresh your web browser and you will see a certificate warning message. This is because we have just enabled HTTPS for web GUI.

To bypass this warning, click the "Advanced" button and then click on the "Accept The Risk And Continue" option to add your localhost to the exception list.

Next, Enter the administrative username and its password to access the Syncthing web GUI.

You can also launch the syncthing-gtk client to initialize Syncthing.

Repeat the same steps on the second machine to setup admin user for web GUI.

5. Synchronize Files Between Multiple Systems With Syncthing

Start Syncthing on both machines by running this command:

$ syncthing

The Syncthing web gui will automatically open in the default browser.

To sync files between systems, we need to pair them with each other using their Device ID. You can view the device ID under "Actions" tab.

Here is the device id of my Ubuntu 20.04 desktop machine:

You should exchange the above ID with other system in order to make the two systems to talk to each other.

To do so, click "Add Remote Device" button from Remote Devices section.

You should see the nearby system's device ID in the next screen. Click on the device ID to pair it with your current system.

Next, go to the "Sharing" tab and select the folders to share with the current device. Optionally, you can check the "Auto Accept" box to automatically create or share folders that this device advertises at the default path. FYI, the default shared path is ~/Sync.

In the "Advanced" tab, you can choose the Compression method, and set Device rate limits.

Finally, click the "Save" button to pair the system2 with system1 machine.

Just in case if the device ID is not shown, manually copy it from the system2 and paste it here.

On other system (i.e. system2), you will see a notification message asking you to add the first system. Click "Add Device" to accept the request and pair them.

Similarly, repeat the above steps on the system2 to pair it with system1 machine. After you have successfully paired the both systems with each other, you will see them listed under the "Remote Devices" section in Syncthing web interface.

Here, "toolbox" is the name of my Fedora desktop.

You will also the "Up to Date" message next to the remote device and also to the "Default Folder" tab on the left pane. Meaning - they are ready to sync!

To view the shared folder's details, click on the "Default Folder" tab on the left pane to expand it. You will see various details of the shared folder such as;

• shared folder id,
• shared folder path,
• state of the folder,
• time interval between periodic scan,
• to which device it is currently shared,
• the time of last scan etc.

We're done now. We have successfully installed and configured Synchthing on both systems and both are paired with each other.

From now on, whatever you put in the ~/Sync folder on one of systems will be synchronized to the other system. You can view the synchronization progress from Syncthing interface:

Whenever you create, modify or delete something from the ~/Sync folder in any one of systems, the changes will be automatically replicated to other system and vice versa.

Once the Synchronization is completed, close the Syncthing web interface and then exit syncthing command by pressing Ctrl+C.

5.1. Create a new shared folder

You can create multiple shared directories and sync them all with other paired devices.

To create a new shared directory, click "Add Folder" under "Folders" section on the left.

In the "General" tab, enter the folder label, folder id and folder path to share with other devices.

Next, go to the "Sharing" tab and select the devices to share this folder.

You can also set a shared folder as Send-only or Receive-only or both Send & Receive. This option is available under "Folder Type" drop-down box in "Advanced" tab.

Finally click "Save" button to activate this share.

The new shared directory will now be listed under the "Folders" section. You can change the shared folder settings at any time by clicking on the "Edit" button.

5.2. Configure firewall

If you are behind a third-party, local firewall or router, allow the ports 22000/TCP and 21027/UDP for incoming and outgoing traffic. Please note that you need to do this on all systems.

If you have configured ufw (uncomplicated firewall) in your Linux machine, you can allow the Syncthing ports using the following command:

$ sudo ufw allow syncthing

If you are using firewalld, then run the following commands to open the necessary ports:

$ sudo firewall-cmd --zone=public --add-service=syncthing --permanent

Finally, restart firewalld service to take effect the changes:

$ sudo firewall-cmd --reload

5.3. Enable remote Web GUI (optional)

The default port for web GUI is 8384. You can access the Syncthing web GUI by navigating to the URL http://localhost:8384 or http://127.0.0.1:8384 from the web browser.

By default, Syncthing web GUI can only be accessed from the localhost itself. If you want to access the web GUI from other systems in the network, you should allow the port 8384 in your local firewall/router.

If you are using ufw, run the following command to allow the syncthing-gui service:

$ sudo ufw allow syncthing-gui

Similarly, you can allow external access to the Syncthing web GUI via firewalld like below:

$ sudo firewall-cmd --zone=public --add-service=syncthing-gui --permanent

Restart firewalld service to apply the changes:

$ sudo firewall-cmd --reload

After allowing the default port via firewall, change the GUI listen address through the web GUI from 127.0.0.1:8384 to 0.0.0.0:8384:

Alternatively, edit Syncthing config.xml file:

$ nano ~/.config/syncthing/config.xml

Find the following line:

<address>127.0.0.1:8384</address>

And change it to:

<address>0.0.0.0:8384</address>

Press CTRL+O followed by CTRL+X to save and quit the file.

Now you can access the web GUI from any remote machine using "https://IPAddress:8384" URL.

5.4. Tunnelling via SSH

If both systems can be accessible via SSH, you can access the remote system's web GUI in your local machine through a SSH tunnel. It is bit safer than opening the Syncthing's web gui port to the outside world.

To start a SSH tunnel, run:

$ ssh -L 9999:127.0.0.1:8384 sk@192.168.225.46

Replace "sk@192.168.225.46" with your remote username and IP address. This will bind to your local port 9999 and forward all connections from there to port 8384 on the target machine. This still works even if Syncthing is bound to listen on localhost only.

Now you can be able to access the remote web GUI from your local system by navigating to https://localhost:9999/.

5.5. Autostart Syncthing

Whenever you want to sync something, you need to manually start syncthing instance. If you wish to start it automatically on every reboot, do the following.

Create a new systemd service for syncthing:

$ sudo nano /etc/systemd/system/syncthing@.service

Add the following lines in it:

[Unit]
Description=Syncthing - Open Source Continuous File Synchronization for %I
Documentation=man:syncthing(1)
After=network.target

[Service]
User=%i
ExecStart=/usr/local/bin/syncthing -no-browser -gui-address="0.0.0.0:8384" -no-restart -logflags=0
Restart=on-failure
SuccessExitStatus=3 4
RestartForceExitStatus=3 4

# Hardening
ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Replace "ExecStart=/usr/local/bin/syncthing" with the correct path of the syncthing executable file.

If you have installed Syncthing using the distribution's package manager, the executable path would be available here - /usr/bin/syncthing.

If syncthing is installed from pre-compiled binaries, the path would be "/usr/local/bin/syncthing".

After updating the correct path, save and close file.

Next, reload systemd units using command:

$ sudo systemctl daemon-reload

Finally, start syncthing service and enable to start automatically on system reboot:

$ sudo systemctl start syncthing@$USER

$ sudo systemctl enable syncthing@$USER

Verify if Syncthing is running using command:

$ sudo systemctl status syncthing@$USER

Sample output from my CentOS box:

● syncthing@ostechnix.service - Syncthing - Open Source Continuous File Synchronization for ostechnix
   Loaded: loaded (/etc/systemd/system/syncthing@.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-11-24 18:07:34 IST; 1s ago
     Docs: man:syncthing(1)
 Main PID: 10932 (syncthing)
    Tasks: 11 (limit: 11480)
   Memory: 20.8M
   CGroup: /system.slice/system-syncthing.slice/syncthing@ostechnix.service
           ├─10932 /usr/local/bin/syncthing -no-browser -gui-address=0.0.0.0:8384 -no-restart -logflags=0
           └─10937 /usr/local/bin/syncthing -no-browser -gui-address=0.0.0.0:8384 -no-restart -logflags=0

Nov 24 18:07:34 centos8.ostechnix.lan systemd[1]: syncthing@ostechnix.service: Service RestartSec=100ms expired, scheduling restart.
Nov 24 18:07:34 centos8.ostechnix.lan systemd[1]: syncthing@ostechnix.service: Scheduled restart job, restart counter is at 23.
Nov 24 18:07:34 centos8.ostechnix.lan systemd[1]: Stopped Syncthing - Open Source Continuous File Synchronization for ostechnix.
Nov 24 18:07:34 centos8.ostechnix.lan systemd[1]: Started Syncthing - Open Source Continuous File Synchronization for ostechnix.
Nov 24 18:07:34 centos8.ostechnix.lan syncthing[10932]: [start] INFO: syncthing v1.11.1 "Fermium Flea" (go1.15.3 linux-amd64) teamcity@>
Nov 24 18:07:34 centos8.ostechnix.lan syncthing[10932]: [4ZC47] INFO: My ID: 4ZC47V7-KW3P2BH-FSLG3QN-2D2W3OO-LI2KEQ5-JJOBXWY-QMFOAI3-FP>
Nov 24 18:07:35 centos8.ostechnix.lan syncthing[10932]: [4ZC47] INFO: Single thread SHA256 performance is 117 MB/s using minio/sha256-s>
lines 1-18/18 (END)

From now on, Syncthing will automatically start at startup and sync the files between the configured systems.

5.6. Troubleshooting

Some times you may encounter database error and Syncthing will refuse to sync the files between devices. In such cases, you can forcibly rescan files and resync the database using the following command:

$ syncthing -reset-database

6. Conclusion

You know now how to synchronize files between multiple systems with Syncthing in Linux. As you can see, file synchronization using Syncthing is quite easy and straight-forward.

Even though there are many cloud services are available to store the data, there are many concerns about the privacy, security and the lack of control over your valuable data.

Thanks to Syncthing, you can now have a greater peace of mind. There is no central server, there is no eavesdropping! You control your own data!

Resources:

• Syncthing Website
• Syncthing GitHub Repository