How To Synchronize Files With Unison On Linux

This comprehensive tutorial explains what is Unison file synchronization tool, how to install Unison on Linux and how to synchronize files with Unison between two different directories and/or disks on the same system and two different Linux machines over local area network.

What is Unison?

Unison is a free, open source, multi-platform, and bidirectional file synchronization tool. It is used to store two replicas of files and directories on two different systems or different disks on the same system.

The replicas of files and directories are modified separately and then brought up-to-date by propagating the changes in each replica to the other.

To put this in simple terms, any changes in the one location will be replicated to other location and vice versa.

Since it is a multi-platform application, you can synchronize files and directories between computers that runs with different operating systems.

For instance, you can synchronize files stored in a Linux machine with a Windows or Unix machines and vice versa. Not just file content, even the file ownership and permissions will also be sychronized.

If the content are different on both replicas, it causes a conflict. Unison will detect and display such conflicts. In that case, you need to instruct Unison which replica should override the other. Updates that do not conflict will be propagated automatically.

Unison works well with any pair of computers connected to a local area network or Internet. The files are synchronized to a remote system via an encrypted ssh connection. 

Another notable feature of Unison is it works well over slow links such as PPP connections. Similar to rsync, Unison will only transfer the parts of a file that have been changed.

Unison's built-in implementation of the rsync algorithm makes transferring updates to existing files very fast. 

Unison is a user-level program, so there is no need to modify the kernel or to have superuser privileges on either host.

The Unison program is created by Benjamin C. Pierce and is released under GPL license. It works under GNU/Linux, Windows and Unix platforms.

Install Unison on Linux

Unison is available in the default repositories in the official repositories of many Linux operating systems.

To install Unison on Alpine Linux, run:

$ sudo apk add unison

Install Unison on Arch Linux and its variants like EndeavourOS and Manjaro Linux:

$ sudo pacman -S unison

Debian, Ubuntu, Linux Mint:

$ sudo apt install unison

Fedora:

$ sudo dnf install unison

CentOS 7:

First, enable [EPEL] repository:

$ sudo yum install epel-release

And then install unison:

$ sudo yum install unison

openSUSE:

$ sudo zypper install unison

There is also simple graphical interface named unison-gtk is available for those who prefer GUI over CLI. It is available in the default repositories in a few Linux distributions.

On Alpine Linux, you can install Unison-gtk using command:

$ sudo apk add unison-gui

On Arch Linux and its variants, install Unison-gtk using command:

$ sudo pacman -S unison-gtk2

If you are on a Debian-based system, install it using the following command:

$ sudo apt install unison-gtk

Please note that if you want to synchronize files between two different systems, you must install Unison on all systems that you want to keep in sync. Also, the same version on both systems is recommended.

Synchronize Files With Unison On Linux

We can use Unison in the following three different modes:

1. Sync files stored in different directories on the same machine,
2. Sync files and directories stored in different disks on the same machine,
3. Sync files and directories between local and remote machines:
   • Sync files with remote machine over a direct socket connection,
   • Sync files with remote machine using SSH.

1. Sync files stored in different directories on the same machine

For the purpose this guide, I am going to create two root directories (i.e. replicas) namely directory1 and directory2.

$ mkdir ~/directory1

$ mkdir ~/directory2

Put some contents on any one of the above directories. I stored some files and directories inside directory1.

Let us take a look at the contents of directory1:

$ ls -ltRF directory1/

Sample output:

directory1/:
total 7780
drwxrwxr-x 2 sk sk    4096 Oct  2 15:35 dir1/
-rw-rw-r-- 1 sk sk 7779007 Sep 25 16:51 file2
-rw-rw-r-- 1 sk sk  179765 Aug 17 12:51 file1

directory1/dir1:
total 140
-rw-rw-r-- 1 sk sk 141729 Aug  6 22:34 file3

As you can see, I have a folder called dir1 and two other files namely file1 and file2 in the root of directory1. Inside the dir1, there is another file called file3.

And the directory2 replica doesn't have any contents.

$ ls -ltRF directory2/
directory2/:
total 0

Let us sync these two directories using unison with command:

$ unison directory1/ directory2/

You will be prompted if you want to continue to sync the contents of directory1 with directory2. Simply press ENTER key. Next, Unison will ask you to confirm the files and directories to synchronize. Press ENTER or press f key to confirm the files and folders that you want to synchronize. Finally, press y and hit ENTER key to proceed with propagating updates.

Contacting server...
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
	/home/sk/directory1
	/home/sk/directory2
This can happen either
because this is the first time you have synchronized these roots, 
or because you have upgraded Unison to a new version with a different
archive format.  

Update detection may take a while on this run if the replicas are 
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted: 
http://www.cis.upenn.edu/~bcpierce/unison

Reconciling changestinue.[<spc>] / file1

directory1     directory2         
dir      ---->            dir1  [f] f
file     ---->            file1  [f] f
file     ---->            file2  [f] f

Proceed with propagating updates? [] y
Propagating updates


UNISON 2.48.4 started propagating changes at 15:43:29.75 on 02 Oct 2020
[BGN] Copying dir1 from /home/sk/directory1 to /home/sk/directory2
[END] Copying dir1
[BGN] Copying file1 from /home/sk/directory1 to /home/sk/directory2
[END] Copying file1
[BGN] Copying file2 from /home/sk/directory1 to /home/sk/directory2
[END] Copying file2
UNISON 2.48.4 finished propagating changes at 15:43:29.81 on 02 Oct 2020


Saving synchronizer state
Synchronization complete at 15:43:29  (3 items transferred, 0 skipped, 0 failed)

Now, all contents of directory1 will be replicated to directory2. You can verify it using ls command:

See? The contents of both replicas are identical. Like I already mentioned, file permissions and inode numbers of files and folders on both locations are also synchronized. You can see that the permissions and inodes are identical as well in the above output.

Did you notice the arrow marks in the output while synchronizing the directories? For example, you should have noticed the following lines in the middle of the output:

directory1     directory2         
dir      ---->            dir1  [f] f
file     ---->            file1  [f] f
file     ---->            file2  [f] f

The arrow marks indicates the direction that the change needs to be propagated. In our case, the arrows points from left to right. Meaning - the dir1, file1 and file2 are propagated to second replica i.e. directory2.

Now create a new file in directory2 replica and see what happens.

$ touch directory2/file4

Sync the two replicas using command:

$ unison directory1/ directory2/

Now you will see that a arrow points from right to left. It means the changes in the second replica are propagated to the first replica. In other words, the contents of directory2 are replicated to directory1.

Contacting server...
Looking for changes
Reconciling changes

directory1     directory2         
         <---- new file   file4  [f] f

Proceed with propagating updates? [] y
Propagating updates


UNISON 2.48.4 started propagating changes at 16:04:08.78 on 02 Oct 2020
[BGN] Copying file4 from /home/sk/directory2 to /home/sk/directory1
[END] Copying file4
UNISON 2.48.4 finished propagating changes at 16:04:08.78 on 02 Oct 2020


Saving synchronizer state
Synchronization complete at 16:04:08  (1 item transferred, 0 skipped, 0 failed)

Unison will automatically recognise which files need to be updated in both replicas. When you run Unison for the first time, it will take a note of modification timestamp, permissions, ownership and inode number of all files and folders in both locations. Based on this information, Unison will decide which files need to be updated when you run the same command again. As stated earlier, all information will be kept under ~/.unison directory.

To verify it yourself, modify the contents of an existing file. I am going to modify the contents of file4 in directory1 replica.

$ echo "Welcome to OSTechNix" > directory1/file4

Now start the synchronization again:

$ unison directory1/ directory2/

This time Unison will sync the modified file(s).

Contacting server...
Looking for changes
Reconciling changes

directory1     directory2         
changed  ---->            file4  [f] f

Proceed with propagating updates? [] y
Propagating updates


UNISON 2.48.4 started propagating changes at 16:05:27.13 on 02 Oct 2020
[BGN] Updating file file4 from /home/sk/directory1 to /home/sk/directory2
[END] Updating file file4
UNISON 2.48.4 finished propagating changes at 16:05:27.14 on 02 Oct 2020


Saving synchronizer state
Synchronization complete at 16:05:27  (1 item transferred, 0 skipped, 0 failed)

As you can see in the above output, Unison has detected that file4 has been changed in the first replica and synchronized it with second replica.

If both replicas are modified and their contents are different, then the changes are in conflict. In such cases, Unison will ask the user how to propagate the updates on both sides. The conflicts will be displayed with left and right arrows like below.

directory1     directory2         
new file <-?-> new file   file6  [] 

If some default action is indicated by an arrow, simply press ENTER key to go on to the next changed file. If you want to do something different with this file, you can press either "<" or ">" to force the change to be propagated from right to left or from left to right.

directory1     directory2         
new file <==== new file   file6  [] <

If you don't want to make any changes, just press "/" to skip this file and leave both replicas alone.

1.1. Unison profiles

Unison uses profiles to sync files and folders between two directories on same or two different machines. A profile is nothing but a text file where we define the actual path of root directories and the path preferences that should be synchronized. All profiles will be stored in .unison directory in your $HOME directory.

When you run Unison for the first time, a default profile named default.prf is created under ~/.unison directory. The default profile doesn't have any preferences defined. So everything stored in one replica will be synchronized to other replica.

We can also create a custom profiles with the list of path preferences. This will help you to avoid synchronizing the entire two replicas.

Using the custom profiles, we can instruct Unison:

• to sync specific files and folders,
• to ignore a specific file or specific type of files from synchronization.

Let us see an example.

Create two replicas and save some files and folders in one replica and leave the other empty.

$ mkdir ~/directory1

$ mkdir ~/directory2

Then, create a new profile named dir_profile.prf:

$ nano .unison/dir_profile.prf

Define the root directories and the paths of files and directories that you want to sync:

# Roots of the synchronization
root = /home/sk/directory1
root = /home/sk/directory2 

# Paths to synchronize 
path = dir1
path = file2

The two roots can be given in either order. Unison will sort them into a canonical order before doing anything else. If you don't specify any paths, Unison will simply synchronize the two entire replicas, beginning from the given pair of roots. If one or more path preferences are given, then Unison will synchronize only these paths and their children.

In the above example, I have defined the dir1 and file2 should be synced to other root directory. Save and close the file.

After creating the profile, start the synchronization process using command:

$ unison dir_profile.prf

You will be prompted if you want to continue to sync the contents of directory1 with directory2. Simply press ENTER key and then press f key to confirm the paths to synchronize. Finally press y and hit ENTER key to proceed with propagating updates.

Contacting server...
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
	/home/sk/directory1
	/home/sk/directory2
This can happen either
because this is the first time you have synchronized these roots, 
or because you have upgraded Unison to a new version with a different
archive format.  

Update detection may take a while on this run if the replicas are 
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted: 
http://www.cis.upenn.edu/~bcpierce/unison

Reconciling changestinue.[<spc>] | dir1

directory1     directory2         
dir      ---->            dir1  [f] 
file     ---->            file2  [f] 

Proceed with propagating updates? [] y
Propagating updates


UNISON 2.48.4 started propagating changes at 15:42:32.99 on 01 Oct 2020
[BGN] Copying dir1 from /home/sk/directory1 to /home/sk/directory2
[END] Copying dir1
[BGN] Copying file2 from /home/sk/directory1 to /home/sk/directory2
[END] Copying file2
UNISON 2.48.4 finished propagating changes at 15:42:33.04 on 01 Oct 2020


Saving synchronizer state
Synchronization complete at 15:42:33  (2 items transferred, 0 skipped, 0 failed)

Now go to the second replica directory2 and see if the contents are syncd.

1.1. Ignore files from synchronization

Sometimes, you might not want to synchronize a specific file or specific type of files. If so, you can define which files to ignore in the profile like below.

ignore=Name text.*
ignore=Name .*~
ignore=Name *.pdf

For more sample profiles, refer here.

1.2. Sync local files and directories with unison-gtk

Some of you may not comfortable with command line. Thankfully, Unison has a simple graphical interface which allows the users to easily create profiles and sync files based on the profiles via a graphical window.

Make sure you have installed unison-gtk package as shown in the installation section above. Launch unison-gtk from menu or application launcher.

You will see the profile selection window upon first launch. If you already have created a profile as described, you will see it there.

From the profiles window, you can create a new profile with new root directories and the paths to synchronize. Or, you can simply edit the existing profile and modify the root directories or add new files or change the paths to synchronize.

If you don't need a profile anymore, just choose it and click Delete.

I am going to create a new profile. To do so, click Add button from profile selection window and click Forward.

Enter a suitable name for your new profile with a small description:

Next choose the kind of synchronization you want to do. Since it is synchronization between two local directories, I chose "local".

Select the two root directories that you want to sync.

Select the following check box if one of your root directory is on a FAT partition. This is typically the case for a USB thumb drive.

Finally, click Apply to create the profile.

The newly created profile will now appear in the profile selection window. Select it and click Open.

The Unison graphical will now show you the changes to be made on both sides. Just like the Unison command line interface, the default action is indicated by an arrow in the graphical interface as well. As you see in the below screenshot, the files are propagated from left to right i.e. first replica to second replica. If the contents of replicas are not identical, Unison will ask you instructions on which replica should override the other.

If you are satisfied with the directions for the propagation of changes, click Go on the tool bar or select Synchronization -> Go from the menu bar.

Once the synchronization is successfully completed, you will see the tick marks under the Status section.

2. Sync files and directories stored in different disks on the same machine

Sync files between two disks are exactly same as above. Just make sure you have mentioned the correct path of the root directory in the external drive.

You know now how to sync files between two directories in the same hard disk and different disk on a local system. The following section explains how to sync files between two different systems.

3. Sync files and directories between local and remote machines

As stated earlier, Unison can sync files between local and remote systems via direct socket connection and via encrypted SSH connection. As you already know, SSH is much better and more secure than the socket method. For the purpose of this guide, I will be using SSH method to sync files.

Make sure you have installed openSSH and Unison on both systems. Also be sure that both system can able to communicate via SSH. Please note that your local and remote systems should have the same version of Unison.

To check the version of Unison, run the following command on your local and remote sysem:

$ unison -version
unison version 2.48.4

Since the files will be synced via SSH, it doesn't matter whether the remote system has unison-gtk or not. Unison commandline interface is just enough on remote machine! On your local machine, you can either use textual or graphical unison interface.

3.1. Sync files and directories between local and remote machines from command line

Now let us sync the directory1 from our local system to the directory1 in remote system using unison by running the following command:

$ unison directory1/ ssh://ostechnix@192.168.225.52/directory1/

Here, ostechnix@192.168.225.52 is the username and ip address of my remote system.

You will be prompted to enter the password of your remote user. Next press ENTER to agree to sync the contents of directory1 of local machine with directory1 of remote machine. Then press f key to confirm the paths to synchronize. Finally press y and hit ENTER key to proceed with propagating updates.

Contacting server...
ostechnix@192.168.225.52's password: 
Connected [//ostechnix//home/sk/directory1 -> //ubuntuserver//home/ostechnix/directory1]
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
	/home/sk/directory1
	//ubuntuserver//home/ostechnix/directory1
This can happen either
because this is the first time you have synchronized these roots, 
or because you have upgraded Unison to a new version with a different
archive format.  

Update detection may take a while on this run if the replicas are 
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty.  This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations.  See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted: 
http://www.cis.upenn.edu/~bcpierce/unison

Press return to continue.[<spc>]   Waiting for changes from server
Reconciling changes

local          ubuntuserver       
file     ---->            file1  [f] f
file     ---->            file2  [f] f

Proceed with propagating updates? [] y
Propagating updates


UNISON 2.48.4 started propagating changes at 14:49:18.44 on 03 Oct 2020
[BGN] Copying file1 from /home/sk/directory1 to //ubuntuserver//home/ostechnix/directory1
[BGN] Copying file2 from /home/sk/directory1 to //ubuntuserver//home/ostechnix/directory1
[END] Copying file1
[END] Copying file2
UNISON 2.48.4 finished propagating changes at 14:49:18.45 on 03 Oct 2020


Saving synchronizer state
Synchronization complete at 14:49:18  (2 items transferred, 0 skipped, 0 failed)

The above command will replicate the contents of directory1 from the local system to the directory1 on the remote system. Now add a new file or update an existing file and try synchronizing again. You should see similar results on both side.

If you want to synchronize the contents to somewhere else other than the home directory on the remote system, you must specify the absolute path of the remote directory by adding an extra slash between remote hostname and the beginning of the path like below:

$ unison directory1/ ssh://ostechnix@192.168.225.52//sync/the/files/here/

If you don't want to sync the entire directory, simply create a custom profile and define the paths to synchronize as described in section titled 1.1. Unison profiles above.

3.2. Sync files and directories between local and remote machines using unison-gtk

Launch Unison-gtk application from the menu or launcher. Click Add button in the profile selection window to create new profile.

Click Forward.

Enter Unison profile name and its description and click Forward.

Here is the important step. Choose "ssh" from Synchronization kind drop-down box. Enter the remote system's IP address or hostname and the remote username. Also check "Enable compression" box to improve the data transfer performance.

Choose local and remote directory locations that you want to synchronize.

If one of your directories are on FAT partition, you must check this option. This is usually the case for USB drives.

Click Apply to complete profile creation.

The newly created profile will now appear in the profile selection window. Select the profile and click Open.

You will be asked to enter your remote user's password. Enter the password and click OK.

After entering the password, Unison will list the changes to be propagated from local to remote. The default action is indicated with an arrow. If you don't want the default action, press "left to right" or "right to left" options in the menu bar to instruct which replica should override the other. Finally, click Go to replicate.

Unison will now replicate the entire content of local directory with the remote directory. After synchronization is complete, you will see the tick mark under Status section.

You can verify the contents of files and directories on both location.

The above method will sync the entire contents of local replica to remote replica. If you want to sync specific files or directories, create a custom profile and define the paths to synchronize as described in section titled 1.1. Unison profiles above.

3.3. Setup SSH key-based authentication (Optional)

Whenever you sync files between two different systems via SSH, you need to enter the password of the remote user. If you don't want to enter the password every time, you can setup password-less SSH authentication as described in the below link. Do not generate SSH key pair with passphrase. Otherwise, you still need to enter the password when synchronising files between systems.

• How To Configure SSH Key-based Authentication In Linux

3.4. Setup cron job (Optional)

We can sync files automatically at specific time interval using cron jobs.

For example, add the following entry in your crontab file to run synchronisation every day (It will run at 00:00):

0 0 * * * unison directory1/ ssh://ostechnix@192.168.225.52//home/ostechnix/directory1/

For more details about setting cron jobs, refer the following guide:

• A Beginners Guide To Cron Jobs

Getting help

I covered only the basics. This is just enough to setup a working synchronisation on your Linux system. If you want to learn more advanced usage, please refer the Unison help section.

To display the local documentation page, run:

$ unison -doc topics

You will see the list of available topics in the local documentation.

Documentation topics:
          about About Unison
         people People
          lists Mailing Lists and Bug Reporting
         status Development Status
        copying Copying
            ack Acknowledgements
        install Installation
       tutorial Tutorial
         basics Basic Concepts
       failures Invariants
        running Running Unison
            ssh Installing Ssh
           news Changes in Version 2.48.4

Type "unison -doc <topic>" for detailed information about <topic>
or "unison -doc all" for the whole manual

To see the detailed information about a topic, for example unison tutorial, run:

$ unison -doc tutorial

You can also display the whole Unison manual using the following command:

$ unison -doc all

I have also attached the official links of Unison at the end of this guide. Go thorough the documentation and familiarize yourself how Unison works. Once you learned enough, try to setup local and/or remote synchronization with Unison in production.

Conclusion

Unison allows you to easily and efficiently keep two or more systems completely synchronized with each other no matter what gets changed on any of the systems. If you want to  maintain two instances of a collection of files and directories on two systems up-to-date and identical, Unison is a good choice! 

Resources:

• Unison Website
• Unison Official Documentation
• Unison GitHub Repository