Skip to content

Automated, Distributed File Syncing and Backups Using Syncthing and Rsync

In this guide, you'll learn how to use Syncthing for local, distributed file syncing across multiple devices, along with Rsync (for macOS, Linux) and Windows' native backup utility for regularly backing up your important data.

Goals

The first goal is to have your important, frequently accessed data, such as music, camera photos etc. synced and available on other devices such as smartphones, laptops, and desktops.

The second goal is to keep this synced, distributed data regularly backed up to an external storage device such as a pen drive. This is important because while you do have your data in multiple places simultaneously, Syncthing is not a backup solution.

For example, if you are syncing your phone's "Camera photos" folder to your desktop, and some photos on the phone are accidentally deleted while Syncthing is running on both the devices, it'll notice the changes and happily delete the synced copies of those photos on your desktop as well.

Therefore, backups are essential.

Syncthing setup

Download and install Syncthing on your devices

Example setup: share a music folder between an Android phone and a desktop PC

As a demo, we'll set up a folder called "DesktopMusic" in your user's Home directory on a desktop to sync with a folder called "SyncedMusic" on an Android device. Once this is set up, adding more folders and/or devices will be relatively straightforward.

  1. On the desktop, create a folder called DesktopMusic in your user's Home directory.
  2. On the smartphone, create a folder called SyncedMusic.
  3. Run Syncthing on your desktop and open the Syncthing app on your smartphone. Make sure both the devices are connected to the same network.

  4. On your desktop, you'll see a screen like this: syncthing main screen

    Click the ➕ Add Folder button in the left column.

  5. In the dialog box that pops up, go to the Folder Path section and enter ~\DesktopMusic if you are on Windows, or ~/DesktopMusic on Linux/macOS. The ~ character automatically translates to your current user's Home directory.

    Note for Linux users: Linux filesystems are case-sensitive—document1.txt is NOT the same as Document1.txt. syncthing add folder

    As in the screenshot, give this folder an appropriate "Folder Label" and click ✅Save.

  6. Click the ➕ Add Remote Device on the bottom-right of the main screen. You'll see this message: syncthing add device

    This link is the device ID of your smartphone. Click on it to add it. Optionally, give the device a name in the "Device Name" section.

  7. In the same dialog box, click on the "Sharing" tab on the top and make sure the DesktopMusic folder that we created is selected. Click ✅Save once you are done.

  8. Soon, you'll see a notification on your smartphone that says "Device XYZ wants to connect". Click Accept. After accepting, you will soon see another notification saying "Device XYZ wants to share folder 'DesktopMusic'". Click Accept again. syncthing accept folder

  9. Tap the Directory field, and navigate to the SyncedMusic folder that you created in point 2 and select it. Once done, tap the ✅ tick button on the top right. If the DesktopMusic folder on your desktop has any files in it, they will sync with the smartphone. syncthing folder added

That's all you need to know for basic file syncing with Syncthing. You can follow the same procedure to add more devices and folders to your setup. Ideally, you will keep the Syncthing service running perpetually on your desktop and run it on your mobile devices as and when necessary (Syncthing can be a battery hog).

For further reading and more advanced configurations, refer to the excellent Syncthing documentation. Of special interest are the concepts of Send-only folder (for devices where "master copies" are kept) and Receive-only folder.

Sample folder structure

In a sample household with one desktop, one smartphone and one laptop, you can try to use a structure like this on your desktop:

HOME/
├ Music      # Synced in Send-only mode with smartphone and laptop
└ MobileStuff     # Parent folder for files synced from your smartphone
  ├ CameraPhotos   # Synced in Receive-only mode with smartphone
  ├ WhatsAppMedia  # Your phone's `/storage/Whatsapp/Media` folder,
                    # contains images,videos etc. shared on WhatsApp.
  └ Similarly, any other folders in your phone that you'd like to sync.

The advantage of a structure like this is that once your desktop or laptop has finished syncing with your smartphone, it is very easy to make backups of everything important, which we will learn in the next section.

Backups

It is recommended to keep a copy of your important files on at least two external drives. One of them will be your daily backup drive—you insert it in your desktop or laptop once a day and make a backup. It will save your skin in case your computer's hard disk dies or if you accidentally delete your synced folders.

The other drive is a weekly/monthly backup—do the same thing as the first one on it once per month, and store it in a different location than your first backup drive. This will save you in case your first backup drive either fails or your baby throws it across the room.

Backups on Windows

Backups on Windows are relatively straightforward using the built-in Backup and Restore functionality.

Insert the backup drive in your computer, search for Backup Settings in the start menu, and use the GUI to configure which folders to back up, how often, and to which drive. Do the same with your other backup drive and you're done.

Backups on macOS and Linux with Rsync

Note: If you are using macOS, it is recommended to use the excellent, intuitive Time Machine utility to manage your backups and call it a day. However, if you are hardcore and wish to control every single aspect of your backups (or if you just prefer the command line), then continue reading.

Rsync is a command-line utility for synchronizing files and folders between disks and (optionally) across the network. It comes preinstalled on macOS and most Linux distributions, so there is no need to install additional software.

These instructions assume you have set up your folders according to the sample folder structure. If not, make sure to substitute accordingly. Again, pay attention to the case when working on Linux—document1.txt is NOT the same as Document1.txt.

Pre-setup

Note for Linux users: Different Linux distributions and flavors have different ways to mount external drives and use different directories as mount points. Please refer to your distribution's documentation to know where external disks are mounted. Other than this detail, the rest of the documentation applies equally to macOS and Linux.

Insert your external hard disk and open Finder. Make a note of what the drive is called. In my case, it is called Transcend:

external disk in finder

Navigate to the disk and create a folder called backups.

Your external disk is mounted in a folder called /Volumes. To locate it, open Terminal, type in the command ls /Volumes and press Enter/Return. You will see an output like the following:

$ ls /Volumes
Macintosh HD    Preboot     Transcend

As noted earlier, my external disk is called Transcend. Therefore, we infer from this command that it is mounted under /Volumes/Transcend. Make a note of this path.

Use the mkdir (make directory) command to create a backups directory in the external drive:

mkdir /Volumes/Transcend/backups

Taking backups

Rsync commands are structured like:

rsync [<options>] <source> <destination>

As a sample backup plan, here is how to back up the MobileStuff directory:

rsync -avu --progress --modify-window=1 ~/MobileStuff /Volumes/Transcend/backups

Here's an explanation of what is going on in the above command:

  • The -avu option has three parts:
    • The a option tells Rsync to sync in "archive" mode.
    • The v option increases verbosity and gives more details on what is going on during the transfer.
    • u tells Rsync to skip any files that already exist on the destination during subsequent backups. However, if the file has been modified at the source, the destination file will be replaced with the newer version.
  • --progress shows progress during transfer.
  • ~ stands for your user's Home directory. /Users/<your username> on macOS, and /home/<your username> on Linux.
  • /Volumes/Transcend/backups is the directory that the files will be synced to. The "Transcend" part will probably be different on your system. Substitute accordingly.
  • --modify-window=1 is a special parameter that you need to pass if you are syncing from a macOS/Linux system to an external drive formatted in Microsoft's FAT filesystem family. Most flash drives are formatted in this filesystem by default. If you are certain that your external drive does not use this filesystem, you can omit this parameter. If you are not certain, leave it in.

This will create a folder called MobileStuff in the destination folder /Volumes/Transcend/backups and sync it with the source folder. Any time you modify or add files to the source folder and rerun the command, it will sync all the latest changes.

Similarly, you can sync other directories:

rsync -avu --progress --modify-window=1 ~/Music /Volumes/Transcend/backups

Creating a backup script

Instead of running the (rather lengthy) Rsync command(s) every time you want to do a backup, you can create a script to simplify the process. In a Terminal window, create a directory called bin:

mkdir ~/bin

In this folder, create a file called backup.sh and add all the commands that you want to run. For example:

rsync -avu --progress --modify-window=1 ~/MobileStuff /Volumes/Transcend/backups
rsync -avu --progress --modify-window=1 ~/Music /Volumes/Transcend/backups
rsync -avu --progress --modify-window=1 ~/Documents/SuperImportantDocs /Volumes/Transcend/backups

Save the file and make it executable using the chmod command:

chmod +x ~/bin/backup.sh

Add the ~/bin directory to your PATH by editing your .bash_profile:

nano ~/.bash_profile

Add the following line to this file and save it:

export PATH=$PATH:$HOME/bin

Close the terminal window and open a new one. Now, any time you run backup.sh in a terminal, it will run all the commands you just saved.

Further reading