taters @ taters @piefed.social

Posts

1

Comments

5

Joined

7 mo. ago

No worries, glad I can help :)

If you are interested, I made a tool based around rsync. It's what I use to handle my backups plus transferring files and folders between my network connected devices.

https://codeberg.org/taters/rTransfer

With my tool, I'll make the following files in a directory:

    
.
├── .sync-computer-fs_01_root
├── .sync-computer-fs_02_boot
├── .sync-computer-fs_03_boot-efi
└── .sync-computer-fs_04_home


  

and then enter the Rsync Backup command information into the appropriate fields of each file. I can run my command with those files and it'll do a dry run followed by a confirmation to continue with the actual transfer.

There's a command option flag to reverse the transfer direction which can act as a "restore" in the case of an OS backup.

If you happen to give it a try, any feedback would be appreciated.

Whatever you choose to do, good luck and be sure to test everything more than you think you should. Both backups and restores. I know it's all to easy to fuck up data transfer in any direction :)

I've been working with the rsync command for a while now and I have created a pretty solid backup system on my own with that command. I've had some terrible luck using back/restore programs like Timeshift on Linux Mint and so I went out of my way to learn how to make a "simpler" backup/restore process for myself.

I am using Alpine Linux so the following commands are based around that file system. It's possible to modify the commands to fit your Operating System's file system by using the lsblk command to see all the mountpoints for your OS. Each mountpoint requires a separate rsync command.

For myself, I have four mount points. / for the root directory. For the boot directory, I will need a command for /boot/ and /boot/efi/ directories. Since my home directory is in a separate partition, I also need a command for /home/.

The following are all the expanded commands I use to backup my entire OS which is backed up to the /backup/ directory:

    
# BACKUP  
# /root/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=/backup/*' '--exclude=/boot/*' '--exclude=home/*' '--exclude=proc/*' '--exclude=sys/*' '--exclude=dev/*' '--exclude=tmp/*' '--exclude=run/*' '--exclude=mnt/*' '--exclude=media/*' / /backup/`  

# /boot/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /boot/ /backup/boot/`  

# /boot/efi/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /boot/efi/ /backup/boot/efi/`  

# /home/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' '--exclude=.cache/*' /home/ /backup/home/`  
  

  

The root partition is the most involved command so I'll break it down slightly. You can check out the rsync man page for more detailed information about each command option.

• --dry-run- is used to perform a test run. If you have any verbose options, it can be used to see what files will be moved without performing any actions. This is important for testing but I also use it to view or review any changes before I commit anything. This option must be removed if you want to commit any changes. I'm leaving it here incase anyone decides to blindly run this command. I don't want to be responsible for any lost data.
• --archive --acls --one-file-system --xattrs --hard-links --sparse are all used to preserve any file/folder attributes and permissions. --archive is commonly used, even for single files while the rest are more important for the operating system in the event of a restore. You can check the man page for more information as I've kind of forgotten their purposes over time.
• --verbose --human-readable --partial --progress is both for verbose outputs and for continuing in case a previous transfer was incomplete.
• --delete rsync will compare the source directory and the destination directory for any changes and will delete any files or directories in destination directory to match the source directory.
• --numeric-ids is used to preserve ownership in number form. I find it useful since I keep multiple device backups on an external SSD.
• --exclude=... This excludes files or directories from being synced. For the / directory it's important to exclude /backup/. If you don't, you will end up recursively saving to the /backup/* directory until your storage is full. /boot/* home/* are excluded because they will be handled by the following rsync commands. proc/* sys/* dev/* tmp/* run/* mnt/* media/* are directories that are used to run the operating system or devices and will not be required after a system reboot.
• / /backup/ is two parts. / is the source directory, what you want backed up. /backup/ is the destination directory. It's where you want your data to end up.
• --exclude=lost+found is not important and can be excluded
• --exclude=.cache/* can also be excluded from the home directory if you want to save space.

You'll most likely need to create the /backup/ directory in order to perform the backup commands.
Running the commands in order as listed, it will first perform a backup on the root directories while creating empty directories for the following commands. After that, each command will be able to complete it's transfer.

Restoring is as simple as using the same command but reversing the Source and Destination directories. If backing up root was / /backup/, then to restore you now use /backup/ /. After performing a restore, it's wise to reboot your system.

    
# RESTORE  
# /root/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=/backup/*' '--exclude=/boot/*' '--exclude=home/*' '--exclude=proc/*' '--exclude=sys/*' '--exclude=dev/*' '--exclude=tmp/*' '--exclude=run/*' '--exclude=mnt/*' '--exclude=media/*' /backup/ /`  

# /boot/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /backup/boot/ /boot/`  

# /boot/efi/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' /backup/boot/efi/ /boot/efi/`  

# /home/  
`rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --compress --delete --numeric-ids '--exclude=lost+found' '--exclude=.cache/*' /backup/home/ /home/`  
  
  

For docker containers, I usually shut them all down before a backup, perform the backup and start the containers again with the following commands:

    
# CONTAINERS DOWN  
docker stop --timeout 30 $(docker ps -a -q)  

# PERFORM BACKUP  

# CONTAINERS UP  
docker start $(docker ps -a -q)  
  
  

For a restore with docker containers, I usually shut down the containers, perform the restore and reboot my system. If you have any containers that require building, you may need to spend time rebuilding after a restore. I haven't figured out how to address that issue yet. It may be helpful to run the command docker system prune -a && docker volume prune -f to remove all docker containers before a restore. Sometimes docker containers remain after performing a restore and must be deleted by hand. You can see which directories can be removed by performing a dry run and looking for the delete errors.

You don't have to use any of this but for me I at least got some insight into how backing up and restoring a file system works with the help of rsync. Hopefully if can give you some insight as well for your needs.

I use these commands near daily since I like to keep my systems as neat and organized as possible. Often I'll create a backup, do my experimenting, perform a restore, make any changes I want and perform another backup with the latest changes. I've had more consistent results going this route compared to using other backup programs.

I know there is a way to create smaller backups using links but I don't think I do that. I prefer a full backup copy anyways since I run my systems in a more minimal way. Space isn't an issue for me to worry about.

I agree with Damage, a Docker containers may be more flexible compared to HomeAssistant add-ons. I currently have two Raspberry Pi's. A Pi4 with a HomeAssistant container and a Pi5 with a simple file server, and a copy of Wikipedia and an unfederated PieFed instance which are all run from containers.

I bought my Pi5 used and it already had a fan installed and judging by how often the fan runs, it seems like a good idea to have. Especially since it's more powerful compared to the Pi4.

If you're comfortable working with terminal commands, you could simply use rsync for creating and maintaining backups. Works great for remote backups and file systems too. I've automated backups for my phone (Android with Termux) both Pi's and my laptop. All those backups get sent to a partition on my laptop. Then that partition is completely backed up to an external SSD storage. Rsync compares files between the source folder and destination folder so that only the differences are transferred and sends less data. Tools like rsnapshot may simplify the whole process if you don't have the time or energy to fiddle with rsync.

I can share my docker compose file for HomeAssistant and some examples of my rsync commands if that's a path you're interested in.

I enjoy it when people go to absurd lengths to not spend money. There's so much room for creativity within what we tools we already have available to us.

I also think that it's important to try new things and see what we can learn from it. Trying and failing is all a part of learning. And we need more ideas and knowledge out there if the idea of open source is to spread and gain popularity.

If a project is not interesting or important to me, that doesn't mean that someone else can't benefit from it. I think if one person finds a project useful or interesting then that's worth sharing alone. And if only one other person wants to contribute their time to that project, then that's still pretty awesome.

I ended up using rsync to do my backups. I have a laptop, an Android phone with Termux, a HomeAssistant docker image on a Raspberry Pi 4 and a home PieFed instance on a Raspberry Pi 5.

Each RPi board will create a complete backup on it's on storage. I'll then make a copy of each backup to my laptop. And finally another copy from my laptop to an external usb storage device. I also made a specific folder on my Android phone for the purpose of syncing with my laptop.

What I like about rsync is that it can be made to only transfer any changed files and not everything each time.

Since I use docker images on both my RPi boards, I made a script that shuts down all the containers before make a backup copy and finally starting the containers again. I even made a script to do something similar to restore from the saved backup.

Took a lot of trail and error to get them working but I am pretty happy with it. I have the scripts here if anyone is interested. I labelled what I did but didn't really leave comments explaining things. They are pretty simple anyways.

I could have used rsnapshot but learning how to use rsync has been interesting enough to me.