Overview
One of CIRCLs tasks is to protect the citizens, companies and all other types of organizations, within, but not limited to Luxembourg from malware, phishing and other digital threats. During the last years, one specific threat was - from an attacker’s perspective - very successful. At the same time it turned out to be devastating (unrecoverable loss of information) or at least expensive and problematic for the victims. We are talking about the rise and success of Crypto Ransomware, which we covered in other articles already:
- TR-41 - Crypto Ransomware - Proactive defenses and incident response (en) (frde)
- Growing Crypto Ransomware campaigns in Luxembourg
- A new wave of crypto ransomware targeting Luxembourg(fr)
Crypto Ransomware is a kind of malware that - unknowingly to the user - efficiently and effectively encrypts files without the user knowing the decryption key, rendering these files unusable by the user. The criminal on whose initiative the Crypto Ransomware has been executed is requesting a certain amount of money in exchange for the key. In most but not all cases the attacker delivers the key. The price for the key varies from several hundred Euros to several hundred thousand Euros.
If we ignore for a moment the possibility that someone doesn’t fall for the threat or has effective countermeasures in place, the post-infection scenario could be the following: 1. Ransomware encrypts the local files 2. Ransomware encrypts all connected files (local, network) 3. Ransomware encrypts accessible backups 4. Bonus: Attacker has already access to the infrastructure and the Backup server, from which he deletes backup files in unrecoverable low-level filesystem operations.
While 1) and 2) are already bad on their own, scenario 3) destroys the last resort of being able to restore a backup without having to collaborate with the criminals (by paying the requested ransom). It can only be circumvented with an offline backup system, where the backup location is only connected when needed or when the backup is incrementally written, never overwriting old files. In any case, if the Backup server file system is exposed (and writeable) to an infected client during the encryption process, the files will be lost.
Side note: of course, the same applies if the attacker has any administrative access to the Backup server, which he can use to delete all backups (scenario 4). Therefore it is essential to secure the Backup machine with utmost diligence. This modus operandi has been witnessed in more targeted attacks.
It is our intention to show that it is possible to use standard Unix/Linux Open Source environments to create a Backup system that is connected to the backup client and at the same time is creating read-only backups. In addition, it creates revisions of the incremental backups that can be rolled-back to any previous backup. As a complementary measure, the incremental backup can be easily copied on a disconnected and offline infrastructure to ensure an air-gap between the potential attackers and the disconnected backup read-only files.
The level of innovation of this proposed system is not very high. However, the combination of these tools in order to achieve a solution against Ransomware deleting/encrypting Backup files is new. While this article focuses on end-user Backup solutions, we know that it’s not easily implementable for everyone, even with a technical background.
But we hope that we can challenge producers of Network Attached Storage (NAS) solutions as well as Backup solutions to implement similar systems to the proposed solution to make it available for their customers in order to help them doing what we strive for: protecting the users.
History & Recognition
Early in September 2018, team CIRCL was discussing in an ad-hoc manner about doing read-only back-ups on squashfs. We were used to it for forensic purposes, but we developed and discussed the idea to make the read-only filesystem available for restoration purposes and backup to to a different location. For building a proof-of-concept, the authors of the document looked into and used the combination of SquashFS and UnionFS, which turned out to be successed by the better maintained Aufs. What happened next was the discovery of SquashFu, an (in terms of Internet history) ancient Backup system based on exactly the tools we discovered to support our task. The focus in 2010 was the ‘squashing’, that means the compression capabilities of SquashFS in order to save space. The almost one decade old (the project was started on Jan 8, 2010) project worked out of the box on a recent Ubuntu.
Thank you Dave Reisner for having implemented this great idea so early! The focus has changed, but the idea and code lives on :)
What we have done since is to modify the code from Dave in order to fit our needs and to bring everything into the new context, like the integration with a Samba/CIFS server.
Disclaimer
Team CIRCL has made every attempt to ensure the accuracy and reliability of the information provided in this document. However, the information is provided “as is” without warranty of any kind. Team CIRCL does not accept any responsibility or liability for the accuracy, content, completeness, legality, or reliability of the information contained on this website.
Team CIRCL shall not be liable for any loss or damage of whatever nature (direct, indirect, consequential, or other) whether arising in contract, tort or otherwise, which may arise as a result of your use of (or inability to use) Squashfu, or from your use of (or failure to use) the information in this document.
In any case, back up your backups or use an alternative/secondary backup system in order to improve your resilience even more!
Software
The proposed system is based on the following Open Source software: - SquashFS - Aufs - Rsync - SquashFu based on and forked from SquashFu - Samba - Ubuntu
About SquashFu
What is SquashFu?
squashfu is a Bash based backup utility that uses the flexibility of rsync, squashfs, and aufs to create incremental backups and offer limited compression. The initial goal was to create a backup solution which provides incremental backups and compression, and which also provides an easy way to roll back.
How does SquashFu work?
Internal technical description:
A directory structure is created as follows (with some terminology included):
backup_root/
|- seed.sfs <-- squash, or seed
|- ro/ <-- squash mount point
|- rw/ <-- union mount point
|- .bins/ <-- incrementals
|-1/
| .....
| .....
| .....
| .....
| .....
|-n/
/var/
|- lib/
|- .squashfu.inv <-- bin inventory list (or binventory)
seed.sfs is created from an initial backup and compressed using SquashFS, which is simply a read only filesystem which focuses on compression. It’s mounted, using a loopback device, on ro/.
At the time of the backup, the next available bin is determined, created, and logged to an inventory sheet with a timestamp. A union is created with all the available bins, mounted in reverse chronological order on top of the seed (newest to oldest) on rw/. At this point, the union represents the state of your files at the end of the last backup. The newest branch is marked as read/write, and rsync is called. Because this top branch is the only writable location in the union, the files rsync generates with the -u (update) flag are placed into this branch. The backup finishes, and the union and seed are unmounted.
At this point, Squashfu ensures compliance with the user’s settings of MAX_BINS. If the current number of used bins exceeds this value, a new seed is generated. The number of old incrementals merged into the new seed is determined by the difference between MAX_BINS and MIN_BINS in the config file. In this way, you always have MIN_BINS available to roll back to, but you’re not forced to recompress your seed at every backup – an operation that may take a long time depending on how big your backup source is.
If and when you want to roll back, execute Squashfu with the -R action, and supply the number of bins you want to roll back. The bins are ordered chronologically, and the oldest “number_of_bins - bins_to_rollback” are mounted on the union mount point.
WARNING: You should not, under any circumstances, add or remove files contained in the bins, nor should you alter your binventory’s time stamps. Doing so many result in your cat being set on fire or your backups being destroyed.
Origin
SquashFu’s original location is at SquashFu and was written by Dave Reisner d@falconindy.com and released in 2010. Dave was so kind to license it under MIT license so that it can be reused in the future.
Installation and configuration
Clone SquashFu
cd ~
git clone https://github.com/CIRCL/SquashFu.git
cd SquashFu
sudo cp squashfu.conf /etc/
Setup directories
sudo -p mkdir /backup-rw/your-username
sudo chown your-username /backup-rw/your-username
sudo mkdir /backup
Edit SquashFu config file
Edit the config file in order to let SquashFu know where we want to store the backups (BKUP_ROOT). Also, let it know about what we want to backup (INCLUDES). The hooks (PREHOOKS, POSTHOOKS) contain commands to stop and start Samba if the underlying locations have been modified.
sudo vi /etc/squashfu.conf
so that it reflects the following settings:
BKUP_ROOT="/backup"
INCLUDES=(
/backup-rw
) #end includes
EXCLUDES=(
) #end excludes
PREHOOKS=(
"sudo /etc/init.d/samba stop"
)
POSTHOOKS=(
"sudo /etc/init.d/samba start"
)
If you intend to use the Samba shares and want to rely on the automtic restart of the service initiated by SquashFu, you might want to consider to add the following line:
# Execute pre and post hooks by default
HOOKS=true
In this case the hooks are executed all the time and the option -h is not necessary to be specified.
Edit Samba config file
Through Samba, two network shares will be exposed: one for writing backus to (backup-rw) and one for browsing the current or previous backups (backup-restore).
sudo vi /etc/samba/smb.conf
so that it reflects the following settings:
[backup-rw]
comment = Initial Backup location
path = /backup-rw/your-username
browseable = yes
read only = no
guest ok = no
create mask = 0700
write list = your-username
[backup-restore]
comment = Writeable Backup location
path = /backup/rw/backup-rw/your-username
browseable = yes
read only = yes
guest ok = no
Start Samba
sudo /etc/init.d/samba start
Copy a bunch of files to the network share backup-rw
Mount the share from your computer and put some files there or just copy files in /backup-rw/your-username locally.
Initiate first backup
Take the first backup of your files
sudo ./squashfu -h -B
:: Looks like this is your first time running SquashFu. Is this correct? [y/N] y
:: Creating seed from sources (this may take a while)
:: Mounting squash
:: Seed creation finished. It has been mounted at /backup/ro if you would like to make sure the proper files are included
:: Backup requested at 2018-09-05 13:17:03+02:00
:: Executing pre-hooks
[ ok ] Stopping samba-ad-dc (via systemctl): samba-ad-dc.service.
[ ok ] Stopping smbd (via systemctl): smbd.service.
[ ok ] Stopping nmbd (via systemctl): nmbd.service.
uid=0(root) gid=0(root) groups=0(root)
:: Creating new bin
DEBUG :: Next available bin = 1
DEBUG :: Asked to create new bin: 1
:: Mounting union
DEBUG :: Requested to mount bins: 1
DEBUG :: rsync -Rua --delete --stats /backup-rw /backup/rw
:: Creating new incremental
Number of files: 162 (reg: 159, dir: 3)
Number of created files: 162 (reg: 159, dir: 3)
Number of deleted files: 0
Number of regular files transferred: 159
Total file size: 27,475,822 bytes
Total transferred file size: 27,475,822 bytes
Literal data: 27,475,822 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 27,495,908
Total bytes received: 3,064
sent 27,495,908 bytes received 3,064 bytes 54,997,944.00 bytes/sec
total size is 27,475,822 speedup is 1.00
DEBUG :: Found 1 bins
:: Unmounting union
:: Unmounting squash
:: Executing post-hooks
[ ok ] Starting nmbd (via systemctl): nmbd.service.
[ ok ] Starting smbd (via systemctl): smbd.service.
[ ok ] Starting samba-ad-dc (via systemctl): samba-ad-dc.service.
:: Backup completed at 2018-09-05 13:17:07+02:00
Load the last backup
Load the backup. -R specifies the number of backups to roll back. In this case, we want the current backup (0), and not a previous version (1-n).
sudo ./squashfu -h -R 0
:: Executing pre-hooks
[ ok ] Stopping samba-ad-dc (via systemctl): samba-ad-dc.service.
[ ok ] Stopping smbd (via systemctl): smbd.service.
[ ok ] Stopping nmbd (via systemctl): nmbd.service.
uid=0(root) gid=0(root) groups=0(root)
:: Mounting squash
:: Mounting union
DEBUG :: Requested to mount bins: 1
:: Executing post-hooks
[ ok ] Starting nmbd (via systemctl): nmbd.service.
[ ok ] Starting smbd (via systemctl): smbd.service.
[ ok ] Starting samba-ad-dc (via systemctl): samba-ad-dc.service.
:: You have rolled back to 2018-09-05 13:17:06+02:00
:: Your files can be found at '/backup/rw'
Validate it can be accessed via SMB/CIFS
Can you mount the share and validate the files you expect are there?
Add more files to the network share
Copy more files into /backup-rw/your-username, either locally or through the network share backup-rw.
Initiate a new backup
Take a new backup
sudo ./squashfu -h -B
:: Backup requested at 2018-09-05 14:42:25+02:00
:: Executing pre-hooks
[ ok ] Stopping samba-ad-dc (via systemctl): samba-ad-dc.service.
[ ok ] Stopping smbd (via systemctl): smbd.service.
[ ok ] Stopping nmbd (via systemctl): nmbd.service.
uid=0(root) gid=0(root) groups=0(root)
:: Unmounting union
:: Creating new bin
DEBUG :: Next available bin = 6
DEBUG :: Asked to create new bin: 6
:: Mounting union
DEBUG :: Requested to mount bins: 6 5 4 3 2 1
DEBUG :: rsync -Rua --delete --stats /backup-rw /backup/rw
:: Creating new incremental
Number of files: 550 (reg: 547, dir: 3)
Number of created files: 4 (reg: 4)
Number of deleted files: 1 (reg: 1)
Number of regular files transferred: 5
Total file size: 224,975,427 bytes
Total transferred file size: 365,328 bytes
Literal data: 365,328 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 389,735
Total bytes received: 164
sent 389,735 bytes received 164 bytes 779,798.00 bytes/sec
total size is 224,975,427 speedup is 577.01
DEBUG :: Found 6 bins
:: Unmounting union
:: Unmounting squash
:: Executing post-hooks
[ ok ] Starting nmbd (via systemctl): nmbd.service.
[ ok ] Starting smbd (via systemctl): smbd.service.
[ ok ] Starting samba-ad-dc (via systemctl): samba-ad-dc.service.
:: Backup completed at 2018-09-05 14:42:29+02:00
Check the Backups
The next command shows you the already taken backups (6 in the example)
sudo ./squashfu -Q
:: SquashFu Usage Report
Bin ID Date Created Size
6 2018-09-05 14:42:26+02:00
5 2018-09-05 13:36:21+02:00
4 2018-09-05 13:29:01+02:00
3 2018-09-05 13:27:57+02:00
2 2018-09-05 13:24:08+02:00
1 2018-09-05 13:17:06+02:00
Incremental Total 216M
backup-server-seed.sfs 24M
Grand Total 240M
Load a previous backup
Now we load a previous backup:
sudo ./squashfu -h -R 1
:: Executing pre-hooks
[ ok ] Stopping samba-ad-dc (via systemctl): samba-ad-dc.service.
[ ok ] Stopping smbd (via systemctl): smbd.service.
[ ok ] Stopping nmbd (via systemctl): nmbd.service.
uid=0(root) gid=0(root) groups=0(root)
:: Unmounting union
:: Mounting union
DEBUG :: Requested to mount bins: 5 4 3 2 1
:: Executing post-hooks
[ ok ] Starting nmbd (via systemctl): nmbd.service.
[ ok ] Starting smbd (via systemctl): smbd.service.
[ ok ] Starting samba-ad-dc (via systemctl): samba-ad-dc.service.
:: You have rolled back to 2018-09-05 13:36:21+02:00
:: Your files can be found at '/backup/rw'
Check the loaded backup content is different to the content of the current backup share
Load the last backup
Load the last backup again
sudo ./squashfu -h -R0
:: Executing pre-hooks
[ ok ] Stopping samba-ad-dc (via systemctl): samba-ad-dc.service.
[ ok ] Stopping smbd (via systemctl): smbd.service.
[ ok ] Stopping nmbd (via systemctl): nmbd.service.
uid=0(root) gid=0(root) groups=0(root)
:: Mounting squash
:: Mounting union
DEBUG :: Requested to mount bins: 6 5 4 3 2 1
:: Executing post-hooks
[ ok ] Starting nmbd (via systemctl): nmbd.service.
[ ok ] Starting smbd (via systemctl): smbd.service.
[ ok ] Starting samba-ad-dc (via systemctl): samba-ad-dc.service.
:: You have rolled back to 2018-09-05 14:42:26+02:00
:: Your files can be found at '/backup/rw'
Check the loaded backup contains the same content as the current backup share
Make it automatic
If everything looks fine and works as expected, the backup can be scheduled to be done automatically, for instance every night at 1 o’clock.
sudo -i
crontab -e
Add the following line:
0 1 * * * /home/your-username/SquashFu/squashfu -h -B
Other interesting commands:
Merge old backups, but keep 5 snapshots
sudo ./squashfu -h -C
Delete a backup
sudo ./squashfu -h -D <bin number>
Restore a file interactively
sudo ./squashfu -h -G <path>
Unmount all filesystems managed by SquashFu
sudo ./squashfu -h -U
License, Authors, Contributors
License
MIT License (as in attached LICENSE file) Copyright (c) 2018 Dave Reisner Copyright (C) 2018 Sascha Rommelfangen Copyright (C) 2018 CIRCL Computer Incident Response Center Luxembourg (SMILE gie).
Authors, Contributors
- Dave Reisner d@falconindy.com
- Sascha Rommelfangen sascha.rommelfangen@circl.lu
Classification of this document
TLP:WHITE information may be distributed without restriction, subject to copyright controls.
Revision
- Version 1.0 20180906