crashplan – chris nelson dot ca

CrashPlan and CrashPlan Pro on Synology using Docker

 How To  26 Responses »
May 222016
 

Important Note
Code42 has removed headless functionality in the latest v6 release of CrashPlan for Small Business (formerly CrashPlan Pro). Existing docker instances may still continue to run for a while, but this ultimately marks the death of any sort of CrashPlan backup for Synology.
https://support.code42.com/CrashPlan/6/Get_started/Changes_to_the_Code42_app_in_version_6

I recommend using BackBlaze B2 with Synology’s Cloud Sync application. There are some drawbacks (e.g. you can’t cloud sync the same folders with different services), but ultimately it is fairly priced and works well.

Archive Content:
Synology introduced Docker to the DSM in version 5+, which has opened up new ways of running some applications in Synology. Not all Synology boxes have enough resources to run docker, check if your Synology NAS can run Docker here.  Using Mike Tabor’s excellent tutorial post I was able to get CrashPlan up and running on my 713+ DiskStation. However, his tutorial only covers the basic CrashPlan application, and not CrashPlanPRO. I’ve written up a quick and dirty guide to get CrashPlan or CrashPlanPRO working on docker in Synology.

Prepare the Synology NAS for Docker

  • Login to Synology, go to Package Center
  • Uninstall CrashPlan from Package Center (if previously installed)
  • Install Docker in Package Center
  • Connect to Synology via SSH as a user with Admin privileges

Get the latest Docker Images for CrashPlan or CrashPlanPRO

First, we will pull a CrashPlan or CrashPlanPRO image from docker hub.  Note that if connected via DSM6, we will need to start the session with “sudo su”, or add “sudo” before each of the docker commands:

If using CrashPlan

If using CrashPlanPRO:

Create and run CrashPlan container using the image.

If multiple volumes need to be specified, add another -v statement after the first one e.g. “-v /volume2/:/volume2”

If using CrashPlan:

If using CrashPlanPRO:

Wait until the container has fully started up (typically less than 2 minutes)

Get the GUID from CrashPlan

In the same SSH session, enter into a bash session for the container:

The command prompt will now take place inside the container. You may see an error which is safe to ignore for now: “ERRO[0000] Unable to connect to local syslog daemon”. In the bash session, enter the following command to get the headless key

It will output something similar to the following. The first number is the port used, 4243. After the comma, the guid key is listed, then the IP for the Synology NAS. In this example below, the “4fx72835-3fcd-4319-afd5-698212c77ba7” key should be recorded to connect to the system with CrashPlan from a desktop machine

4243,4fx72835-3fcd-4319-afd5-698212c77ba7,0.0.0.0

Update the workstation to connect to CrashPlan running in Docker

.ui_info

If using Windows, update C:\ProgramData\CrashPlan\.ui_info with the guid recorded in the command above, and the IP address of the Synology NAS. If on a Mac, the path is /Library/Application Support/CrashPlan/.ui_info.  For example:

4243,4fx72835-3fcd-4319-afd5-698212c77ba7,10.0.0.99

ui.properties

If using Windows, update C:\Program Files\CrashPlan\conf\ui.properties and ensure serviceHost is enabled and pointing to the Synology NAS. On a Mac, this path is typically /Library/Application Support/CrashPlan/conf/.ui_info.  For example:

serviceHost=10.0.0.99

Connect to CrashPlan

After connecting successfully to the CrashPlan instance, adopt a previous computer (if applicable) to ensure the backup history and settings are retained.

If you have large backup sets and memory needs to be adjusted, double click the CrashPlan logo in the top right of the program and apply a custom command. In the example command below, the memory usage is increased to 2GB in order to work with very large file sets. Do not increase this value beyond what is available on the Synology NAS.  More information on this command can be found here.

java mx 2048, restart

Troubleshooting

CrashPlan on Docker can’t see my files

Sometimes the mapping for the root volume doesn’t seem to stick for sub shares in the container. To fix this, stop the container and edit it in Docker. Add each share used in CrashPlan backups to the container and mount them the same way they were referenced originally. For example, if CrashPlan backs up files on a share named “Photo” on volume1, then add that share to the container with a mount path of “/volume1/Photo”.

CrashPlan Updated to v4.8 and now my docker image is broken

The following reply by reddit user ajr6037 has a solution: https://www.reddit.com/r/synology/comments/550806/best_crashplan_docker_image/d87beo9

Fixing CrashPlan 4.3.0 on Synology

 How To  407 Responses »
Jul 022015
 

CrashPlan Issue For Synology – Fixing the 4.3.0 update

If your Synology NAS can run docker, you may want to consider an alternative to the CrashPlan package. See here: CrashPlan and CrashPlan Pro on Synology using Docker

Note, see bottom of page for information on DSM 6 and CrashPlan 4.6.0

Recently CrashPlan updated to version 1427864400430, which broke the CrashPlan package Patters had created for Synology installations again. They then (July 2nd?) released 1427864410430 as well, which means that if your CrashPlan installation on Synology broke with version 1427864400430, you will need to upgrade, wait for it to break again, and then fix with version 1427864410430. In general the fix process has been the same over the last few breaks, and I’ve outlined those steps below for the latest version.

There is one major change introduced from CrashPlan. Starting with the 1427864400430 update, a guid (think of it as a key) is needed to securely log into the headless client. Thanks to Mathew Stocks, I’ve made those instructions available as well.

The log for the failing CrashPlan package would look something like this for a failing 1427864400430 upgrade:

Upgrades available at central.crashplan.com:443 Downloading a new version of CrashPlan. Download of upgrade complete – version 1427864400430. Installing upgrade – version 1427864400430 Upgrade installed – version 1427864400430 CrashPlan stopped, version 4.2.0, GUID 559750658046558476

If the package was attempted to start again, it would write a line similar to the following, and then stop:

Synology repairing upgrade in /var/packages/CrashPlan/target/upgrade/1427864400430.

Note that the following instructions are for the 1427864410430 upgrade, which is the latest (as of July 2nd, 2015) update. If your Synology didn’t make it to this version, run the following steps, but substitute “1427864400430” where there are any references to “1427864410430”.

Connect as root to SSH

First, connect to Synology using SSH and the root account (uses admin password).  Connecting as the admin account will give you permission failures!

In SSH, run commands to extract the update

In SSH, run unique command to cleanup the package

The next command will be different for everyone, as part of the file path is randomized.  In this case, you will need to see what your path is first by running ls, and then editing the second statement below, replacing whatevervalue with the rest of  your path:

Start CrashPlan package, check log

After the above completes, you should be able to start the CrashPlan package again in Synology and get it running once more. The log should show:

CrashPlan started, version 4.3.0 GUID 559750658046558476

Get the headless key from the .ui_info file

While connected to SSH as root, run the following command to get the headless key file. The funky formatting of this command is because the file does not contain the special newline character:

The command will output a line similar to the following (everyone’s will be different):
4243,13d436c0-230a-4242-b258-574e60e62a9f

Update .ui_info on the computer connecting to Synology

On the computer where you are connecting to CrashPlan on Synology, go to the .ui_info file under the CrashPlan directory. NOTE: I had to manually upgrade my computer’s version of CrashPlan (via https://www.code42.com/crashplan/thankyou/) in order to see the .ui_info file.

Typically on Windows, this will be found under:
C:\ProgramData\CrashPlan\.ui_info

Or on a Mac in the following location (need to have finder set to show hidden files):
/Library/Application Support/CrashPlan/.ui_info

Enter in the new key in the file, replacing the current value, and save it. You should now able able to connect to the headless client. Note that you may need to also update the values in ui.properties again if you had to upgrade CrashPlan.

CrashPlan 4.6.0

There are some tools referenced in the comments for this page which might make upgrades easier for some people. But I’ll also give the latest files & paths using the original instructions above – the following commands can be used for v4.6.0:

unzip -o /var/packages/CrashPlan/target/upgrade/1435813200460_382.jar "*.jar" -d /var/packages/CrashPlan/target/lib/ unzip -o /var/packages/CrashPlan/target/upgrade/1435813200460_382.jar "lang/*" -d /var/packages/CrashPlan/target/

Find latest package folder:
ls -l /var/packages/CrashPlan/target/upgrade/1435813200460_382.*

Replace the whatevervalue with the folder name:
mv /var/packages/CrashPlan/target/upgrade/1435813200460_382.whatevervalue/upgrade.sh /var/packages/CrashPlan/target/upgrade/1435813200460_382.whatevervalue/upgrade.sh.old

Synology DSM v6

I don’t recommend upgrading to v6 at this time (March 2016) if you do not have an Intel based CPU in your Synology NAS as there have been issues reported with the java packages in the comments here as well as on Patter’s page. If you do have an Intel based CPU you will need to use 7z commands on DSM6 instead of zip commands. Examples for CrashPlan 4.6.0:
sudo 7z e -y -o/var/packages/CrashPlan/target/lib /var/packages/CrashPlan/target/upgrade/1435813200460_382.jar "*.jar" sudo 7z e -y -o/var/packages/CrashPlan/target/lang /var/packages/CrashPlan/target/upgrade/1435813200460_382.jar "lang/*"

Fixing CrashPlan 4.2.0 on Synology after DSM 5.2 update

 How To  158 Responses »
May 122015
 

The latest round of updates from CrashPlan (4.3.0) has broken the CrashPlan package on Synology again. Please see http://chrisnelson.ca/2015/07/02/fixing-crashplan-4-3-0-on-synology/ for the newest instructions.

Archived information for 4.2:

After installing Synology DSM 5.2, CrashPlan recently pushed an update (4.2.0) which broke the CrashPlan package Patters had created for Synology. The log for the failing CrashPlan package would look something like this:

CrashPlan started, version 3.7.0, GUID 559750658046558476 Upgrades available at central.crashplan.com:443 Downloading a new version of CrashPlan. Download of upgrade complete – version 1425276000420. Installing upgrade – version 1425276000420 Upgrade installed – version 1425276000420 CrashPlan stopped, version 3.7.0, GUID 559750658046558476

If the package was attempted to start again, it would write a line similar to the following, and then stop:

Synology repairing upgrade in /var/packages/CrashPlan/target/upgrade/1425276000420.

This seemed to be similar to the previous issue seen in 3.7.0, so I used the following steps to fix crashplan on my synology box: Note that this will likely be fixed by Patters soon, this is just a workaround if your CrashPlan install has stopped as a result of the 4.2.0 update.

Connect as root to SSH

First, connect to Synology using SSH and the root account (uses admin password).  Connecting as the admin account will not give you enough permissions.

In SSH, run commands to extract the update

In SSH, run unique command to cleanup the package

The next command will be different for everyone, as part of the file path is randomized.  In this case, you will need to see what your path is first by running ls, and then editing the second statement below, replacing whatevervalue with the rest of  your path:

Start CrashPlan package, check log

After the above completes, you should be able to start the CrashPlan package again in Synology and get it running once more. The log should show:

CrashPlan started, version 4.2.0 GUID 559750658046558476

Fixing CrashPlan on Synology after the 3.7.0 update (Synology repairing upgrade in /var/packages/CrashPlan/target/upgrade/1388728800370. )

 How To  191 Responses »
Jan 102015
 

If you recently upgraded to DSM 5.2 and had CrashPlan break, this is likely due to the CrashPlan 4.2.0 update. Please see http://chrisnelson.ca/2015/05/12/fixing-crashplan-4-2-0-on-synology-after-dsm-5-2-update/

NOTE: This manual fix should no longer be needed after the 0029 update from Patters:

0029 Updated to CrashPlan version 3.7.0, improved detection of temp folder (prevent use of /var/@tmp), added support for Annapurna Alpine AL514 CPU (armhf) in DS2015xs, added support for Marvell Armada 375 CPU (armhf) in DS215j, abandoned practical efforts to try to support Code42’s upgrade scripts, abandoned inotify support (realtime backup) on PowerPC after many failed attempts with self-built and pre-built jtux and jna libraries, back-merged older libffi support for old PowerPC binaries after it was removed in 0028 re-write

Archive content:
CrashPlan recently pushed an update (3.7.0) to all clients which broke the CrashPlan package Patters had created for Synology. The log for the failing CrashPlan package would look something like this:

CrashPlan started, version 3.6.4, GUID 574956158631543214 Upgrades available at central.crashplan.com:443 Downloading a new version of CrashPlan. Download of upgrade complete – version 1388728800370. Installing upgrade – version 1388728800370 Upgrade installed – version 1388728800370 CrashPlan stopped, version 3.6.4, GUID 574956158631543214

If the package was attempted to start again, it would write a line similar to the following, and then stop:

Synology repairing upgrade in /var/packages/CrashPlan/target/upgrade/1388728800370.

After some investigation by Stig, Casper, and Harv on the comments for the package a working solution was found to manually install the update files and disable the failing upgrade.sh file. Note that this will likely be fixed by Patters soon, this is just a workaround if your CrashPlan install has stopped as a result of the 3.7.0 update.

1. First, connect to Synology using SSH and the root account (uses admin password).  Connecting as the admin account will not give you enough permissions.
2. Run the following commands:

 
3. The next command will be different for everyone, as part of the file path is randomized.  In this case, you will need to see what your path is first by running ls, and then editing the second statement below, replacing whatevervalue with the rest of  your path:

4. After the above completes, you should be able to start the CrashPlan package again in Synology and get it running once more.

Backing up data on AWS EC2 Micro instances using the CrashPlan headless Linux client

 Me & my ramblings  3 Responses »
May 272013
 

I’ve been using a combination of scripts to do local backups on my Amazon EC2 micro instance I use to serve this website – AutoMySQL Backup and some cron jobs which ran rsync for various paths on a rotation. For example:

#!/bin/sh
rsync -a /var/www/html /mnt/backup/filebackup/weekly
rsync -a /var/lib/g2data /mnt/backup/filebackup/weekly

This is frankly a pretty lazy way to do it. I’m not protected at all if something wipes out the backup destinations or the EBS drive goes bad, and this method uses up a lot of EBS space because there are multiple sets of the files. I could use the AWS EC2 framework to script out EBS snapshots, but that’s just going to further increase my monthly Amazon bill without any ability to be very specific about point in time restores for files. Instead, I thought I should make use of something that I’m already paying for: CrashPlan.

As noted in a previous post, I’m using CrashPlan to backup our desktop & laptop computers, as well as my file server (Synology NAS). I have a CrashPlan+ Family Unlimited plan which means I can add up to 10 computers and store unlimited backups from them to the CrashPlan cloud included in the plan (local backups, or peer to peer Crashplan backups are always free and don’t require a plan).

Install CrashPlan to a Amazon Linux AMI

CrashPlan offers a number different clients, including a headless java client for Linux. This is perfectly suited to the micro instance I’m using in EC2 – the Amazon Linux AMI which is based on RedHat/Centos. I installed the headless client using the following options – note that I’m using the latest version at the time of this post (3.5.3) in my commands below, but you can see the latest download link on their page. I’m also using sudo in my commands, you can remove pr ignore that if it is isn’t needed in your Linux configuration.

sudo yum install grep sed cpio gzip coreutils
wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.5.3_Linux.tgz
sudo tar -xzf CrashPlan_3.5.3_Linux.tgz
cd CrashPlan-install/
sudo install.sh

At this point the installer launches and will ask questions about where the files should go. Their suggestions are reasonable for my configuration and I was able to simply follow the defaults, hitting enter the whole way through the install. Once the install finishes, it will start the CrashPlan service automatically.

 

Connect to the headless CrashPlan Linux server with a remote client

Now that the service has been started, a remote client needs to connect to the server in order to further configure backup options. The easiest and most secure way to do that is by making use of SSH’s ability to tunnel to the server. The following instructions are for Windows, but similar steps can be performed on other operating systems. First, install the CrashPlan client if it is not already installed on your computer, but don’t start the program. Next, locate and edit the ui.properties file using a text editor. This file is typically located here: C:\Program Files\CrashPlan\conf\ui.properties for Windows systems. As shown below, remove the # to uncomment the line, and change the port to 4200. When done, save the file and exit.

Edit servicePort for ui.properties

Next the SSH tunnel needs to be enabled for the client to connect to the server via SSH. Open PuTTY and create a new connection to your Linux server. Under the configuration menu, navigate to Connection, SSH, then click the Tunnels option in the menu. On that page, enter “4200” as Source port, enter “localhost:4243” as the Destination, and click the Add button. Once completed, connect to the server as normal via the configured SSH session and leave the terminal window open.

putty tunnel add

putty tunnel added

At this point the CrashPlan client can be started. It will first ask for CrashPlan credentials, then display the usual interface. Note that the compression and dedupe options can be resource heavy – which means during the first backups for the server it will likely consume a lot of CPU, particularly for EC2 micro instances which have low CPU throughput (bursting) to start with. This CPU usage should reduce over time as the backup deltas get smaller.

crashplan ui linux remote

Configure CrashPlan Linux to backup /var or other hidden directories (if needed)

Note that by default, several directories and file structures are hidden in the CrashPlan client for Linux. In my case I want to backup files under /var, as that is where my gallery2 files reside, as well as my web content. In order to expose that folder structure for CrashPlan the my.service.xml configuration file should be edited, and the “pattern regex=”/var/” line under the Linux area should be removed. First stop the CrashPlan service and edit the config file (assuming you installed using default file paths):

sudo service crashplan stop
sudo vi /usr/local/crashplan/conf/my.service.xml

Next, look for a line like this under the Linux area and remove the following data from the file (e.g. dd in vi):

<pattern regex="/var/"></pattern>

Save the file (Esc, :wq + enter in vi) and then start the service back up. After connecting again using the client, the /var folder should now be visible.

sudo service crashplan start

crashplan ui linux remote file selection

My backup solution – CrashPlan & Synology

 Me & my ramblings  2 Responses »
May 192013
 

I’ve got a lot of data. I’ve been been shooting RAW photos for a decade (almost 200 GB at this point), have a large (legit!) music collection (74 GB), and have a lot of other files from various projects over the years I want to hang onto. In total, I’ve got about 330 GB I want to keep. This used to be a very expensive proposition – stuffing it all in Amazon S3 or using backup services that charge by file size was tough to swallow. That landscape has changed recently. I’ve been using CrashPlan+ Family Unlimited for my home backups for about a year and couldn’t be happier. Unlimited cloud backups for up to 10 computers for $9-14 per month (depending on subscription length) is an amazing deal.

More than being a good deal I’ve also been very impressed with the CrashPlan software as well. It does the typical things you want to see in backup software – good performance, ability to set transfer rates/time of day, data deduplication, compression, and encryption. However it’s hidden strength is its great flexibility for backup targets while maintaining security. In addition to the unlimited cloud storage you can also have local encrypted backups, encrypted backups to another one of your computers running CrashPlan (p2p using the same account), or even send your encrypted backups to a friend (p2p with different CrashPlan accounts). These options create a perfect backup scenario for me – I know I have a local copy of files which I can get at quickly (compared with downloading them all) but they are also stored in the cloud to protect against catastrophic loss (e.g. a house fire).

CrashPlan destination options

Their software is available for multiple platforms, and they support a headless java client on Linux. This means the software can be installed on a lot of different machine types and opens up a lot of different options. The most important one for me is support in Synology. I’ve been using Synology NASs as my file server for many years as they are very customizable and powerful. With some hard work invested, patters was able to get the headless client running on a wide range of newer Synology devices. Using his packages and instructions I’ve got all of the files on my Synology file server backing up locally, as well as to the cloud:

Desktop/laptops

  • Backup to local Synology NAS Crashplan target (Vol2)
  • Backup to Crashplan cloud

Synology NAS (Vol1)

  • Backup Vol1 to local Synology NAS Crashplan target (Vol2)
  • Backup to Crashplan cloud

Some screen shots of what this looks like:

synology package center

crashplan ui synology remote

If you find the post at pcloadletter.co.uk hard to follow, Scott Hanselman has a great guide on his site on how to setup CrashPlan on Synology.

Other than having to restart the Synology CrashPlan package after updates, everything has worked amazingly well together. I was able to customize everything the way I needed to but still feel like I’m well protected. If you aren’t using a backup solution you like, I highly recommend giving CrashPlan a try – the following link will save you 20% off their prices: http://www.crashplan.com/ff20