Apr 222014
 

What is Monit?

The simplest way of describing the value Monit is that it is a powerful monitoring tool, which can take action in the event of failures or alerts. Monit is able to keep an eye on processes, files (size, permissions, checksum), the system (CPU, RAM, Load), as well as protocols (HTTP, FTP, SMTP, SSH, etc). If any of the monitored items change in specific ways, Monit can spring into action with an alert, run separate programs, or attempt to correct the problem with a prescribed action (e.g. restart a service).

Monit web interface

How do I install Monit?

Installing Monit is fairly easy these days, as the majority of Linux repositories include the package. If you were running Ubuntu, you could issue the following command to install the package:

If you were running Redhat/CentOS, you could issue the following command to install the package:

How do I configure Monit?

After Monit has been installed, all configuration changes can be made via the monitrc file via text editor, in this case I’m using vi. Note that all of the paths for pid files and other items in my examples are for Ubuntu – you may need to do some edits if you are on a different version of Linux.

Once the file has been loaded in the text editor, you will see that there are already a number of options set (e.g. Monit is set to check every 2 minutes by default) in addition to a number of items which are commented out. For this example I’m not changing any of the default values, and you can move to the bottom of the file and begin adding new entries to be monitored by Monit.

Add email notification

First I will configure Monit to use a the local system as the default email server, set the format of the alert/status emails to be delivered, and the email address I want email alerts to go to. If you are able to, it is best to setup an alias email address to use for monitoring (e.g. monitoring@yourdomain.com), that way you can easily add additional rules or filters for your email based on the email address.

Add the Monit web interface

Monit has a nice web interface included, but is disabled by default. One important thing to note – the web interface allows the user to start, stop, and restart monitored services, as well as make monitoring changes. As such, I highly recommend using an extremely long and complex password in addition to a non standard user name for the account specified in this step to enable the web interface (format: allow accountname:password). It may also be in your best interest to limit the port used (2812 in this example) via firewall to your own IP address.

Monitor disk usage

Monit has the ability to monitor disk usage for a system, in addition to specific files. In this case, the monitor has been configured to watch the free space available on the /dev/vda drive, and send an alert if it is 95% full. The group tag seen in this example is used to organize different monitors, it doesn’t impact the usage in any way.

Monitor system load

Monit can also monitor the host system’s load – in this case sending an alert if the load average over the course of 5 minutes is larger than 3.

Monitor MySQL

In this example, Monit is monitoring MySQL in two different ways. The first way is via the specified pid file – Monit is reading that file, determining the process id, and then keeping an eye on that process on the system. The second way is via service testing – Monit is checking that MySQL has port 3306 open with the mysql specific protocol (it supports a number of options). If the process fails either of those monitoring checks, then the stop & start program scripts are run, restarting MySQL.

Monitor SSH

In this example Monit is checking the SSH daemon via pid and SSH protocol service response.

Monitor Apache2

In this example, the Apache web server is being checked four different ways. The first two have already been shown in previous examples, monitoring via pid and service protocol (in this case http). However, just because the Apache webserver is running doesn’t mean it is properly serving files. In this example, Monit is asking for a specific URL, “/monit/token” from the web server to verify it is able to serve static content. Monit is also keeping an eye of the cpu usage of this process – it is alerting when CPU usage is high, but it could also restart the service on extended CPU usage if desired.

One thing to note about the above monitor – it will fail unless the Apache web server serves back a monitor token file – we need to create that file for the web server to use first:

Monitor crashplan backup service

I’m a fan of CrashPlan and I use it on all of my machines, including my servers. However, I’ve found that the CrashPlan backup engine can be unstable on some Linux distros when on a server with limited memory, so I’ve always added it as a monitored service to keep an eye on it and restart it if it shuts down:

Implement the Monit changes

After making changes to the Monit configuration file, save it and have Monit reload to see the changes. If you are not sure if the Monit service is not already running (e.g. it is a new install), then the following command can be used:

If Monit is already running, then you can just tell it to reload the configuration file:

Hopefully this introduction is enough to get your creative monitoring juices flowing. There are lots of other examples for monitoring different things using Monit, and the platform is flexible enough that it can almost do anything you need.

May 272013
 

I’ve been using a combination of scripts to do local backups on my Amazon EC2 micro instance I use to serve this website – AutoMySQL Backup and some cron jobs which ran rsync for various paths on a rotation. For example:

#!/bin/sh
rsync -a /var/www/html /mnt/backup/filebackup/weekly
rsync -a /var/lib/g2data /mnt/backup/filebackup/weekly

This is frankly a pretty lazy way to do it. I’m not protected at all if something wipes out the backup destinations or the EBS drive goes bad, and this method uses up a lot of EBS space because there are multiple sets of the files. I could use the AWS EC2 framework to script out EBS snapshots, but that’s just going to further increase my monthly Amazon bill without any ability to be very specific about point in time restores for files. Instead, I thought I should make use of something that I’m already paying for: CrashPlan.

As noted in a previous post, I’m using CrashPlan to backup our desktop & laptop computers, as well as my file server (Synology NAS). I have a CrashPlan+ Family Unlimited plan which means I can add up to 10 computers and store unlimited backups from them to the CrashPlan cloud included in the plan (local backups, or peer to peer Crashplan backups are always free and don’t require a plan).

Install CrashPlan to a Amazon Linux AMI

CrashPlan offers a number different clients, including a headless java client for Linux. This is perfectly suited to the micro instance I’m using in EC2 – the Amazon Linux AMI which is based on RedHat/Centos. I installed the headless client using the following options – note that I’m using the latest version at the time of this post (3.5.3) in my commands below, but you can see the latest download link on their page. I’m also using sudo in my commands, you can remove pr ignore that if it is isn’t needed in your Linux configuration.

sudo yum install grep sed cpio gzip coreutils
wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.5.3_Linux.tgz
sudo tar -xzf CrashPlan_3.5.3_Linux.tgz
cd CrashPlan-install/
sudo install.sh

At this point the installer launches and will ask questions about where the files should go. Their suggestions are reasonable for my configuration and I was able to simply follow the defaults, hitting enter the whole way through the install. Once the install finishes, it will start the CrashPlan service automatically.

 

Connect to the headless CrashPlan Linux server with a remote client

Now that the service has been started, a remote client needs to connect to the server in order to further configure backup options. The easiest and most secure way to do that is by making use of SSH’s ability to tunnel to the server. The following instructions are for Windows, but similar steps can be performed on other operating systems. First, install the CrashPlan client if it is not already installed on your computer, but don’t start the program. Next, locate and edit the ui.properties file using a text editor. This file is typically located here: C:\Program Files\CrashPlan\conf\ui.properties for Windows systems. As shown below, remove the # to uncomment the line, and change the port to 4200. When done, save the file and exit.

Edit servicePort for ui.properties

Next the SSH tunnel needs to be enabled for the client to connect to the server via SSH. Open PuTTY and create a new connection to your Linux server. Under the configuration menu, navigate to Connection, SSH, then click the Tunnels option in the menu. On that page, enter “4200” as Source port, enter “localhost:4243” as the Destination, and click the Add button. Once completed, connect to the server as normal via the configured SSH session and leave the terminal window open.

putty tunnel add

putty tunnel added

At this point the CrashPlan client can be started. It will first ask for CrashPlan credentials, then display the usual interface. Note that the compression and dedupe options can be resource heavy – which means during the first backups for the server it will likely consume a lot of CPU, particularly for EC2 micro instances which have low CPU throughput (bursting) to start with. This CPU usage should reduce over time as the backup deltas get smaller.

crashplan ui linux remote

Configure CrashPlan Linux to backup /var or other hidden directories (if needed)

Note that by default, several directories and file structures are hidden in the CrashPlan client for Linux. In my case I want to backup files under /var, as that is where my gallery2 files reside, as well as my web content. In order to expose that folder structure for CrashPlan the my.service.xml configuration file should be edited, and the “pattern regex=”/var/” line under the Linux area should be removed. First stop the CrashPlan service and edit the config file (assuming you installed using default file paths):

sudo service crashplan stop
sudo vi /usr/local/crashplan/conf/my.service.xml

Next, look for a line like this under the Linux area and remove the following data from the file (e.g. dd in vi):

<pattern regex="/var/"></pattern>

Save the file (Esc, :wq + enter in vi) and then start the service back up. After connecting again using the client, the /var folder should now be visible.

sudo service crashplan start

crashplan ui linux remote file selection