Sep 5, 2018 - Python decorators

Python decorators

I’m not a big fan of Python (yet), but it certainly has some nice features, like ‘decorators’.

A decorator wraps a method and lets you do specific things before and after the method is called. This can be very useful, to check the state of the application before the method is called (for example if the user is logged in, has the permission to execute the method, etc.).

I recently had a class with various methods which had to connect to a remote service, do some stuff with the service, and then disconnect again to clear up the resources. I could have added the same connect and disconnect code to each of the methods, but using a decorator is of course much easier and maintainable. Here’s how it looked like:

I created a decorator ‘service_required’ which connects to the service and makes the service accessible to the class in which it is needed (see the ‘service’ variable). It is little bit like ‘dependency injection’ which Python doesn’t offer per se. Then it calls the decorated function. Thereafter it disconnects from the service again.

Note 1: The wrapper function takes the arguments ‘self’, ‘*args’, and ‘**kwargs’, that is ‘self’ a reference to the ‘calling’ class (that’s needed to set the ‘service’ variable of the ‘calling’ class), ‘*args’ and ‘**kwargs’ which are the arguments passed to the decorated methods.

Note 2: The decorator calls the decorated function, stores the returned value, does its cleanup tasks (disconnecting from the server) and then returns this value. This is important, otherwise the return value of the decorarated method is simply lost.

from functools import wraps


class Service():
    """
    This is an example service, which could be some remote
    service endpoint, a database, etc. In order to be used
    we must first connect to it and then after our request
    is finished we have to disconnect again, so that any
    resources are released again.
    """

    connected = False

    def connect(self):
        self.connected = True
        print "Connection established."

    def add_one(self, value):
        if not self.connected:
            raise Exception("This service is not online")
        return value + 1

    def disconnect(self):
        self.connected = False
        print "Connection closed."


def service_required(func):
    """
    Decorator which makes sure that the service is
    connected and gets properly disconnected again
    after a method call.
    """

    @wraps(func)
    def _wrapper(self, *args, **kwargs):
        self.service = Service()
        self.service.connect()

        value = func(self, *args, **kwargs)

        self.service.disconnect()
        self.service = None
        return value

    return _wrapper


class Example():
    """
    An example class which uses the service
    to do something.
    """

    service = None

    @service_required
    def do_something(self):
        value = self.service.add_one(1)
        print value

    @service_required
    def and_again(self):
        value = self.service.add_one(2)
        print value


example = Example()
example.do_something()
example.and_again()

Edit (2018/09/06)

I just noticed, an even better implementation is to wrap the function call in a try finally statement, this way you can be really sure that the close methods are called, even if the function call throws an exception. The only way the close method could be missed is if the function call terminates the application (either deliberatly or due to an error).

The decorator would then look like this:

def service_required(func):
    """
    Decorator which makes sure that the service is
    connected and gets properly disconnected again
    after a method call.
    """

    @wraps(func)
    def _wrapper(self, *args, **kwargs):
        self.service = Service()
        self.service.connect()

        try:
            return func(self, *args, **kwargs)
        finally:
            self.service.disconnect()
            self.service = None

    return _wrapper

Aug 24, 2018 - Default server setup tasks

Default server setup tasks

Following up on the previous blog post Protect your server here are some more typical tasks I set up on a fresh system.

Firewall

In the earlier post I showed the output of the iptables-save command of my typical iptables firewall configuration (this can be easily loaded with iptables-restore). However, it might be easier to understand looking at the iptables commands itself:

#!/bin/bash

iptables -F
iptables -A INPUT -i lo -p all -j ACCEPT
iptables -A OUTPUT -o lo -p all -j ACCEPT
iptables -A OUTPUT -o eth0 -p all -j ACCEPT
iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 300 \
	--hitcount 2 -j LOG --log-prefix "Possible SSH attack! " --log-level 7 
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 300 \
	--hitcount 2 -j DROP
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -P INPUT DROP

This script sets up the firewall in the following way:

  • Accept all input and output on the local loopback interface lo
  • Allow all output on the network card eth0
  • Allow all input from already established connections
  • Track the connection attempts on port 22 (SSH) (needs recent module)
  • Block connections on port 22 for 5 minutes for IP addresses which try more than twice to establish a connection within 5 minutes
  • Allow all other connections on port 22 (the order is important, has to come after the ‘block’ rule!)
  • If none of the rules matched, drop connection

Talk to me

…using ‘mini smtp’ and ‘mutt’.

If you want to be up-to-date about what’s going on your server, you should let your server send you an email occassionaly. For example to tell you the outcome of the daily backup, etc.

This requires to setup the local mail system. In my opinion the easiest way is to use msmtp together with a gmail account and mutt.

apt-get install msmtp mutt

Add your gmail account settings to the /etc/msmtprc config file:

# Set default values for all following accounts.
defaults
auth	on
tls		on
tls_trust_file	/etc/ssl/certs/ca-certificates.crt
logfile		~/.msmtp.log

# A gmail account
account		gmail
host		smtp.gmail.com
port		587
from		xxx@googlemail.com
user		xxx@googlemail.com
password		xxx

# Set a default account
account default : gmail

Send a test eMail from the command line:

echo "This is just a test..." | mutt -s "Just a test" "xxx@example.com"

Get notified when someone logs in

As you can now send eMails from the command line, you can make use of this for example for getting notified when someone logs in via SSH.

Create a script, e. g. /usr/local/sbin/login_notify.sh:

#!/bin/sh

EMAIL_TO="xxx@example.com"

SUBJECT="SSH Login Notification"

TIME=`date +"%d-%m-%Y %T"`

MESSAGE="
A user signed into your server through SSH.
-------------------------------------------
Username: ${PAM_USER}
IP Address: ${PAM_RHOST}
Time: ${TIME}"

if [ ${PAM_TYPE} = "open_session" ]; then
	echo "${MESSAGE}" | mutt -s "${SUBJECT}" "${EMAIL_TO}"
fi

exit 0

Then add a hook into the pam service /etc/pam.d/sshd which calls your notification script on each SSH login:

...
session required pam_exec.so /usr/local/sbin/login_notify.sh

Backups with rsnapshot

rsnaphot uses rsync and hard links in a very clever way to create easy accessible backups without wasting much disk space.

On my home NAS I have two hard disks, one holding the data which is shared via NFS, the other is the backup drive. I perform 7 daily backups followed by a weekly backup. That way I always have an incremental backup of the last 7 days, plus snapshots of every week. For example at the moment my backup drive looks like that:

drwxr-xr-x 3 root    root    4.0K Aug 24 04:20 daily.0
drwxr-xr-x 3 root    root    4.0K Aug 23 04:10 daily.1
drwxr-xr-x 3 root    root    4.0K Aug 22 03:50 daily.2
drwxr-xr-x 3 root    root    4.0K Aug 21 03:57 daily.3
drwxr-xr-x 3 root    root    4.0K Aug 20 03:51 daily.4
drwxr-xr-x 3 root    root    4.0K Aug 19 03:53 daily.5
drwxr-xr-x 3 root    root    4.0K Aug 18 04:09 daily.6
drwxr-xr-x 3 root    root    4.0K Aug 14 03:36 weekly.0
drwxr-xr-x 3 root    root    4.0K Aug  7 03:36 weekly.1
...

Where each directory represents a “snapshot” of the particular time.

After installing rsnapshot edit/add these values of the config file /etc/rsnapshot.conf

# All snapshots will be stored under this root directory.
# (E.g. that's the mount point of my backup hard disk)
snapshot_root  /backup

# Comment out the backup levels and create
retain	daily	7
retain	weekly	52

# Specify the log file
logfile	/var/log/rsnapshot.log

# The directories to backup and excludes
# (E.g. that's the mount point of my NFS share hard disk and I 
# exclude all 'tmp' directories)
backup	/nfs/	all/
exclude	tmp/

I use a bash scripts which launch rsnapshot and send my an email with the result, so that I notice soon when something’s going wrong. There’s only one thing which is worse than having no backups, it’s wrongly assuming you have working backups ;-)

#!/bin/bash
mount /backup

/usr/bin/rsnapshot daily

umount /backup

tail -14 /var/log/rsnapshot.log | mutt -s "rsnapshot backup daily" xxx@example.com

(the ‘weekly’ one is basically the same; just make sure you don’t run them at the same time)

Aug 3, 2018 - Useful R snippets (1)

Useful R snippets (1)

Coming from ‘traditional’ programming languages I struggled quite a bit getting my head around how to work with R dataframes. If you are in the same situation, have a look at the snippets, hope that’s helpful.

For the snippets I just use the ‘iris’ example dataset

library(datasets)
data("iris")
data.frame(iris)

df <- iris

First of all, forget about arrays for a while. It is tempting, but try to not see an R dataframe as array.

Access specific elements, rows, columns

You can address the column of a dataframe by its name (a bit like an associative array):

df$Sepal.Length

This returns a list, in which you can address the elements with the double bracket notation, e.g. get the first element:

df$Sepal.Length[[1]]

You can also set this element:

df$Sepal.Length[[1]] <- 123

Get all values of a column, e. g. the first column (returns a list!):

df[, 1]

Get all values of a row, e. g. the first row (returns a dataframe!):

df[1, ]

Create subsets of a dataframe

Specific rows

# like before, only include first row
subDf <- df[1, ]

# take a specific range:
subDf <- df[1:10, ]

# only rows which match a specific criteria
subDf <- df[ which(df$Species == 'setosa' | df$Species == 'virginica'), ]

Specific columns

Create a subset with only certain columns:

subDf <- subset(df, select=c('Sepal.Length', 'Sepal.Width'))

Merge rows

By renaming, which requires temporarily conversion from ‘factors’ to ‘characters’.

df$Species <- as.character(df$Species)
df[ df == 'versicolor' ] <- 'versicolorAndVirginica'
df[ df == 'virginica' ] <- 'versicolorAndVirginica'
df$Species <- as.factor(df$Species)

Add a new column

For example for the result of a calculation of another column

df$Sepal.HalfWidth <- df$Sepal.Width / 2

Iterate over dataframe using sapply

Apply a function to each data element and add the result back to dataframe into a new column

square <- function(n) {
  n * n;
}
df$Sepal.Width.Squared <- sapply(df$Sepal.Width, square)
# Could be shorted to:
df$Sepal.Width.Squared.2 <- sapply(df$Sepal.Width, function(n) n * n)

Add rows

df <- rbind(df, c(10, 5, 2.5, 1.5, 'virginica'))

Add columns

df$NewColumn <- (1:150)

Order data by some kind of summary statistic

Can be handy for plotting. E. g. order by mean, low to high, then plot:

ag <-aggregate(x = df[c('Sepal.Length')], by=df[c('Species')], mean)
orderedSpecies <- factor(df$Species, levels=ag[order(ag$Sepal.Length), 'Species'])
plot(df$Sepal.Length ~ orderedSpecies, ylab='Sepal Length', xlab="Species")