Sep 26, 2018 - Neat Java 8 features - Lambda expressions

Comments

Neat Java 8 features - Lambda expressions

At work I’m still bound to Java 7, which is a bit a pity because since Java 8 there are some pretty neat features available.

This first of three posts is about Lambda expressions.

Next posts:

Lambda expressions

Lambda expressions allow you to pass functions on to methods, objects, etc.

E.g. you can create a function with the -> operator:

// A function which takes an Integer and returns a String:
Function<Integer, String> func = i -> {
    return  "This is '"+i+"'' as String";
};

You can then pass it on to another object, e. g.

someObject.setSomeFunction(func);

In the object you can make use of this function by calling its apply method

func.apply(100);

Here’s a complete example:

import java.util.function.Function;

public class Test {

    private Function<Integer, String> func;
    
    public void setFunction(Function<Integer, String> func) {
        this.func = func;
    }

    public String doSomething(int i) {
        return func.apply(i);
    }
    
    public static void main(String[] args) {
        Test test = new Test();
        
        Function<Integer, String> func = i -> {
            return "This is '"+i+"' as String";
        };
        
        test.setFunction(func);
        
        String result = test.doSomething(100);
        
        System.out.println(result);
    }
}

Instead of defining the function and then passing it on you more commonly see something like this, which is equivalent but shorter:

test.setFunction(i -> "This is '"+i+"' as String");

Function is a Functional interface, which means an interface which only has one abstract method. With the -> operator you effectively provide the implementation for this abstract method.

You can define your own Functional interfaces. Let’s say you have a Handler interface which provides a method to process a certain request, in this example case consisting of a String and an integer, but that could as well be an HTTP request, UI event, etc.

@FunctionalInterface
public interface Handler {
    public String handle(String item, int amount);
}

Annotate it with the FunctionalInterface annotation. It’s not necessary but helps spotting issues early!

You can use the Handler interface in other classes for example like this (directly providing an implementation via the -> operator):

public class Test {

    private Handler handler;
    
    public void setHandler(Handler handler) {
        this.handler = handler;
    }
    
    public String doSomething(String arg1, int arg2) {
        return handler.handle(arg1, arg2);
    }
    
    public static void main(String[] args) {
        Test test = new Test();

        test.setHandler((x, y) -> {
            return "Successfully handled " + x + " and " + y;
        });

        String result = test.doSomething("Test", 1);
        System.out.println(result);
    }
}

Where possible many core Java interfaces have been annotated with the FunctionalInterface Annotation, for example the Actionlistener, so that you can easily provide an implementation:

JButton button = new JButton("Click me");
button.setActionCommand("doSomething");
button.addActionListener(e -> {
    System.out.println("Handling event "+e.getActionCommand());
});

Coming back the Handler example: Instead of the ‘inline’ implementation …

test.setHandler((x, y) -> {
    return "Successfully handled " + x + " and " + y;
});

… you can of course use already implemented methods:

HandlerImpl h = new HandlerImpl();
test.setHandler(h::handle);

This :: syntax is a method reference (also a new feature of Java 8)

If this method is a static method, this could be even shorter as you don’t need an instance:

test.setHandler(StaticHandlerImpl::handle);

Next post: Neat Java 8 features - Stream API

Sep 5, 2018 - Python decorators

Comments

Python decorators

I’m not a big fan of Python (yet), but it certainly has some nice features, like ‘decorators’.

A decorator wraps a method and lets you do specific things before and after the method is called. This can be very useful, to check the state of the application before the method is called (for example if the user is logged in, has the permission to execute the method, etc.).

I recently had a class with various methods which had to connect to a remote service, do some stuff with the service, and then disconnect again to clear up the resources. I could have added the same connect and disconnect code to each of the methods, but using a decorator is of course much easier and maintainable. Here’s how it looked like:

I created a decorator ‘service_required’ which connects to the service and makes the service accessible to the class in which it is needed (see the ‘service’ variable). It is little bit like ‘dependency injection’ which Python doesn’t offer per se. Then it calls the decorated function. Thereafter it disconnects from the service again.

Note 1: The wrapper function takes the arguments ‘self’, ‘*args’, and ‘**kwargs’, that is ‘self’ a reference to the ‘calling’ class (that’s needed to set the ‘service’ variable of the ‘calling’ class), ‘*args’ and ‘**kwargs’ which are the arguments passed to the decorated methods.

Note 2: The decorator calls the decorated function, stores the returned value, does its cleanup tasks (disconnecting from the server) and then returns this value. This is important, otherwise the return value of the decorarated method is simply lost.

from functools import wraps


class Service():
    """
    This is an example service, which could be some remote
    service endpoint, a database, etc. In order to be used
    we must first connect to it and then after our request
    is finished we have to disconnect again, so that any
    resources are released again.
    """

    connected = False

    def connect(self):
        self.connected = True
        print "Connection established."

    def add_one(self, value):
        if not self.connected:
            raise Exception("This service is not online")
        return value + 1

    def disconnect(self):
        self.connected = False
        print "Connection closed."


def service_required(func):
    """
    Decorator which makes sure that the service is
    connected and gets properly disconnected again
    after a method call.
    """

    @wraps(func)
    def _wrapper(self, *args, **kwargs):
        self.service = Service()
        self.service.connect()

        value = func(self, *args, **kwargs)

        self.service.disconnect()
        self.service = None
        return value

    return _wrapper


class Example():
    """
    An example class which uses the service
    to do something.
    """

    service = None

    @service_required
    def do_something(self):
        value = self.service.add_one(1)
        print value

    @service_required
    def and_again(self):
        value = self.service.add_one(2)
        print value


example = Example()
example.do_something()
example.and_again()

Edit (2018/09/06)

I just noticed, an even better implementation is to wrap the function call in a try finally statement, this way you can be really sure that the close methods are called, even if the function call throws an exception. The only way the close method could be missed is if the function call terminates the application (either deliberatly or due to an error).

The decorator would then look like this:

def service_required(func):
    """
    Decorator which makes sure that the service is
    connected and gets properly disconnected again
    after a method call.
    """

    @wraps(func)
    def _wrapper(self, *args, **kwargs):
        self.service = Service()
        self.service.connect()

        try:
            return func(self, *args, **kwargs)
        finally:
            self.service.disconnect()
            self.service = None

    return _wrapper

Aug 24, 2018 - Default server setup tasks

Comments

Default server setup tasks

Following up on the previous blog post Protect your server here are some more typical tasks I set up on a fresh system.

Firewall

In the earlier post I showed the output of the iptables-save command of my typical iptables firewall configuration (this can be easily loaded with iptables-restore). However, it might be easier to understand looking at the iptables commands itself:

#!/bin/bash

iptables -F
iptables -A INPUT -i lo -p all -j ACCEPT
iptables -A OUTPUT -o lo -p all -j ACCEPT
iptables -A OUTPUT -o eth0 -p all -j ACCEPT
iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 300 \
	--hitcount 2 -j LOG --log-prefix "Possible SSH attack! " --log-level 7 
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 300 \
	--hitcount 2 -j DROP
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -P INPUT DROP

This script sets up the firewall in the following way:

  • Accept all input and output on the local loopback interface lo
  • Allow all output on the network card eth0
  • Allow all input from already established connections
  • Track the connection attempts on port 22 (SSH) (needs recent module)
  • Block connections on port 22 for 5 minutes for IP addresses which try more than twice to establish a connection within 5 minutes
  • Allow all other connections on port 22 (the order is important, has to come after the ‘block’ rule!)
  • If none of the rules matched, drop connection

Talk to me

…using ‘mini smtp’ and ‘mutt’.

If you want to be up-to-date about what’s going on your server, you should let your server send you an email occassionaly. For example to tell you the outcome of the daily backup, etc.

This requires to setup the local mail system. In my opinion the easiest way is to use msmtp together with a gmail account and mutt.

apt-get install msmtp mutt

Add your gmail account settings to the /etc/msmtprc config file:

# Set default values for all following accounts.
defaults
auth	on
tls		on
tls_trust_file	/etc/ssl/certs/ca-certificates.crt
logfile		~/.msmtp.log

# A gmail account
account		gmail
host		smtp.gmail.com
port		587
from		xxx@googlemail.com
user		xxx@googlemail.com
password		xxx

# Set a default account
account default : gmail

Send a test eMail from the command line:

echo "This is just a test..." | mutt -s "Just a test" "xxx@example.com"

Get notified when someone logs in

As you can now send eMails from the command line, you can make use of this for example for getting notified when someone logs in via SSH.

Create a script, e. g. /usr/local/sbin/login_notify.sh:

#!/bin/sh

EMAIL_TO="xxx@example.com"

SUBJECT="SSH Login Notification"

TIME=`date +"%d-%m-%Y %T"`

MESSAGE="
A user signed into your server through SSH.
-------------------------------------------
Username: ${PAM_USER}
IP Address: ${PAM_RHOST}
Time: ${TIME}"

if [ ${PAM_TYPE} = "open_session" ]; then
	echo "${MESSAGE}" | mutt -s "${SUBJECT}" "${EMAIL_TO}"
fi

exit 0

Then add a hook into the pam service /etc/pam.d/sshd which calls your notification script on each SSH login:

...
session required pam_exec.so /usr/local/sbin/login_notify.sh

Backups with rsnapshot

rsnaphot uses rsync and hard links in a very clever way to create easy accessible backups without wasting much disk space.

On my home NAS I have two hard disks, one holding the data which is shared via NFS, the other is the backup drive. I perform 7 daily backups followed by a weekly backup. That way I always have an incremental backup of the last 7 days, plus snapshots of every week. For example at the moment my backup drive looks like that:

drwxr-xr-x 3 root    root    4.0K Aug 24 04:20 daily.0
drwxr-xr-x 3 root    root    4.0K Aug 23 04:10 daily.1
drwxr-xr-x 3 root    root    4.0K Aug 22 03:50 daily.2
drwxr-xr-x 3 root    root    4.0K Aug 21 03:57 daily.3
drwxr-xr-x 3 root    root    4.0K Aug 20 03:51 daily.4
drwxr-xr-x 3 root    root    4.0K Aug 19 03:53 daily.5
drwxr-xr-x 3 root    root    4.0K Aug 18 04:09 daily.6
drwxr-xr-x 3 root    root    4.0K Aug 14 03:36 weekly.0
drwxr-xr-x 3 root    root    4.0K Aug  7 03:36 weekly.1
...

Where each directory represents a “snapshot” of the particular time.

After installing rsnapshot edit/add these values of the config file /etc/rsnapshot.conf

# All snapshots will be stored under this root directory.
# (E.g. that's the mount point of my backup hard disk)
snapshot_root  /backup

# Comment out the backup levels and create
retain	daily	7
retain	weekly	52

# Specify the log file
logfile	/var/log/rsnapshot.log

# The directories to backup and excludes
# (E.g. that's the mount point of my NFS share hard disk and I 
# exclude all 'tmp' directories)
backup	/nfs/	all/
exclude	tmp/

I use a bash scripts which launch rsnapshot and send my an email with the result, so that I notice soon when something’s going wrong. There’s only one thing which is worse than having no backups, it’s wrongly assuming you have working backups ;-)

#!/bin/bash
mount /backup

/usr/bin/rsnapshot daily

umount /backup

tail -14 /var/log/rsnapshot.log | mutt -s "rsnapshot backup daily" xxx@example.com

(the ‘weekly’ one is basically the same; just make sure you don’t run them at the same time)