flopska.com - computer science

Highlighting pandas .to_latex() output in bold face for extreme values

2021-01-07T00:00:00+01:00

When preparing a table with experimental results for publication, one often wishes to highlight the output of extreme cells, for example by putting them in bold.

However, pandas does not readily support this use case. The closest one gets with stock pandas is table.style.highlight_max(axis=1) which highlights the maximum values in yellow. Note that this does not work properly with multi-indexed columns: Here only one extreme value per row will be highlighted.

So, what one wishes is to automatically output a pandas dataframe as latex while highlighting certain column-sets using their maximum and others by using their minimum. Let's do that!

Code

We start by creating a small test dataframe:

import pandas as pd

test = pd.DataFrame({"foo" : [0,2,0,0.5,2,1],
    "bar":[1,1,0,1.5,1,1],
    "method": ["x","x","x","y","y","y"]},
    index=[0,1,2,0,1,2]).pivot(columns="method")
test

Now, define the function which does the magic:

def bold_extreme_values(data, format_string="%.2f", max_=True):
    if max_:
        extrema = data != data.max()
    else:
        extrema = data != data.min()
    bolded = data.apply(lambda x : "\\textbf{%s}" % format_string % x)
    formatted = data.apply(lambda x : format_string % x)
    return formatted.where(extrema, bolded)

Additionally, we have to tell python which columns we would like to highlight by the maximum and for which to use the minimum:

col_show_max = { "foo": True, "bar" : False}

Now we run our method on the dataframe and output the result:

for col in test.columns.get_level_values(0).unique():
    test[col] = test[col].apply(lambda data : bold_extreme_values(data, max_=col_show_max[col]),axis=1)
print(test.to_latex(escape=False))

\begin{tabular}{lllll}
\toprule
{} & \multicolumn{2}{l}{foo} & \multicolumn{2}{l}{bar} \\
method &              x &              y &              x &              y \\
\midrule
0 &           0.00 &  \textbf{0.50} &  \textbf{1.00} &           1.50 \\
1 &  \textbf{2.00} &  \textbf{2.00} &  \textbf{1.00} &  \textbf{1.00} \\
2 &           0.00 &  \textbf{1.00} &  \textbf{0.00} &           1.00 \\
\bottomrule
\end{tabular}

And that's it :)

EDIT: Martin Isaksson built upon this, see here. Thanks for letting me know!

Affine transformations in python

2020-11-24T00:00:00+01:00

Here is just a short note on how to do an affine transformation in python succinctly.

Code

import numpy as np
import matplotlib.pyplot as plt
data = np.ones((1000,3)) # needed for the transformation
data[:,0:2] = np.random.uniform(low=-1,high=1,size=(len(data),2)) # create some data

deg = np.pi/4 # 90°
rot = np.array([
[np.cos(deg),-np.sin(deg),0],
[np.sin(deg), np.cos(deg),0],
[0          ,0           ,1]])
trans = np.array([
[1,0,1],
[0,1,0],
[0,0,1]])

# this is where the magic happens
data = data.dot((rot@trans).T)[:,:2]

So, doing an affine transformation with a dataset is, given the transformation matrix, actually a one-line in Python with numpy.

Result

Before the transformation:

After the transformation:

Understanding the local outlier factor (LOF) algorithm

2020-11-20T00:00:00+01:00

This article explains the well-known LOF-algorithm. We provide intuition for density-based outlier detection, show the problems inherent to this task and then take a look at how LOF solves these problems.

Motivation

Outlier detection is important for many real world applications - typical examples include fraud detection, network intrusion detection or the detection of faulty sensor data. Although there is no "real" definition of what constitutes an outlier, the well-established definition is due to ¹:

[...] an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism.

How does one go about finding these observations? The literature distinguishes different models to detect outliers ²: extreme values, clustering models, distance-based models, density-based models, probabilistic models, and information theoretic models. For our purposes, distance-based and density-based models are the most relevant. Note that both are closely linked.

Intuition

The original paper ³ introduced the local outlier factor (LOF) as a density-based model. Going with the disambiguation from ², this means that "the local density of a data point is used to define its outlier score". Let's visualize this:

Looking at the figure, we expect $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} O_1$ and $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} O_2$ to have the highest outlier score - even though $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} O_1$ has the same distance from the cluster in the bottom left as all the points have to each other in the cluster on the top right.

The idea of LOF is to look at the neighborhood of each point individually and to compute its "outlierness" based on its neighbors. To this end, one quantifies the difference of the density of the point to the density of its neighboring points. Put more bluntly: Imagine living in a villa in central Manhattan - then you are probably an outlier because Manhattan is a high-density area but you are in a less dense spot. Now, imagine living a rural area. Here you are also in a less dense spot but so are all of your neighbors and you are not an outlier.

Formal Definition

Let's get to the definitions of the paper and see how they map to our intuition of density-based outlier detection. All of the following definitions are repeated in their original form from ³. Note that $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} D$ refers to the full data set.

k-distance of an object p: For any positive integer $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} k$ , the $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} k$ -distance of object $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} p$ , denoted as k-distance(p) is defined as the distance $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} d(p,o)$ between $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} p$ and an object $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o \in D$ such that:

for at least $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} k$ objects $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o' \in D \setminus \{p\}$ it holds that $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} d(p,o') \leq d(p,o)$ , and
for at most $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} k-1$ objects $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o' \in D\setminus \{p\}$ it holds that $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} d(p,o') < d(p,o)$ .

Put differently, the k-distance of an object is the distance to the k-th nearest object. Or, starting from an object $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o$ , we compute the distance to all objects $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o' \in D$ , sort them in increasing order and take the k-th element of this list.

k-distance neighborhood of an object p: Given the k-distance of $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} p$ , the k-distance neighborhood of p contains every object whose distance from $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} p$ is not greater than the k-distance, i.e., $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} N_{k-distance(p)}(p) = \left\{q \in D\setminus\{p\} : d(p,q) \leq k-distance(p) \right\}$ .

Let's illustrate this with an example. Suppose we have five data point in our data set $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} D=\{o_1,\dots,o_5\}$ and we are interested in the 3-distance of $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o_1$ as well as its neighborhood. We compute $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} d(o_1,o_n)$ for all $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} n =2,\dots, 5$ and order the results. Assume these are the distances:

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} d(o_1,o_2) &= 0.2\\ d(o_1,o_3) &= 4.0\\ d(o_1,o_4) &= 0.5\\ d(o_1,o_5) &= 0.5 \end{aligned}$

Then our 3-distance is 0.5 and there are three objects in the neighborhood.

In Euclidian space, think of this as putting a (hyper-)sphere with its center at $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o_1$ and increasing the radius until k elements are in it. As distances can be equal there might be more than k elements in the circle.

reachability distance of an object p w.r.t. object o: Let $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} k$ be a natural number. The reachability distance of object $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} p$ w.r.t. $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} o$ is defined as

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} reach-dist_k(p,o) = max \{ k-distance(o),d(p,o)\}$

This is used to smooth the density estimation: For objects far away from each other, their distance is the actual distance. For objects close to each other, their distance is their k-distance.

Please note that the reach-distance is not a proper distance measure as is not symmetric.

local reachability density of an object p: The local reachability density of p is defined as

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} lrd_{MinPts}(p) = \frac{|N_{MinPts}(p)|} {\sum_{o \in N_{MinPts}(p)} reach-dist_{MinPts}(p,o)}.$

Here, MinPts "replaces" parameter k and we look at the MinPts-neighborhood of p.

In general, a density is mass per volume, i.e., this definition implicitly assumes each element in the neighborhood of p to have unit mass. "The volume" is given by the sum of all reachability-distances of the element in the neighborhood of a point.

With these preliminaries done, we can finally define the local outlier factor:

(local) outlier factor of an object p: The local outlier factor of p is defined as

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} LOF_{MinPts}(p) = \frac{\sum_{o\in N_{MinPts}(p)}\frac{lrd_{MinPts}(o)}{lrd_{MinPts}(p)}}{|N_{MinPts}(p)|}.$

We can actually rewrite this formula slightly,

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} \frac{\sum_{o\in N_{MinPts}(p)}\frac{lrd_{MinPts}(o)}{lrd_{MinPts}(p)}}{|N_{MinPts}(p)|} &= \frac{\sum_{o\in N_{MinPts}(p)}lrd_{MinPts}(o)}{lrd_{MinPts}(p)|N_{MinPts}(p)|}\\ &= \frac{\sum_{o\in N_{MinPts}(p)}lrd_{MinPts}(o)}{\frac{|N_{MinPts}(p)|} {\sum_{o \in N_{MinPts}(p)} reach-dist_{MinPts}(p,o)}|N_{MinPts}(p)|}\\ &= \frac{\sum_{o\in N_{MinPts}(p)}lrd_{MinPts}(o)\sum_{o \in N_{MinPts}(p)} reach-dist_{MinPts}(p,o)}{|N_{MinPts}(p)|^2}, \end{aligned}$

so the local outlier factor of a point is given by the density of its neighboring points, times the sum of the reach-distance to these points.

Based on this last formula, we see that a point has a large local outlier factor if its neighbors are in dense areas (then the first sum becomes large) and if these points are far away, i.e., the point has no direct neighbors. It turns out that this notion captures the locality aspect of outlier detection surprisingly well.

Discussion

The main algorithmic difficulty of LOF is finding the neighbors of each point in the data set. The scikit-learn implementation mitigates this problem by constructing a Ball Tree or a kd-tree. To speed up the computation further, one could use a more fuzzy approach: Instead of of looking at a specific neighborhood one looks at a so-called " $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \epsilon$ -neighborhood" and thus allows a controllable error. See, e.g., ⁴.

The main difficulty from a user's point of view is setting the parameter k appropriately - the best value of k depends on the data set and the algorithm is very sensitive with regard to its parameter.

References

D Hawkins. Identification of Outliers. Springer Netherlands, 1980. ↩
C C Aggarwal. Outlier Analysis. Springer International Publishing, 2nd ed., 2017. ↩↩
M M Breunig et al. LOF: identifying density-based local outliers. Proceedings of the 2000 ACM SIGMOD international conference on Management of data (New York, NY, USA, May.-2000), 93–104. ↩↩
P Ram and K Sinha. Revisiting kd-tree for Nearest Neighbor Search. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage AK USA, Jul.-2019), 1378–1388. ↩

The repertoire method and the radix-based solution to the Josephus problem

2020-07-31T00:00:00+02:00

This post summarizes how one uses the repertoire method (as presented in Concrete Mathematics by Graham, Knuth and Patashnik). First we look at the repertoire method without the need for a radix-based solution and afterwards we discuss the solution given in the book for Exercise 16.

General method

Suppose we are given the recursion

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} T(0) &= \alpha \\ T(n) &= \beta n^2 + \gamma n + \delta + T(n-1). \end{aligned}$

Starting the recursion, we find that $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} T(n)$ is of the form $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} T(n) = \alpha A(n) + \beta B(n) + \gamma C(n) + \delta C(n)$ and we can use the repertoire method.

There a two ways to proceed: One is to guess values for the parameters and to find fitting functions for $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} A(n),B(n), C(n), D(n)$ , the other one is to guess functions and to find fitting values for $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \alpha,\beta,\gamma,\delta$ . We proceed by guessing functions.

Function guessing

The idea here is to guess functions of which a linear combination results in a closed form solution to the recurrence given above. We start by guessing that

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned}T(n) &= 1.\end{aligned}$

It follows that

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned}T(0) = \alpha = 1\end{aligned}$

and

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned}T(n) &= \beta n^2+\gamma n + \delta + T(n-1) \\ 1 &= \beta n^2+\gamma n + \delta + 1.\end{aligned}$

By comparing the coefficients for all $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} n$ we see that $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \beta=\gamma=\delta=0$ . Hence, $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} A(n) = 1$ .

From setting $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} T(n) = n$ , we find that $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (\alpha,\beta,\gamma,\delta) = (0,0,0,1)$ . Hence, $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} D(n) = n$ .

Setting $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} T(n) = n^2$ gives us $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (\alpha,\beta,\gamma,\delta) = (0,0,2,-1)$ . Hence,

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} n^2 &= 2C(n) - D(n) \\ &= 2C(n) - n \\ C(n) &= \frac{n^2+n}{2}. \end{aligned}$

Finally, we set $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} T(n) = n^3$ and get $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (\alpha,\beta,\gamma,\delta) = (0,3,-3,1)$ . Hence,

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} n^3 &= 3 B(n) - 3 C(n) + D(n) \\ B(n) &= \frac 1 3 n^3 + \frac 1 2 n^2 + \frac 1 6 n, \end{aligned}$

solving our recurrence. Comparing the result with this video we set $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (\alpha,\beta,\gamma,\delta) = (7,2,0,7)$ . This gives

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} T(n) &= 7 A(n)+ 2 B(n) + 0 C(n) + 7 D(n) \\ &= 7 + 2 \left( \frac 1 3 n^3 + \frac 1 2 n^2 + \frac 1 6 n\right) + 7 n \\ &= 7 + \frac 2 3 n^3 + n^2 + \frac{22}{3} n \end{aligned}$

as in the video.

Radix-based solution

On page 16 the book states that a recurrence of the form

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} f(j) &= \alpha, \qquad\qquad\;\text{for } 1 \leq j < d; \\ f(dn + j) &= cf(n) + \beta_j \quad \text{for } 0 \leq j < d \text{ and } n \geq 1 \end{aligned}$

has the radix-changing solution

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} f((b_m b_{m-1}\dots b_1 b_0)_d) = (\alpha_{b_m}\beta_{b_{m-1}}\beta_{b_{m-2}}\dots\beta_{b_1}\beta_{b_0})_c.$

Parameter guessing

Armed with this knowledge we can now solve exercies 1.16:

1.16 Use the repertoire method to solve the general four-parameter recurrence

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} g(1) &= \alpha \\ g(2n+j) &= 3g(n) + \gamma n + \beta_j, \quad \text{for } j = 0,1 \text{ and } n \geq 1. \end{aligned}$

Hint: Try the function $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} g(n) = n$ .

Let's start by using the hint given in the exercise. Using the same approach as before, setting $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} g(n) = n$ implies

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} g(1) &= 1 = \alpha \\ 2n &= 3n + \gamma n + \beta_0 \\ 2n + 1 &= 3n + \gamma n + \beta_1. \end{aligned}$

The values $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (1,0,1,-1)$ for the parameters $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (\alpha,\beta_0,\beta_1,\gamma)$ satisfy these equations for all n.

Thus $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} n = A(n) + C(n) - D(n)$ .

Now, if we, by an educated guess, let $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (\alpha,\beta_0,\beta_1,\gamma)$ be $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} (1,0,1,0)$ we get

$\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} \begin{aligned} g(1) &= 1 \\ g(2n) &= 3g(n) \\ g(2n+1) &= 3g(n) + 1. \end{aligned}$

This looks exactly like the recurrence to which we already know the solution and gives us $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} A(n), B(n)$ and $\renewcommand{\R}{\mathbb R} \newcommand{\E}{\mathbb E} \newcommand{\Var}{\mathrm{Var}} C(n)$ .

Combining both results we find the closed form.

A hint of Eclipse in Visual Code

2019-05-21T00:00:00+02:00

Professionally I am a heavy eclipse user. I heard that IntelliJ is better by now but I have yet to make the switch. However, the focus of eclipse clearly is Java programming. For my own projects I tend to use python - and while vim is awesome I prefer the support of Microsoft's Visual Code.

Eclipse shortcuts in Visual Code

The Eclipse IDE has its own (huge) set of shortcuts: to move around code, to search, to refactor, to run tests, to run a gradle task, ... . The list is endless. Some time ago we literally printed the standard shortcuts on two sheets of paper and marked the ones we knew - so that we could focus on memorizing the rest.

Compared to all the support eclipse provides when refactoring java, the support of Visual Code seems minimal. Checking the section 'Refactoring' in the documentation, we can:

Extract Variable
Extract Method
Sort Imports

Don't ask me why there is no Inline - but at least there is something.

The refactoring actions can be called from the command palette (Ctrl+Shift+P). Additionally one can provide one's own keyboard shortcuts. Because I don't want to memorize some (arbitrary) new set of commands I stick with the ones I know from eclipse.

So, in order to define the settings: Open the command palette, search for Shortcuts and open the JSON-Config.

[
    {
        "key": "shift+alt+l",
        "command": "python.refactorExtractVariable"
    },
    {
        "key": "shift+alt+m",
        "command": "python.refactorExtractMethod"
    },
    {
        "key": "ctrl+shift+o",
        "command": "python.sortImports"
    }
]

Copy and paste the lines above, eh voilá. Now commit these to memory and get used to refactoring in python :)

Nice LaTeX plots with matplotlib

2019-05-07T00:00:00+02:00

There exist various ways to use LaTeX fonts in the matplotlib plots. When I first googled this I came upon embedding *.pgf files directly into the document which is then supposed to use the same font as the rest of the document. However I had problems with the correct size of the resulting images.

Now I embed the *.pdf files directly which has the added advantage that it is easy to preview the files without running LaTeX. Before I always saved two files per image: One pdf for viewing, one pgf for including.

Setup

matplotlibrc

The following settings are necessary in the matplotlibrc file. Create this file if it does not exist and put it in the folder from which you run your notebooks. I think it's pretty self-explanatory what each setting does, otherwise the matplotlib-help is of great value.

axes.labelsize     : 12
axes.titlesize     : 14
figure.titlesize   : 14
legend.fontsize    : 10
savefig.bbox       : tight
xtick.labelsize    : 10
ytick.labelsize    : 10

font.cursive       : Zapf Chancery
font.family        : lmodern
font.monospace     : Courier, Computer Modern Typewriter
font.sans-serif    : Helvetica, Avant Garde, Computer Modern Sans serif
font.serif         : Times, Palatino, New Century Schoolbook, Bookman, Computer Modern Roman
font.size          : 10

pgf.rcfonts        : False

text.usetex        : true

Jupyter Notebook

To save to a pdf file from your notebook:

plt.savefig('figures/correlation.pdf')

LaTeX document

In the LaTeX document the image can be included using this code:

\begin{figure}[h!]
   \centering
   \includegraphics[scale=0.8]{figures/correlation.pdf}
   \caption{The value of the incremental correlation coefficient computed for two random variables.}
   \label{fig:correlation-coefficient}
\end{figure}

Also you have to add

\usepackage{lmodern}

to your front matter.

Result

The resulting image looks like this, I think it's pretty neat.

The sizes can also be adopted separately for each plot. For more information on font sizes, look here.

Edit: KarelZe showed me another way to achieve this result: https://gist.github.com/KarelZe/778b77dcc8dd30e59dae8f14c139eb28

Redis Cluster als Basis für einen verteilten Chat

2015-04-03T00:00:00+02:00

Anforderungen
Software Stack
Software Architektur
Hardware Architektur
Ausblick

Anforderungen

mehrere Clients pro Benutzer
Integrationstest des Servers als Slim Client
Verlauf abrufbar
verschiedene Server mit Authorisierung (falls einzelne Komponenten ausfallen immer noch lauffähig)
Authorisierung an jedem Server möglich
Persistierung der Nachrichten auf allen Servern/Nodes
kein Datenschutz
Clients pullen, Server pushen
Nachrichten so schnell wie möglich auf alle anderen Server
alle schreiben an alle (ein "chatroom", broadcast)
Login mit E-Mail und Password

Software Stack

redis wird als zentrale Datenbank für die Benutzer und die gesendeten Nachrichten verwendet. Mit redis 3.0.0 ist clustering und somit der verteilte Einsatz in den stable-Branch gekommen.
jedis ist das empfohlene Java Interface für die Kommunikation mit redis.
restlet stellt ein Framework für RESTful Anwendungen zur Verfügung. Neben dem Server können auch Clients realisiert werden. Diese Funktionalität wird in den Integrationstests verwendet.
junit
hamcrest
lombok

Software Architektur

Mit restlet werden zwei Endpunkte für Anfragen zur Verfügung gestellt:

http://localhost:8182/user/
http://localhost:8182/message/

/user/

public interface RESTUser {

    @Put("json")
    Representation login(ChatUser chatUser);

    @Post("json")
    ChatUser createUser(ChatUser chatUser);

}

Mit der createUser(ChatUser chatUser) Methode kann ein neuer Benutzer am System registriert werden. Die E-Mail Adresse ist dabei eindeutig, d.h. es kann pro E-Mail Adresse nur einen Benutzer im System geben. Wird der Nutzer angelegt enthält der Rückgabewert die generierte ID und den Authentifizierungsschlüssel des neuen Benutzers.

Die login(ChatUser chatUser) Methode wird zum Einloggen eines Benutzers verwendet. Der Benutzername und das Passwort werden überprüft und bei einer Übereinstimmung wird der entsprechende Authentifizierungsschlüssel als Cookie zurückgegeben. Bei einem Fehler sendet der Server ein HTTP Status 401.

/message/

public interface RESTMessage {

    @Post("json")
    Message sendMessage(Message message);

    @Get("json")
    List<Message> readMessages();

}

Das Senden einer Nachricht setzt voraus, dass der Benutzer sich vorher authorisiert hat. Das bedeutet, es muss neben der Nachricht auch ein Cookie mit dem Authentifizierungsschlüssel gesendet werden. War das Senden der Nachricht mit sendMessage(Message message) erfolgreich enthält die zurückgegebene Nachricht den timestamp, bei dem die Nachricht am Server eingegangen ist.

Zum Lesen von Nachrichten ist keine Authentifizierung erforderlich. readMessages liefert alle auf dem Server gespeicherten Nachrichten zurück.

Hardware Architektur

Der Chat verwendet die klassische Architektur für Webapplikationen: Die Anfragen der Nutzer werden von einem Load Balancer entgegen genommen und auf die Web Server verteilt. Soll eine größere Last verarbeitet werden, können weitere Web Server hinzugefügt werden. Die Authentifizierungsinformationen für Benutzer sind clientseitig in Cookies und serverseitig in der Datenbank gespeichert. Somit hält die Webapplikation keinen State: Fällt ein Server aus routet der Load Balancer die Anfragen an einen anderen Server und der Benutzer bemerkt nichts vom Ausfall eines Servers.

redis cluster hat einen Failover Mechanismus eingebaut: Beim Ausfall eines Masters wird automatisch ein bestehender Slave zum Master. In der verwendeten Konfiguration können also im Besten Fall bis zu drei Datenbankserver ausfallen.

Ausblick

In der oben beschriebenen Konfiguration ist der Load Balancer ein Single Point of Failure. Eine Möglichkeit auf den Ausfall des Load Balancers zu reagieren ist der Einsatz von sogenannten Elastic IP Addresses (siehe Elastic IP Addresses bei AWS). Fällt der Load Balancer aus kann seine IP zeitnah einem parallel laufenden Load Balancer zugewiesen werden.

Getting started with vagrant and ansible

2014-04-22T00:00:00+02:00

vagrant is a tool to create and configure reproducible development environments. Wat?!

Using virtual machines as your development environment has got multiple benefits: When you're a developer it's easy to provide every team member with the same environment. When your a designer you don't have the hassle of setting up the workspace and can focus on the design instead. When you're done and want to deploy your software you can use the same process you used to setup your virtual machines to provision your production system.

Sounds dope. Where can I get some?

Download the vagrant version for your system and install it. On Ubuntu that's

sudo dpkg -i vagrant_1.5.4_x86_64.deb

Eventually you have to install the virtualbox packages, too. Enter

sudo apt-get install virtualbox

That's it. Now you're ready to setup your first virtual machine. Keep calm, I will guide you:

vagrant init hashicorp/precise32 
vagrant up
vagrant ssh

Easy, heh? We just downloaded an Ubuntu precise image, created a so called Vagrantfile, booted the machine and connected to it. Halt the machine by disconnecting from the ssh session and entering vagrant halt.

Speedup vagrant execution with a ramdisk

2014-04-02T00:00:00+02:00

Currently I am using vagrant in conjunction with ansible to provide all team members of our university project with a consistent development and testing environment. Before I push my changes I want to make sure that every ansible playbook I wrote is working and idempotent. To achieve this goal I have to test the scripts a lot and in doing so I spin up a dozen of new machines.

To be able to run the necessary software the machines have to download a lot of packages. Some are only available as source and have to be compiled. Because the bottleneck when running the machines often lies within the HDD I thought it would be nice to speed it up a bit by putting the virtual machines on a ramdisk.

Setting up the ramdisk

In Ubuntu 13.10 'saucy' it is rather simple to set up a ramdisk:

sudo mkdir /media/ramdisk
sudo mount -t ramfs ramfs /media/ramdisk
sudo chown -R user:group /media/ramdisk

Note: Please don't store any important information on the ramdisk as the data is lost when powering off the computer.

To automatically mount the partition when booting add this to your /etc/fstab:

ramfs       /media/ramdisk      ramfs       defaults        0       0

It is not possible to limit the size of the ramdisk. In extreme cases the host system may run out of memory.

Moving vagrant and virtualbox into the RAM

Vagrant usually stores it's files in ~/.vagrant.d/. To use another path set the VAGRANT_HOME environment variable.

cp -r ~/.vagrant.d /media/ramdisk/vagrant_home
export VAGRANT_HOME=/media/ramdisk/vagrant_home

Now you have to change the location of your virtual machines from within virtualbox. Open the client by entering virtualbox into the terminal, go to 'File -> Preferences...' and enter your new ramdisk location as 'Default Machine Folder'.

Performance improvements

All tests were run with a new Vagrantfile and a 'raring64' box.

Start up time without ramdisk:

time vagrant up
...
vagrant up  2,63s user 1,99s system 12% cpu 35,825 total

Start up time with ramdisk:

time vagrant up
...
vagrant up  2,55s user 2,07s system 17% cpu 26,853 total

As you can see starting the image from a ramdisk is faster than starting from a regular HDD. This is the case because vagrant copies the base box (in this case 'raring64') before starting it. However, there are no performance gains while running the vm (tested with hdparm -Tt /dev/sda). This is the case because virtualbox is already caching the hard drive access into the ram.