forsooth!

Partial Bézier Curves

2019-04-22T00:00:00+02:00

Motivation

I recently worked on some visualization with d3, rendering a directed graph of nodes with a force simulation. An easy way to render the edges of the graph is to simply draw a straight line, but that’s not very pretty. Instead, a neat trick is to introduce an invisible node inbetween two connected nodes, and then render a quadratic Bézier curve between the nodes, using the invisible node as control point.

This example shows the curved link between two nodes, without force simulation, the nodes can be dragged around interactively. The nodes are transparent to illustrate the actual ends of the curve.

The relevant code for rendering the path of the edges in d3:

var edgeElements = edgeLayer
    .selectAll(".edge")
    .data(edges);

edgeElements.enter()
    .append("path")
    .attr("class", "edge")
    .attr("stroke", "#ddd")
    .attr("stroke-width", 5);

// on update
edgeLayer
    .selectAll(".edge")
    .attr("d", e => {
        return "M" + e.source.x + "," + e.source.y
            + "Q" + e.intermediate.x + "," + e.intermediate.y
            + " " + e.target.x + "," + e.target.y;
    });

(See SVG:path docs for explanation of the M and Q commands.)

This connects the centers of each node, which is great for undirected graphs, but not very useful for directed graphs. Where and how should the arrow be rendered? And how can the curve be rendered only from node edge to node edge? This is particularly important if the nodes are transparent, as above, or have non-circular shapes, for instance based on an SVG pattern fill, a logo, etc..

Solution

Finding the touching points

The first step must be finding the touching points of the curve with the rim of the node. The “rim” will be assumed based on the radius of the node, even if the node itself might have an irregular shape, but since I want a little spacing between node and curve anyway, this is sufficient.

Wikipedia has a more thorough explanation of Bézier curves, but for what I’m doing I need this function:

\[ B\colon [0, 1] \Rightarrow \mathbb{R}^2 \] \[ B(t) \, = \, (1 - t)^2 P_0 + 2 (1 - t) t P_1 + t^2 P_2 \]

Where \(P_0\) and \(P_2\) are the end points of the curve and \(P_1\) is the control point. It’s easy to see that \(B(0) = P_0\) and \(B(1) = P_2\).

Let’s assume the radius of our node around \(P_0\) is \(r\), then we need to find the smallest \(t\), such that \(|B(t) - P_0| = r\), i.e. the \(t\) at which the curve is at a distance of \(r\) from \(P_0\) for the first time. In most curves there’ll only be one such point, but just in case the nodes are very close together, perhaps even overlapping, the first one is the point we want. The same can be done from the end.

I tried to solve this mathematically, but even for quadratic Bézier curves this grows into big expressions and polynomials of 4th degree, so I went with a binary search for a close approximation instead:

function calculateBezierPoint(P0, P1, P2, t) {
    return {
        x: (1 - t) * (1 - t) * P0.x + 2 * (1 - t) * t * P1.x + t * t * P2.x,
        y: (1 - t) * (1 - t) * P0.y + 2 * (1 - t) * t * P1.y + t * t * P2.y
    };
}

function findProximityPoint(P0, P1, P2, r) {
    var rSquared = r * r;

    var dt = 0.05;
    var t = dt;
    var lastWasLower = true;

    while (t >= 0 && t <= 1) {
        var p = calculateBezierPoint(P0, P1, P2, t);
        p.t = t;

        var distSquared = (p.x - P0.x) * (p.x - P0.x) +
                          (p.y - P0.y) * (p.y - P0.y);
        if (lastWasLower && distSquared > rSquared) {
            dt /= -2;
            lastWasLower = false;
        } else if (!lastWasLower && distSquared <= rSquared) {
            dt /= -2;
            lastWasLower = true;
        }

        if (Math.abs(dt) < 0.001 && distSquared > rSquared) {
            return p;
        }

        t += dt;
    }

    return {
        x: P0.x,
        y: P0.y,
        t: 0
    };
}

And the relevant code in d3 to render the path of the edge, which now first calculates the points on the curve close to the node edges and then uses those points as start and end of the curve:

edgeLayer
    .selectAll(".edge")
    .attr("d", e => {
        var start = findProximityPoint(e.source, e.intermediate, e.target, e.source.radius + 2);
        var end = findProximityPoint(e.target, e.intermediate, e.source, e.target.radius + 2);

        return "M" + start.x + "," + start.y
            + "Q" + e.intermediate.x + "," + e.intermediate.y
            + " " + end.x + "," + end.y;
    });

This works quite well, but it isn’t perfect. Since we use the same control point with different start points, the curve differs quite a bit. The ends also don’t necessarily point towards the center of the nodes anymore, which you can easily see by moving the control point around one of the points.

Adding an arrow

To illustrate this, below the same example with the original edge included, and including an arrow marker to better see where it points. For that the end’s proximity point must be pushed out a bit further (by the length of the arrow) and the arrow is achieved like this:

var defs = svg.append('svg:defs');
var arrowMarker = defs.append("marker")
    .attr("id", "arrow")
    .attr("markerWidth", 10)
    .attr("markerHeight", 10)
    .attr("refX", 4)
    .attr("refY", 3)
    .attr("orient", "auto")
    .attr("markerUnits", "strokeWidth");
arrowMarker.append("path")
    .attr("d", "M1,3 L0,5 L6,3 L0,1 z")
    .attr("fill", "#ddd");

// applying it to the edges

var edgeElements = edgeLayer
    .selectAll(".edge")
    .data(edges);

edgeElements.enter()
    .append("path")
    .attr("class", "edge")
    .attr("stroke", "#ddd")
    .attr("stroke-width", 5)
    .attr("marker-end", "url(#arrow)");

Construct a partial curve

Ideally the curve between the two touching points follows the original curve. We know the values \(t_{start}\) and \(t_{end}\) for which \(B(t_{start}) = P_{start}\) and \(B(t_{end}) = P_{end}\), so \(B(t)\) could simply be rendered for \(t \in [t_{start}, t_{end}]\) instead of \(t \in [0, 1]\). Unfortunately SVG’s path commands don’t support such a feature.

To achieve the effect nonetheless, we first define a new function \(C\) which follows \(B\) along the relevant interval, but itself using the domain \([0, 1]\):

\[ C\colon [0, 1] \Rightarrow \mathbb{R}^2 \] \[ C(t) = B(t_{start} + (t_{end} - t_{start}) t) \]

We also know the start and end points:

\[ P_{start} = C(0) = B(t_{start}) \] \[ P_{end} = C(1) = B(t_{end}) \]

Finally, we want \(C\) to be a Bézier curve, so it must have the same structure. Calling the control point \(X\) we have:

\[ C(t) \, = \, (1 - t)^2 P_{start} + 2 (1 - t) t X + t^2 P_{end} \]

Putting it all together it is: \[ (1 - t)^2 P_{start} + 2 (1 - t) t X + t^2 P_{end} \, = \, \] \[ (1 - (t_{start} + (t_{end} - t_{start}) t))^2 P_0 + \] \[ 2 (1 - (t_{start} + (t_{end} - t_{start}) t)) (t_{start} + \] \[(t_{end} - t_{start}) t) P_1 + (t_{start} + (t_{end} - t_{start}) t)^2 P_2 \]

which means

\[ (1 - t)^2 ((1 - t_{start})^2 P_0 + 2 (1 - t_{start}) t_{start} P_1 + t_{start}^2 P_2) + \] \[ 2 (1 - t) t X + \] \[ t^2 ((1 - t_{end})^2 P_0 + 2 (1 - t_{end}) t_{end} P_1 + t_{end}^2 P_2) \, = \, \] \[ (1 - (t_{start} + (t_{end} - t_{start}) t))^2 P_0 + \] \[ 2 (1 - (t_{start} + (t_{end} - t_{start}) t)) (t_{start} + \] \[(t_{end} - t_{start}) t) P_1 + (t_{start} + (t_{end} - t_{start}) t)^2 P_2 \]

and sympy helps us with that, yielding:

\[ X \, = \, t_{start} t_{end} (P_0 - 2 P_1 + P_2) + (1 - t_{start} - t_{end}) P_0 + (t_{start} + t_{end}) P_1 \]

So finally we can construct a new Bézier curve from \(P_{start}\) to \(P_{end}\), using control point \(X\). In code it looks like this:

edgeLayer
    .selectAll(".edge")
    .attr("d", e => {
        var start = findProximityPoint(e.source, e.intermediate, e.target, e.source.radius + 2);
        var end = findProximityPoint(e.target, e.intermediate, e.source, e.target.radius + 10);
        var P0 = e.source;
        var P1 = e.intermediate;
        var P2 = e.target;
        var tStart = start.t;
        var tEnd = 1 - end.t;

        var adjustedIntermediate = {
            x: tStart * tEnd * (P0.x - 2 * P1.x + P2.x) + (1 - tStart - tEnd) * P0.x + (tStart + tEnd) * P1.x,
            y: tStart * tEnd * (P0.y - 2 * P1.y + P2.y) + (1 - tStart - tEnd) * P0.y + (tStart + tEnd) * P1.y,
        };

        return "M" + start.x + "," + start.y
            + "Q" + adjustedIntermediate.x + "," + adjustedIntermediate.y
            + " " + end.x + "," + end.y;
    });

For the final result (the wrongly cropped curve included for comparison):

Better HTML rendering for spacemail

2017-12-12T00:00:00+02:00

Problem

The rendering of HTML emails is rather ugly, with white/brown backgrounds, totally breaking with the theme. It also is surprisingly slow, especially for long threads and emails containing a lot of quoting in HTML.

Solution

There’s a variable mm-text-html-renderer, which by default is set to shr. I prefer using the value w3m, which uses w3m and is much faster and prettier. It displays everything in the current theme, highlights links, and displays styles, e.g. strong/bold.

It also can be set up to honour a fill-column while rendering.

Setup

As already pointed out: w3m needs to be installed. Via your favourite packet manager (e.g. pacman, apt, yum, brew).

(setq mm-text-html-renderer 'w3m)
(setq w3m-fill-column 72)

For more information try SPC h d v and type mm-text-html-renderer and RET. There are other renderers available, which might work even better for you.

PoC||GTFO

2017-11-08T00:00:00+02:00

What’s that?

PoC||GTFO is a samizdat with a focus on Reverse Engineering and hacking. It is wonderfully put together, with pretty TeX work, old school ads, and illustrations.

Due to their strict licensing:

Legal Note: We politely ask that you copy this document far and wide.

I’ll mirror them here. All issues are available below. Many of these PDFs are polyglots, e.g. they also are valid zip files, and sometimes even executables, the PDFs have more details on that.

file	sha256sum
pocorgtfo00.pdf	`c4d1d1091187b98a9bb28452bc6564a1e8c0ce10d20ba2b4a20f8b7798f7ab64`
pocorgtfo01.pdf	`a0f93a265d38257a06fd7fd210f73ea9c55a94ac1305c65c0510ada236c2cc88`
pocorgtfo02.pdf	`f427e8d95c0ac15abe61d96fb75cfb55df1fd5ac9e713cf968f3602267ca155e`
pocorgtfo03.pdf	`7094f5c6a3936e0d0b8f5e42b4d1940413f568e9a3617be0d7d6dc73cb3420e1`
pocorgtfo04.pdf	`1d1567b8ac533cd142a8af560266ca60939fed02e3af1f6fd0816b26473afd01`
pocorgtfo05.pdf	`9623609a9c0ecd95674e6da3de322baa141f5460cbcb93eeaade22eaf2c80640`
pocorgtfo06.pdf	`bf4d8846fbbb1071c7ec033004eda8ea8809676fe388db6faa020d781cb8ac26`
pocorgtfo07.pdf	`601534f4355c5e0eb292c6dd6edaf5055625d23e0de869f88193606415e6a35f`
pocorgtfo08.pdf	`7a942c425f471f99d8cba8da117cc4a53cddb3551e4b16c8b9feae31b5654a33`
pocorgtfo09.pdf	`8ad70d4dd0c0f53e8c479d1d573e5a365ea673acafa9fd61fa5231e18502a6ad`
pocorgtfo10.pdf	`1e350e30383fd332678654b6067fe4b6ea3d25d7f41a24a4c81fe913b295c9de`
pocorgtfo11.pdf	`44d56d717c7b3baf7e11aa6624d5a80a90b132a519e61b9682a5f4a635b04c78`
pocorgtfo12.pdf	`441216e475e69564192f2121daa5dd465835072718366b75b08b9272ff9cf08b`
pocorgtfo13.pdf	`c881c67557af52864654791a2a494f329a2fa397236bf0e961508f0769b0a3f5`
pocorgtfo14.pdf	`b9db617dcc146cc99f4379b3162a35818d884bf4032ab854b6ec00b5ec98138d`
pocorgtfo15.pdf	`c9b3f5026640efae12d75e62868931e2b2b5ad98a9b858408266ac5c35815bf4`
pocorgtfo16.pdf	`10f0cb977f03824737a413079ded14b237b7ee155a5397e804586ab7151ed0a3`
pocorgtfo17.pdf	`40b8985521e671b59c305d2f5512f31b95f1e8c59b9c05ad2ca6413a99d59c97`
pocorgtfo18.pdf	`cd1e5a8f9fdda3a950556ad77db706ab8a01c97662e79c46d417e0e7858aa742` or `f427637767d7aea764e0f2045ea4de86bbf5410a43535e70285add347047b283`
pocorgtfo19.pdf	`891b6c4e0cc8f88af2b8c2467c1558b806d2f21be4c7518e7833c27885713464`

Python/Ruby Polyglot

It also inspired me to play around with polyglots a bit. Albeit nowhere near as clever as the things I’ve read in the issues already, I wanted to write a script that can be run with Python and Ruby. I wanted the results to differ meaningfully, and try to exploit as many tricks as possible to actually make both interpreters run the same lines of code, rather than doing some eval tricks or whatever.

I came up with this script, which encrypts when run with Python3 and decrypts when run with Ruby2, I explicitly disabled syntax highlighting, so it isn’t so obvious what’s going on:

key = """PoC||GTFO"
def read(*) STDIN.read.chars end
def ord(x) x.ord end
"""
import sys
begin, read, STDOUT, end = \
__name__, sys.stdin.read(), sys.stdout, sys.exit
#"

i = 0
for c in read \
    :begin; \
    v = (i % 2) * 2 - 1 and 1 - 2 * (i % 2); \
    STDOUT.write("%c" % ((256 + ord(c) + v * ord(key[i % 9])) & 0xff)); \
    i += 1; \
end

Running it encrypts stdin to stdout:

% cat > input <<EOF
Olympian bards who sung
Divine ideas below,
Which always find us young
And always keep us so.
EOF

% <input python polyglot | hexdump -C
00000000  c3 bf c3 bd 36 c3 b1 c3  b4 22 0d 28 c3 91 12 c3  |....6....".(....|
00000010  b2 2f c3 a8 c3 b7 c3 99  23 22 20 c3 90 04 32 c3  |./......#" ...2.|
00000020  b2 c3 ab c3 83 c3 b0 23  27 19 c3 bf 22 c2 a4 c3  |.......#'..."...|
00000030  ad 1d 11 1b 24 c3 90 c3  b3 22 c3 b0 c3 b3 30 c3  |....$...."....0.|
00000040  98 c3 84 08 18 c3 ba 20  c3 ac c2 a4 1a 18 31 12  |....... ......1.|
00000050  29 04 c3 9d c3 aa c3 ad  27 10 c3 9a 26 23 c2 b1  |).......'...&#..|
00000060  36 c3 b3 c3 b9 27 13 c3  84 c3 b2 1e c3 b5 c3 9d  |6....'..........|
00000070  c3 a5 c3 b0 30 0d 33 24  c3 90 c3 bc 22 c3 a9 c3  |....0.3$...."...|
00000080  b4 c3 99 21 2d c3 91 23  00 c3 ab c2 8e           |...!-..#.....|
0000008d

Python cannot decrypt it:

% <input python polyglot | python polyglot | hexdump -C
00000000  c2 af c2 8e c3 b3 75 78  c3 9b c2 b9 c3 a2 c2 82  |......ux........|
00000010  c3 82 c2 83 c3 ac 6c 7b  c2 92 c3 8f c3 9c c3 91  |......l{........|
00000020  c2 80 c2 95 c3 af 76 6f  7c c2 9c c3 9d c3 98 c3  |......vo|.......|
00000030  89 c2 90 c3 9f 28 71 c3  96 c2 bd c3 95 c3 95 c2  |.....(q.........|
00000040  80 c2 84 c3 9f 74 77 c3  a9 c2 84 7e c2 b9 c3 88  |.....tw....~....|
00000050  c2 8b c3 9d 70 28 c3 93  c3 84 c3 ab c3 83 c3 99  |....p(..........|
00000060  c2 95 c2 9a 6e 71 c3 a0  c2 bc c2 94 c3 97 c3 93  |....nq..........|
00000070  42 c3 b3 77 7d c3 a0 c2  bf 7e c2 a3 c3 8e c2 86  |B..w}....~......|
00000080  c2 9a 69 74 c3 a9 c2 b9  c3 ad c3 95 c2 80 c2 8d  |..it............|
00000090  c3 9f 6d 78 c2 92 c3 8d  c3 a7 c2 82 c3 93 c2 91  |..mx............|
000000a0  c2 a8 12                                          |...|
000000a3

But Ruby can:

% <input python polyglot | ruby polyglot
Olympian bards who sung
Divine ideas below,
Which always find us young
And always keep us so.

And the other way around:

% <input ruby polyglot | python polyglot
Olympian bards who sung
Divine ideas below,
Which always find us young
And always keep us so.

Current Python/Ruby versions:

% python --version
Python 3.6.3

% ruby --version
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]

How does it work?

I won’t go into detail too much. It’s not a very complicated script, but the main tricks employed are:

Ruby doesn’t have """ quotes
Ruby doesn’t require parentheses for function calls
Python has different block syntax
Ruby has silly operator precedence

There are three tricks in there that are not immediately obvious, I think, but I won’t spoil any furher.

The main trick with the """ quotation could be used to write two entirely separate scripts, of course, but that’d be rather boring, so I only used it to get a foot in the door, setting up some symbols I couldn’t easily write around.

Email in spacemacs

2017-10-22T00:00:00+02:00

Motivation

Email is tedious. Especially at work I get hundreds of emails every day, which need to be managed somehow. A lot of those emails are noise, and ideally I’d only ever have to worry about emails that are signal.

Additionally, writing email requires a good editor with vim bindings (duh!), spacemacs delivers. Navigating, tagging, and searching emails with modal keybindings also makes dealing with them less annoying.

This documents my setup, which uses spacemail, isync, msmtp, and notmuch.

Setup

isync/mbsync

isync is a command line tool that syncs a remote IMAP account and a local mail directory. I simply run it periodically to poll for new emails. My fork supports the macOS Keychain for password retrieval.

My ~/.mbsyncrc looks like this:

IMAPAccount forsooth
Host imap.forsooth.org
User or
KeychainName <KeychainName>
KeychainAccount <KeychainAccount>
SSLType IMAPS
AuthMechs LOGIN

IMAPStore forsooth-remote
Account forsooth

MaildirStore forsooth-local
Subfolders Verbatim
# The trailing "/" is important
Path ~/Mail/or@forsooth.com/
Inbox ~/Mail/or@forsooth.com/INBOX/

Channel forsooth
Master :forsooth-remote:
Slave :forsooth-local:
Patterns "INBOX"

Create Slave
SyncState *

This syncs everything in the folder “INBOX”. I leave everything serverside in that folder, because it avoids moving emails around physically and syncing back and forth with IMAP. All the tagging is done locally via notmuch.

In order to sync multiple folders, the list can be extended to include them, it also supports regular expressions. Either way the Calendar folder of Exchange accounts should be ignored, as Exchange does some non-RFC things that might break the sync (at least that used to happen to me with offlineimap, I haven’t seen it with mbsync).

For my work account I also require a specific SSL certificate for the handshake, and the auth configuration differs a little.

CertificateFile "/usr/local/etc/openssl/certs/some-authority.pem"

Credentials via macOS Keychain (recommended)

My config uses KeychainName and KeychainAccount to retrieve the password from the macOS Keychain. This requires Keychain support added in my fork, see above, until it is merged upstream (perhaps).
Credentials via GPG (discouraged)

An alternative to the Keychain is retrieving the password for the account from ~/.authinfo.gpg, an encrypted file containing login information for SMTP and IMAP:
```
machine imap.forsooth.org login or@forsooth.org port 465 password <hunter2>
machine imap.forsooth.org login or@forsooth.org port 993 password <hunter2>
```
In that case you’d have to set a password command like this:
```
PassCmd "gpg --quiet --for-your-eyes-only --no-tty --decrypt ~/.authinfo.gpg"
```
Note that anyone can do this, if they have access to your machine while you are logged in, if you use gpg-agent to avoid constant password prompts. The Keychain method is more secure.

Migrate from offlineimap

If you started with offlineimap and want to switch to mbsync, then that’s easy, just follow the steps outlined here.

notmuch

notmuch processes a local mail directory, building a database that tags and indexes all mails for text searches.

My ~/.notmuch-config is given below, each setting is described in detail in the notmuch documentation and the template for that file.

[database]
path=/home/or/Mail

[user]
name=or
primary_email=or@forsooth.org

[new]
tags=unread;inbox;new;
ignore=

[search]
exclude_tags=deleted;spam;

[maildir]
synchronize_flags=true

[crypto]
gpg_path=gpg

Importantly there is the setup of the user that sends the email and the tags new mail gets automatically.

Here is the rough structure of the script to process new email, ideally running after offlineimap syncs the mail.

# find out how many threads are unread before the sync
previous_unread_count=$(notmuch count --output=threads tag:unread)
notmuch new

# count new and unread email after the sync
new_count=$(notmuch count tag:new)
unread_count=$(notmuch count --output=threads tag:unread)

# if something changed...
if [[ "$new_count" -gt 0 && \
      "$unread_count" -gt 0 && \
      "$unread_count" -ne "$previous_unread_count" ]]; then
  # use $new_count and $unread_count for some notification, omitted here
fi

# remove the "new" tag from all emails that have it
notmuch tag -new tag:new

tagging rules

The main script containing all the rules is ~/Mail/.notmuch/hooks/post-new, which may look like this:

notmuch tag +sent folder:sent
notmuch tag +draft folder:drafts

notmuch tag +me tag:new to:or@forsooth.org
notmuch tag +cron tag:new from:"Cron Daemon"
notmuch tag +invitation-response tag:new 'vcalendar AND (subject:"accepted" OR subject:"declined" OR subject:"tentative")'
notmuch tag +ticket tag:new 'subject:ticket'
notmuch tag +mailing-list tag:new some-topic@mailinglist.com
notmuch tag +build tag:new from:build-tools@some-service.com
notmuch tag +review tag:new subject:"Code Review"
notmuch tag +cake tag:new cake

notmuch tag +noise -inbox -unread tag:new AND tag:build AND NOT tag:me AND "successfully"
notmuch tag +noise -inbox -unread tag:new AND tag:invitation-response
notmuch tag +noise -inbox -unread tag:new AND tag:cron
notmuch tag +noise -inbox -unread tag:new AND tag:review AND NOT tag:me
notmuch tag -inbox tag:new AND tag:mailing-list
notmuch tag -inbox -unread from:or@forsooth.org AND NOT to:or@forsooth.org

notmuch tag +muted "$(notmuch search --output=threads tag:muted)"
notmuch tag +noise -inbox -unread tag:new AND tag:muted

notmuch tag -unread -inbox tag:new AND tag:sent

notmuch tag +signal tag:inbox

This is a normal bash script, which allows some pretty cool rules that’d be hard or impossible to apply in the usual email clients.

In the example config I always filter for mails with the tag new, which makes the processing much faster. Additional tags are applied based on several rules, which identify them as coming from cronjobs or a build service. Many such emails don’t have to be read, so they get the inbox tag removed right away. I also give them a noise tag and the remaining emails a signal tag for some statistics later on.

You may wonder why inbox and unread tags are used in the way they are. I usually filter for tag:inbox AND tag:unread in spacemacs, as you’ll see in the next section, but they both serve a different purpose:

inbox - the email is in the inbox and probably needs some action
unread - the email is worth looking at but hasn’t been read yet

An example is the mailing-list tag, which keeps an email’s unread tag, but removes inbox, because usually there’s a lot of traffic and I just read through them, without expecting each of them to require a response or work.

Anything with both tags will show up in the inbox, after reading it will lose the unread tag, but remain in the inbox until it is archived explicitly.

Note that any email in a thread given the tag muted will also get muted and removed from inbox.

Another example of how this scripting can do a bit more than just tagging is the following block, which pages me whenever an email mentions cake in the office.

  cake_query="tag:new AND tag:cake AND tag:unread AND NOT from:or@forsooth.org"
  num_cake=$(notmuch search $cake_query | wc -l | awk '{print $1}')
  if [[ "$num_cake" > 0 ]]; then
      cake_email="/tmp/cake_email.txt"
      cat  > $cake_email <<EOF
From: pager@forsooth.org
To: pager@forsooth.org
Subject: "${num_cake}x" cake!

There's cake!
EOF
      sendmail -t < $cake_email
  fi

msmtp

In order to send emails via SMTP, I now use msmtp, which can act as a sendmail replacement but sends via an SMTP server. Previously I set up spacemacs to send via SMTP, but that required the .authinfo.gpg method above to retrieve the password. I forked msmtp to also support macOS Keychain access.

My ~/.msmtprc looks like this:

defaults
tls on
# might need this for your server
# tls_trust_file /usr/local/etc/openssl/certs/some-authority.pem

account or
host smtp.forsooth.org
from or@forsooth.org
auth on
user or
keychain_name <KeychainName>
keychain_account <KeychainAccount>

account default : or

Credentials via macOS Keychain (recommended)

Just like in the mbsync configuration, this specifies a KeychainName and KeychainAccount to look up the password in the macOS Keychain. This is currently requires my fork, see above, but hopefully it can be merged upstream.
Credentials via GPG (discouraged)

Again, the password can be read from .authinfo.gpg instead. The configuration would look like this:
```
passwordeval "gpg --quiet --for-your-eyes-only --no-tty --decrypt ~/.authinfo.gpg"
```
But as mentioned above, this allows anyone with access to an unlocked machine to run the command and to retrieve your password. The Keychain method is more secure.

spacemacs

Finally, we need a way to read and write emails. This is where spacemacs comes in.

With a small layer defined in ~/.emacs.d/private/layers/spacemail/packages.el my .spacemacs config contains:

(defun dotspacemacs/layers ()
  ;; ...
  dotspacemacs-configuration-layer-path '("~/.emacs.d/private/layers/")
  dotspacemacs-configuration-layers '(;; ...
                                      spacemail
                                      ;; ...
                                      )

  ;; ...
  )

(defun dotspacemacs/user-config ()
  ;; ...
  ;; use msmtp as a sendmail replacement to send mails
  (setq message-send-mail-function 'message-send-mail-with-sendmail)
  (setq sendmail-program "/usr/local/bin/msmtp")
  ;; get the "From:" address from the envelope Emacs generates
  (setq message-sendmail-envelope-from 'header)
  ;; ...
  )

And that is it. In the package file above there are several shortcuts set up. Most importantly SPC a m i for inbox/unread mail and SPC a m n for new mail.

Once in a mail listing useful shortcuts are (evil mode):

h, j, k, l navigate
RET open mail on cursor
s search via filter, e.g. tag:build, cake AND NOT tag:me, hey there
/ search in buffer
t tag mail on cursor; use -<tag> or +<tag>
Z to switch to tree view
r respond to sender
R respond to all

And now all your vim bindings and spacemacs features will work when writing emails, which is much more nicer than some typical TextWidget.

For instance, you can format paragraphs quickly and wrap them at the proper width, including quoted text, which can just be selected with V and reformated with gq, while keeping the > prefix correctly at the beginning of each line.

Last but not least you can take org-mode links to emails! Inside an email just hit SPC a o l to store a link (this is not specific for the emails, it works in other files as well) and insert it in an org-mode document via , i l. This is very nice to make TODOs to respond to specific emails, or attach an important thread to a projet or task.

Ignore directories in projectile

2017-10-10T00:00:00+02:00

Problem

Putting a .projectile into a directory is enough to make it a project root in spacemacs, making it easy to search the entire directory and list its files.

Listing files in a project is easily done with SPC p f. There is a native mode and an alien mode, the former uses Emacs code to do the file listing, the latter uses external programs. The default is alien and usually that’s fine and much faster. It will use git inside a git repository, find when there’s just a .projectile.

See the projectile documentation for details.

However, if you have projects, like me, that have several subdirectories that contain git repositories, like so:

.
└── src
    ├── repo1
    │   └── .git
    ├── repo2
    │   └── .git
    └── repo3
        └── .git

then projectile cannot use git to list files, as it would, if the root were a git repository. It uses find and with that it also lists all .git directories. Adding “.git” to .projectile or projectile-globally-ignored-directories doesn’t help in this case. At least I couldn’t make it work.

Solution

So I ended up just adjusting the find command projectile uses.

(setq projectile-generic-command
  "find . -type f ! -ipath '.git*' ! -ipath '*/.git*' ! -ipath '*/build/*' -print0")

That also hides files in build subdirectories. And the file list becomes much more manageable.

Custom TODOs in org-mode

2017-10-08T00:00:00+02:00

Setup

It’s easy to add your own TODO keywords to your org-mode configuration, you can cycle through them with S-<left> and S-<right> or select them with t, if you use evil bindings.

(setq org-todo-keywords
      '((sequence "TODO(t!)" "WAIT(w!)" "|" "DONE(d)" "CANCELLED(c)")
        (sequence "GOAL(g)" "|" "ACCOMPLISHED(a)")
        (sequence "READ(r)" "|" "DONE(d)")
        (sequence "WATCH(W)" "|" "DONE(d)")
        ))

All stated before a vertical bar | are states that still require some action, all states after it indicate the item no longer requires action, i.e. they are “done” states.

The letters in parentheses are the shortcuts you can use to select them after pressing t, and the ! adds a log entry whenever that state is entered. For example, the task below was set to WAIT and a bit later it became a TODO again. Certainly not needed for every state change, but I like it for those two, because a task might go through these states several times.

* TODO Do something
- State "TODO"       from "WAIT"       [2017-10-08 Sun 13:53]
- State "WAIT"       from "TODO"       [2017-10-08 Sun 13:31]

Configure style

By default all active TODOs are displayed in the same style as TODO itself, all inactive ones are displayed in the style of DONE. If you want some visual clews to make it more obvious at a quick glance at your todo list, then you can configure each keyword to your heart’s content.

(setq org-todo-keyword-faces
      '(("WAIT" . org-warning)
        ("CANCELLED" . (:foreground "#657700" :weight bold))))

Here I simply used the org-warning face for WAIT, because it works well. For CANCELLED I wanted something very like DONE, but somewhat faded to set it apart from DONE tasks.

Define clock-stopping states

Changing to DONE (or any other closing state) while clocked into a TODO‘s heading will automatically clock out of that heading. However, some states might not close the TODO and still should result in a clock out. For instance, in my setup that’s the state WAIT. Luckily this sort of thing is supported as well, you can provide a list of states that end a clock:

(setq org-clock-out-when-done '("DONE" "WAIT"))

Avoid certain states as default

Using the appropriate shortcut (e.g. T in evil-org-mode) on a TODO headline will create another TODO headline below it, and it’ll default to the same TODO state the current headline has.

This isn’t desirable for states like WAIT, as it is an unlikely starting point for a new TODO. The following code shows how to control the keyword for that situation.

(defun pick-default-todo-keyword-instead-of-wait (new-mark old-mark)
  (if (and
        (eq old-mark nil)
        (string= new-mark "WAIT")) "TODO" nil))
(setq org-todo-get-default-hook '(pick-default-todo-keyword-instead-of-wait))

Solarized spacemacs

2017-10-08T00:00:00+02:00

Setup themes

Solarized by Ethan Schoonover is wonderful, and I use it almost everywhere, just like DejaVu fonts.

In spacemacs I set it up like this:

(defun dotspacemacs/user-init ()
  (setq-default
   dotspacemacs-themes '(solarized-dark
                         solarized-light)
   dotspacemacs-default-font '("DejaVu Sans Mono for Powerline"
                               :size 24
                               :weight normal
                               :width normal
                               :powerline-scale 1.6)
   solarized-use-variable-pitch nil
   solarized-scale-org-headlines nil
   solarized-height-minus-1 1
   solarized-height-plus-1 1
   solarized-height-plus-2 1
   solarized-height-plus-3 1
   solarized-height-plus-4 1
   ))

This keeps all headings for org-mode at the same font size, which guarantees that your fill-column marker and tags are properly aligned across headings of different levels. It also makes navigating org files more pleasant in general, as the buffer is truly monospaced again and your cursor doesn’t take any detours.

I suspect there are similar heading font size settings for other themes, but I haven’t investigated that.

Cycle themes

With the setting above you also get the light theme, which is useful for bright days, presentations, and presentations on bright days. SPC T n cycles through the two themes.

Note: if you use org-bullets, then you will notice that the hidden *s become visible upon switching the theme, because they still have the background color of the opposite theme. This can be remedied by saving the file and reloading it via SPC b R.

No indentation in org-mode

2017-10-07T00:00:00+02:00

Problem

By default org-mode indents the body after every heading based on its depth, this includes drawers and metadata like status changes.

* First heading
  CLOSED: [2017-10-07 Sat 13:22]
  :PROPERTIES:
  :CUSTOM_ID: foobar
  :END:
  Arbitrary text...

** Another heading
   Arbitrary text...

I started to dislike this, because for some reason it became a bit of a hassle to maintain proper indentation, especially with source/example/quote blocks, which also started at that indentation, but then it was inconsistent to drop the indentation for their content. I also struggled with finding a good indentation for bullet points: do they start with an extra space at that indentation level or not?

So I found org-indent-mode, which magically displays everything with indentation, without having the actual spaces in the file. That doesn’t work well for me, as it hides the actual structure of the file and things like the fill-column marker and tags are all over the place.

Solution

I gave up for it for a while, but recently found org-adapt-indentation, which simply turns off the indentation altogether, if set to nil. The only drawback is that drawers and metadata also isn’t indented. I kinda liked it for them, but I can live with that compromise.

(setq org-adapt-indentation nil)

This makes above example look like:

* First heading
CLOSED: [2017-10-07 Sat 13:22]
:PROPERTIES:
:CUSTOM_ID: foobar
:END:
Arbitrary text...
- bullet points
  - and another

** Another heading
Arbitrary text...

So that’s great. I start bullet points at column 0, without a leading space, and all is well.

Migrate existing files

The next problem was fixing my existing org files. The following script does just that:

#!/usr/bin/env python3
import sys

filename = sys.argv[1]

def get_indentation(line):
    i = 0
    while i < len(line) and line[i] == " ":
        i += 1

    return i

output = ""
indentation_to_remove = 0
first_line = False
for line in open(filename):
    line = line.rstrip()
    if line.startswith("*"):
        first_line = True
        indentation_to_remove = 0
        output += line + "\n"
        continue

    if not line:
        output += "\n"
        continue

    if first_line:
        first_line = False
        indentation_to_remove = get_indentation(line)

    if not line.startswith(" " * indentation_to_remove):
        indentation_to_remove = get_indentation(line)

    line = line[indentation_to_remove:]

    output += line + "\n"

open(filename, "w").write(output)

And I ran it as seen below. This replaces the files, so best back them up or better yet use git, which also allows you to inspect the diff.

find . -iname '*.org' -exec remove-org-indentation.py {} \;
find . -iname '*.org_archive' -exec remove-org-indentation.py {} \;
find templates -iname '*.txt' -exec remove-org-indentation.py {} \;