The Original Semantic Line Breaks

I found the advice from an Enlightened Unix Master about arranging input text for the the venerable troff typesetting system that seems to be ancestral to Semantic Line Breaks.

The following appears in Brian Kernighan’s UNIX for Beginners, which first appeared in 1974, and in the 1979 second edition of the same document.

Most documents go through several versions (always more than you expected) before they are finally finished. Accordingly, you should do whatever possible to make the job of changing them easy.

First, when you do the purely mechanical operations of typing, type so subsequent editing will be easy. Start each sentence on a new line. Make lines short, and break lines at natural places, such as after commas and semicolons, rather than randomly. Since most people change documents by rewriting phrases and adding, deleting and rearranging sentences, these precautions simplify any editing you have to do later.

Kernighan definitely had editing with the ed text editor in mind when he wrote this. The above advice is in Section III, Document Preparation, Section II of UNIX for Beginners has material on using ed, and ed is definitely “line oriented”. The unit of text that ed addresses is the line, by number or by matching contents. Kernighan is also writing about troff or nroff input. That input has formatting directives or commands mixed in with the text to be formatted, but line breaks do not affect rendered output.

The Semantic Link Breaks web page summarizes its concept as:

When writing text with a compatible markup language, add a line break after each substantial unit of thought.

That’s certainly what Kernighan advises.

Bonus Kernighan script

UNIX for Beginners, Second Edition has a little shell script that shows how to do a cheap spell check in 1979 (and maybe 1974):

cat ...     # collect the files
| tr ...    # put each word on a new line
| tr ...    # delete punctuation, etc.
| sort      # into dictionary order
| uniq      # discard duplicates
| comm      # print words in text but not in dictionary

This script has some similarities to Douglas McIlroy’s famous “find the N most common words in a text file” script:

tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -k1.1nr |
sed ${1}q

The Programming Pearls column with the “N most common words” script has the date June, 1986, while the spell checking script is probably from 1974.

If you were to naively put arguments on Kernighan’s spell checking script, it might look like this:

cat $* |
    tr -cs A-Za-z '\n' |
    tr A-Z a-z |
    sort |
    uniq |
    comm -2 -3 - /usr/share/dict/words

Here’s the thing: the comm command sounds like it does what you want, show you lines from stdin (the sorted, unique, non-punctuated words from the input files) and words, the official system dictionary.

But it won’t. You’ll get a lot of unhelpful error messages like this:

comm: file 2 is not in sorted order
comm: input is not in sorted order

I assume that script worked in 1974 or 1979, but it doesn’t in the 21st century. We have Locales now, and we’re making collation into everyone’s problem.

Both the sorted, unique, non-punctuated words from the file you’re interested in, and the dictionary file have to be sorted in dictionary order according to the locale’s collation. A modern working script would need to look something like:

export LC_COLLATE=en_US.UTF-8

sort -d /usr/share/dict/words > word.utf8_collate

cat $* |
    tr -cs A-Za-z '\n' |
    tr A-Z a-z |
    sort -d    |
    uniq       |
    comm -2 -3 - word.utf8_collate