expand_pointing: fit a marked-up psalm or canticle to a tone

This program, expand_pointing, takes a plain text file with some markup, called `pointing', and fits the text to a psalm tone. The pointing is intended to be close to what is conventionally used in printed texts of pointed psalms. The text is usually that of a psalm or canticle.

The output is a Lilypond input file, and is intended to be processed by Lilypond for a score.

Prerequisites

This program has only been tested on a Linux system, although it is quite likely that since it is written in Python it would work on any system with Python installed. But you'd need to copy the files by hand to sensible locations -- the install target of the Makefile is only for *nix systems.

To use the program you need Python 2.4 or later (only tested with 2.4 and 2.5; not python 3.x), and python-ply, the parser/lexer package.

Although Lilypond is not needed to run the package, the output is of little use without Lilypond. The Lilypond files generated are intended for Lilypond 2.8.7.

Installation

This should be as usual. Unpack the .tar.gz file, change to the directory thus created (expand_pointing-VERSION), and run make install as root. There is no configure script, since the program is just straightforward Python.

Usage

Before running the program, you need to create a text file of input. The format of these is described below.

For simple use, run

      expand_pointing text_file > ly_file
    

Other options are:

• -V: just display the version number.
• -h: just output the usage.
• -i ily_file: specify the Lilypond header file for inclusion. This must be supplied when using -T. It must also be supplied when using the default template_psalm.ly file. (You can always name an empty file if you do not need it.)
• -H: hyphenation_dictionary_file: specify the hyphenation dictionary; default is the hyphen_dict.txt file supplied. The default contains only the words found in the Coverdale psalter as in the 1662 BCP.
• -p psalm_template_file: the template for the output Lilypond file; default is the template_psalm.ly file supplied.
• -T: if this is given then no pointing is done, but the tonale is generated as a Lilypond file sent to standard output.
• -x tonale_file: this allows the tonale to be specified in a text file which is read by the program. It is described below; see the example sarum_tonale.txt for an example, with the separate gloria*.txt files.
• -o output-format: at present the only fully working output-format is lilypond, which is the default; gregorio may also be used, but this is experimental.

The textual input files

These normally contain a few header lines, followed by the actual pointed text. The header lines all start with #.

The pointed text has one line for each verse, optionally preceded by a verse number terminated by a `.'. The text is used, with the markup described below.

A very small example

Here is a very short, one-verse, example of what might be in an input file:

      #tone I2
      #psalm 107
      #incipit "Confitemini Domino"
      27. They reel to and fro, and stagger like a drunk-_en man: and are _at their wits' end.
    

The underscores in the verse marks the start of the mediation or ending.

The heading items used in the text form of the psalms

In addition to the text of the psalm itself, the following may be used:

• #comment Starts a comment line, which is ignored.
• #psalmThe number of the psalm, used for the heading.
• #toneThe tone and ending, as VIII2 for the 2nd ending of tone VIII. Tonus peregrinus should be specified as `tp'.
• #incipitThe Latin incipit, in quotation marks ("").
• #bcpdayThe day when the psalm is to be used according to the 1662 BCP. Not currently output.
• #intonation_on_noneIf this is specified, no verse will start with the intonation, not even the first.
• #intonation_on_allIf this is specified, every verse will start with the intonation.
• #reintonationThe `reintonation' is to be used. Only for tonus peregrinus. (This is my term for the first section of the 2nd half-verse.)

The markup used in the text form of the psalms

= (equals) indicates a hard hyphen, one that should always be printed no matter how the text is printed. Space after, no space before.

- (hyphen/minus) indicates a soft hyphen, one that is used to split syllables on different notes, or because of a line break. No space before or after.

~ (tilde) also indicates a soft hyphen; it is a useful alternative to - to mark hyphens added compared to the original source. For example, a source may not completely hyphenate a word that contains the start of the ending; such a word should be hyphenated manually using ~ to augment the original. No space before or after.

^ (circumflex) indicates (syllables of) two successive words which use up one note of the normal tone, but in this case the note is sung twice. This is a tie in the original. No space before or after.

-^ (hyphen circumflex) Like ^, but the two syllables are parts of the same word. Souces may not distinguish between this and ^, using a tie for both, but this program requires the distinction. No space before or after.

() (open and close parentheses) marks a non-syllable, to indicate that a note normally used is not used in this verse. Sometimes sources use an em-dash for this; this program does not support an em-dash, because it is too easy to confuse it with the different types of hyphen. Space as if () were the letters of a syllable; in the middle of a hyphenated word, ()- may be used. Note that in some cases this must be used more times than the source; for example, if two notes of the ending are omitted, this should be marked as _() () in the text.

.. (two full stops) is used as a pseudo-syllable, indicating that the previous syllable should be sung to two notes rather than the usual one, the note naturally its own and the note following. This is normally indicated by a diaresis on the (previous) syllable. Space as if .. were the letters of a syllable; in the middle of a hyphenated word, ..- may be used. If the tone leaves the reciting note in the middle of a syllable -- that is, the syllable has two notes, one the reciting note and the other the first note of the ending -- use _.. for this.

CAPITALS are used for the word(s) set to the intonation. But this does not affect the fitting of words to tone.

* (asterisk) marks the flex, a pause within a long half-verse. Space before but not after.

` (backquote) for tonus peregrinus only, in the 2nd half-verse, marks the use of the reintonation reciting note, for one syllable, after we've switched to the main reciting note. This may not work yet.

_ (underscore), before a syllable, marks the first syllable of the mediation or ending.

| (vertical line), before a syllable, marks the last syllable of the re-intonation (used only for the 2nd half-verse in tonus peregrinus).

The tonale

The program includes the tonale specified using the -x option: after this, give the name of a text file specifying the tonale. Again, the Sarum tonale is supplied as an example. This is described below.

The form of the tonale definition file

A particular tone is contained within tone -- endtone. Items are each preceded by a keyword. The items making up a tone are:

1. tone: The tone label itself, a string. This is used in concatenation with the ending label to identify the tone-ending combination used in the psalm .txt file.
2. name: Its description. Not currently used.
3. clef: The clef, c or f followed by the line on which it appears, 1--4 bottom to top.
4. final: the final of the mode, as a note letter.
5. notes: The notes of the tone itself, without the ending. The octave of the first note is chosen so that it is no more than a fourth from the absolute pitch specified by the middle space. Single notes may be written just as a letter. Notes are written as letters a--h, German fashion; so b for b-flat and h for b-natural. A / may follow a note intended to take a stressed syllable, and ? may follow a note optionally inserted where the words require extra notes; these are only output in the tonale produced by -T.

   Ligatures are written as a series of notes enclosed in brackets or parentheses. The program generates the correct ligature automatically; fortunately the psalm tones do not have any ambiguous cases. For example, if you write g [f e] then if these are used for two syllables then the [f e] will be converted to a clivis, but if a tie combines all three notes, then they will be converted to a climacus.

   The notes are divided by : and | into sections. The intonation is before the first |, then the reciting note, then after another | the mediation, then a : for the end-of-half-verse. After the : and before another | comes the reintonation (or second intonation), only for tonus peregrinus.

6. ending: This introduces one ending, as described separately.

For the ending the items are:

1. ending: The ending label itself, a string. This is used in concatenation with the tone label to identify the tone-ending combination used in the psalm .txt file.
2. name: Its description. Not currently used.
3. notes: As for the tone, but just a sequence of notes.
4. gloria: The name of a file containing the gloria, written as a `psalm' of just two lines.

A # introduces a comment which is ignored.

Outstanding bugs

No attempt has been made to internationalize the program.

No divisio (perhaps a half-bar) is output at a semi-colon.

There is no support for the flex.

backquote may not yet be working properly.

The template ps has a hard-coded include mop.ily

The source location (line number and position in line) is not always correct for hyphenation errors and warnings.

It would be good if the backend were more separated, so that the Lilypond-related and Gregorio-related backends became plugins.

The Latin hyphenation dictionary is just a stub. Probably Latin (unlike English) can be hyphenated by rule, but this is not implemented.

Tonale emacs mode

A very simple emacs mode, tonale-mode, is provided to give syntax highlighting of the tonale text file. This is not yet installed -- you could install it manually in your site-lisp directory.

The remainder of the page is of little interest to anybody who just wants to use the program.

The program itself

This was my first essay in using Python; indeed, I used Python partly to teach myself the language. So don't be surprised, if you are proficient with Python, when you see clumsy constructions.

Programming: lexer/parser vs direct string manipulation

I originally wrote the program without a lexer and parser -- it seemed to me to be on the borderline between requiring simple handling, and full lexing/parsing.

The numbers of lines of code increased by about 20% overall when I changed to using the parser. The main program decreased a little in size, about 12%, but not enough to compensate for the 400 or so lines of the lexer/parser. The total number of nonblank lines without a parser was 1142, with 1382.

Having a parser made it easier to tweak details of input markup, and easier to extend if I ever think of doing so.

However, it became harder to have certain sorts of input e.g. anything which needs lexer states, like strings terminated by end-of-line (partly because python-ply's support for lexer states was incomplete). It was harder to do error reporting; syntax errors in particular are poorly handled, and it was harder to keep track of the source location. There is a messy relationship between the AST and the rest of main data structures (inheritance wasn't quite right, and doesn't match the obvious division into python modules; I used containment, but that introduced rather artificial differences between what was held in the AST structures, and what wasn't).

Python

Since this was my first essay in Python, it prompted a few remarks on Python in general, as used for such a program.

• The object-orientation is much nicer than perl5 -- it's properly part of the language.
• Overall, I thought the language design was a good balance between conciseness and verbosity. I found it easier than I'd expected, when coming back after a break, to understand the code.
• The string-handling functions seem ad-hoc. It is sometimes nonobvious which is self: e.g. split on string has string as self, the split-places as an argument; split on regexp has regexp as self, string as arg.
• regexps clumsier than perl's, especially picking out the group values. I suspect there are tricks I haven't learned.
• Lines tend to become long, because you can't just break them anywhere. I used extra parentheses around expressions just to be able to break lines -- it made the code look like lisp.
• The visibility control (cf C++ private) didn't seem adequate, relying too much on the use of the _ convention. It was much too easy to pick up private data accidentally. (I have yet to try the Bastion module.)
• I didn't like the equivalent to ?: i.e. cond and trueval or falseval. I still prefer Algol 60's syntax.
• Print statements were still my favourite tool for debug. Perhaps I should have used the logging module.