Some TeX Developments

The regular arrival of TUGBoat is always welcome, and a chance for an interesting read. Everyone will have their favourite entry, but I thought I’d pick out one or two highlights.

Hans Hagen writes about creating e-books using ConTeXt. Now, there is a lot of very good information in the article about the problems at hand. However, what makes it a highlight for me is the take he has on the e-book concept. There are a lot of very good points about the limits of an electronic book: for example, drop a book and it will be fine, drop an e-book …

Another highlight for me is not an original article but a reprint. Now reprints on TeX topics are not unusual, but in this case they’ve looked a bit wider and included one from Print, a magazine dedicated to visual design. Looking at books as design objects, rather than as something to code, is a refreshing change (at least for me)!

The third highlight for me is not a single article, but a set of book reviews. Boris Veytsman has taken on the task of co-ordinating (and often writing) reviews, and the current TUGBoat includes four. Hopefully this will be a regular feature, and help keep us all up to date with what’s going on in print for both TeX users and the wider world.

I recently stumbled across the fact that UIT have published English translations of several of Herbert Voß’s (La)TeX books. Herbert is a prolific author of TeX support material, perhaps most notable the Math Mode guide. He’s also the current maintainer of PStricks. As you’d therefore expect, the titles available include Typesetting Mathematics with LaTeX and PS Tricks, along with a LaTeX Quick Reference and Typesetting Tables with LaTeX. That’s right: there is a whole book just for setting tables correctly! The translations all seem to have appeared over the last year or so, and I think would be great additions to the bookshelf.

There are lots of very clever ideas in LuaTeX, and it’s easy to miss some of the good stuff there is. One that many people might miss is category code tables. As any TeX programmer rapidly becomes aware, category codes are central to TeX, and the construction

\catcode`\<char> = <number>\relax

is one you soon get used to. The problem comes when several people start altering the codes: there is no easy way to get back to a known position.

A good illustration of this is verbatim material. The way that something like LaTeX’s \verb macro works is by setting the category code for all of the ‘special’ characters to ‘other’:

\let\do\@makeother
\dospecials

where both \@makeother and \dospecials are provided by the LaTeX kernel. That works because \dospecials is defined as

\do \ \do \\\do \{\do \}\do \$\do \&\do \#\do \^\do \_\do \%\do \~

and so maps the function \@makeother to all of the ‘special’ characters. Using that, a (simplified) verbatim command looks like

\makeatletter
\newcommand*\stdverb{%
  \begingroup
    \let\do\@makeother
    \dospecials
    \@stdverb
}
\newcommand*\@stdverb[1]{%
  \catcode`#1=\active
  \lccode`\~=`#1%
  \lowercase{\let~}\endgroup
  \ttfamily
}
\makeatother

In most cases, this works fine. However, if someone makes another character ‘special’ then things go wrong:

\catcode`\+=\active
\newcommand+{oops}
\stdverb=#{+=

Of course, you could loop over every character, which would be slow for 8-bit input but with UTF-8 input that becomes impractical. Of course, you could add each active character to \dospecials, but this is dependent on everyone sticking to good practice.

This is where category code table come in. These are pre-set lists of category codes, which can be applied in one go. Heiko Oberdiek’s luatex package provides a LaTeX interface for these, meaning we can do:

\usepackage{luatex}
\makeatletter
\newcommand*\luaverb{%
  \begingroup
    \BeginCatcodeRegime\CatcodeTableOther
    \@stdverb
}
\makeatother

(reusing the same internal macro as before). now trying

\catcode`\+=\active
\newcommand+{oops}
\luaverb=#{+=

works as expected, as all of the category codes change in one go. Simple, clear and effective!

Today sees the release of TeX Live 2011. As most readers will know, TeX Live is a cross-platform TeX system, and is the basis of the MacTeX distribution for Mac OS X. TeX Live 2011 is very much evolutionary, with only a small list of changes from earlier releases. Perhaps the most notable is that the Biber program is now available for major platforms: it was added to the on-line updates for TeX Live 2010, but this will be the first appearance on the DVD version. I guess that Karl Berrry will now be busy catching up on CTAN updates, so that on-line updating of TeX Live can resume.

This morning I had a very worrying set of e-mails from Google about the security of the blog. I’m working on sorting it out at the moment: you’ll see the site alter pretty quickly as I get the hang or restoring SQL data. The posts themselves and comments should all be sorted, but I still need to fix the categories and so forth.

Spaces in file names are a constant issue for LaTeX users. As many people will know, TeX is not really very happy with spaces, as they are used to delimit the end of input in a lot of low-level macros. This shows up particularly in two areas: graphics and shell escape. For graphics, the excellent grffile package will deal with many of the issues. When using shell escape, the issue is usually that \jobname may be slightly odd. For TeX Live users, that is not so much of a problem as the name is automatically quoted to protect the space. However, MiKTeX does things a bit differently, and uses a star in place of a space. So you end up with

\edef\example{\jobname}
\show\example
> \example=macro:
->test*file.
l.2 \show\example

which is not exactly helpful. However, it is possible to deal with this, as recently mentioned on TeX.SX. As * cannot normally appear in file names, and \jobname makes all characters have category code 12, a simple approach is to do a quick replacement

\edef\Jobname{\jobname}
\catcode`\*=\active
\def*{ }
\edef\Jobname{"\scantokens\expandafter{\Jobname\noexpand}"}
\catcode`\*=12 %
\show\Jobname

If you want to deal with both TeX Live and MiKTeX, you’d of course need first to test which system is in use (for example using \pdftexbanner).

The TeX Q&A site is attractive to many people not only because of the way it works as a website, but also because the site makes it’s data available for reuse. One thing that many people want to do with useful information is to print it. Now, you can simply print from your browser, but the layout won’t necessarily be ideal. Enter the data back-end, and  TeXprinter. TeXprinter takes the raw information from a question and formats it as a PDF or LaTeX file. That way, you can print something that is actually readable! All of this is done in Java, so there is nothing to install beyond the program itself: great stuff.

There was recent question on TeX.SX about expansion, where I suggested using \romannumeral for full expansion. This idea comes up quite often, so I thought it would be useful to look at how it works.

The \romannumeral primitive was intended to turn integers into roman numerals. As such, the input to \romannumeral should be an integer, in the same way that the input to \number is an integer. However, while \number produces output for both positive and negative integer input, \romannumeral produces no output at all for valid negative integer input. That is pretty clear with a simple case such as

\romannumeral -15

but becomes more powerful when used along with the ` syntax which TeX allows for including an integer

\romannumber -`0

Here, the `0 is converted into a integer which TeX treats as complete: `0 is 48, but `01 is the (terminated) integer 48 followed by a separate 1 and not the integer 481. The important thing for expansion, however, is that TeX always looks for an optional space to gobble after an integer, even in a case like `0 where the integer is automatically terminated.

How does this help with expansion? It’s all to do with how TeX terminate numbers. If we have the demonstration macro

\def\demo#1{%
  \detokenize\expandafter{\romannumeral-`0#1}%
}

the \romannumeral will produce no output (as the number it finds is -48). However, it will expand #1 until it finds an unexpandable token. That means that something like

\def\testa{\testb}
\def\testb{\testc}
\def\testc{abc}
\demo{\testa}

will be expanded to abc without needing to know in advance how many expansions are needed. In a real case, this is a great way to fully-expand \if... tests and so on, leaving only the result, but without needing to know how many expansions are needed (and having long \expandafter runs).

What you should notice here is that TeX will reinsert the expanded result, and will not complain if something non-expandable appears

\def\testa{\relax}
\demo{\testa}

will print \relax, as this is inserted with no errors or loss of unexpandable tokens.

The only proviso here is that TeX stops at the first non-expandable item. So something like

\def\testa{ \testb}
\def\testb{123}
\demo{\testa}

will stop at the space in \testa and not expand \testb. For real applications, that is not usually an issue as we are usually aiming to expand the beginning of the input.

My siunitx package continues to attract feature requests, even though for me and many people it is essentially ‘feature complete’. Many of these requests are as a result quite complex, and probably somewhat esoteric, but I do aim to be as accommodating as I can. At the same time, I want to avoid breaking things for the majority. So my approach is to take a few issues at a time and to work on them for each 2.x release, plus of course the regular bug fixes. At the same time, I’m keen to review my own work and to tighten up on some parts of the code. That’s particularly important in adding new features, as there are places where it turns out a bit more structure is needed.

For version 2.3, I’m focussing on the mechanics of tabular alignment. This entire part (around 1500 lines of code) is going to be rewritten, hopefully making things more reliable, improving performance and also ease of maintenance. For version 2.4, I’ll probably look again at how tablular material is collected up by siunitx,  and also at the parser for numbers: I’ve had some ideas to improve performance. Beyond that, I have a few general thoughts, for example the idea of ‘multiple error‘ parsing: that looks tricky!

Anther area to work on is the option interface. Some of these are not perfect, and v2.3 will see some revisions. I have further revisions already in mind, but don’t want to do too much at once so again have some ideas for v2.4. At the same time, I think there are some performance enhancements available by recoding parts of the option system. I’ll be doing that as part of the general review, and so again we should see evolution in that area. I’ll return to this general issue in another post.

That is quite a list, and so I’ll certainly be kept busy. I have other things to do as well, both in the TeX world and outside, so at the moment my thinking is v2.3 in July, with v2.4 in the autumn (perhaps October or November) and v2.5 vaguely pencilled in for ‘first half of 2012’. Hopefully that will keep everyone happy.

Now that TeX Live 2010 is frozen, work has started on test builds for TeX Live 2011.  This is really part of a continuing evolution, and so there are (so far) no big surprises for me. Everything installed smoothly on my Mac, and I’ll be updating my PC tomorrow if I have a chance. I do hope other people are getting on with downloading the test code!

(For those people who, like me, are TeXworks fans, I notice that TeXworks v0.4.1 is included in the Mac version of TeX Live 2011 and so presumably in the Windows version too. Great work by everyone who helped getting it to build successfully on the Mac, and of course everywhere else!)