Some TeX Developments

Following on from my last post, I thought it might be useful to give some simple example of how the xparse package works and why it’s useful. I want to do this to show end users of LaTeX how it can replace \newcommand, so the example will not involve anything too complex, code-wise.

First, why would you want to use xparse’s \NewDocumentCommand in place of \newcommand? First, \NewDocumentCommand can make macros that take a mixture of arguments that \newcommand cannot. With \newcommand, you can make a macro that takes a number of mandatory arguments, or ones where the first argument is option and in square brackets, but that is it. Anything else then needs the use of TeX programming or internal LaTeX macros: not really helpful for end users. The second thing is that \newcommand macros are not ‘robust’. This shows up where you need to \protect things, which can be very confusing. Macros created with \NewDocumentCommand are robust, and this means that they work more reliably.

I’m going to illustrate moving from \newcommand to \NewDocumentCommand with a series of simple examples. For all of them, you need to load the xparse package:

\usepackage{xparse}

Macros with no arguments

The simplest type of macro is one with no arguments at all. This isn’t going to show off xparse very much but is is a starting point. The traditional method to do this is

\newcommand\NoArgs{Text to insert}

which becomes

\NewDocumentCommand\NoArgs{}{Text to insert}

That does not look too bad, I hope. Notice that I’ve got an empty argument in the xparse case: this is where the arguments are listed, and with \NewDocumentCommand there always has to be a list of arguments, even if it is empty. That’s a contrast with the \newcommand approach, where we only need to mention arguments when there are any.

One or more mandatory arguments

The most common type of argument for a macro is a mandatory one. With \newcommand, we’d give a number of arguments to use:

\newcommand\OneArg[1]{Text using #1}
\newcommand\TwoArgs[2]{Text using #1 and #2}

\NewDocumentCommand is a bit different. Since it can work with different types of argument, each one is give separately as a letter. A mandatory argument is ‘m’, so we’d need

\NewDocumentCommand\OneArg{m}{Text using #1}
\NewDocumentCommand\TwoArgs{mm}{Text using #1 and #2}

This is still pretty similar to \newcommand: the useful stuff starts when life gets a little more complicated.

One of more optional (square brackets) arguments

To really get something clever out of xparse, the arguments need to be a little more varied than I’ve show so far. Let’s look at optional arguments, which LaTeX puts in square brackets. If I want the first argument to be optional, then LaTeX can help me

\newcomand\OneOptOfTwo[2][]{Text with #2 and perhaps #1}
\newcomand\OneOptOfThree[3][]{Text with #2, #3 and perhaps #1}

If I want anything else, I’m on my own (so no more \newcommand examples!). First, let’s do the two example using xparse. An optional argument in square brackets, which works like a \newcommand one, is ‘O’ followed by {}:

\NewDocumentCommand\OneOptOfTwo{O{}m}%
  {Text with #2 and perhaps #1}
\NewDocumentCommand\OneOptOfTwo{O{}mm}%
{Text with #2, #3 and perhaps #1}

How about two optional arguments? It’s pretty obvious:

\NewDocumentCommand\TwoOptOfThree{O{}O{}m}%
  {Text with #3 and perhaps #1 and #2}

What if we want something as a default value for the optional argument? With \newcommand, that would be

\newcommand\OneOptWithDefault[2][default]%
  {Text using #1 (could be the default) and #2}

which would become

\NewDocumentCommand\OneOptWithDefault{O{default}m}%
  {Text using #1 (could be the default) and #2}

The same idea applies to each optional argument: whatever is in the braces after the O is the default value.

More complex optional arguments

You might be wondering why we need the ‘{}c after ‘O’ when there is no default value: why not just ‘o’? Well, there is ‘o’ as well. Unlike \newcommand, \NewDocumentCommand can tell the difference between an option argument that is not given and one that is empty. To do that, it provides a test to see if the argument is empty:

\NewDocumentCommand\OneOptOfTwoWithTest{om}{%
  \IfNoValueTF{#1}
    {Do stuff with #2 only}
    {Do stuff with #1 and #2}%
}

Don’t worry if you forget to do the test: the special marker that is used here will simply print ‘-NoValue-’ as a reminder!

Two types of optional argument

Sometimes you might want two different optional arguments, and be able to tell which is which. This can be done by using something other than square brackets, often using angle brackets (‘<’ and ‘>’). We can do that using the letter ‘d’ (or ‘D’ if we give a default).

\NewDocumentCommand\TwoTypesOfOpt{D<>{}O{}m}%
  {Text using #1, #2 and #3}

What input syntax does this make? Let’s look at some examples

\TwoTypesOfOpt{text}             % One mandatory
\TwoTypesOfOpt[text]{text}       % A normal optional
\TwoTypesOfOpt<text>{text}       % A special optional
\TwoTypesOfOpt<text>[text]{text} % Both optionals

How did that work? The first two characters after the ‘D’ are used to find the optional argument, so in this case ‘<’ and ‘>’.

Finding stars or other special markers

Another common idea in LaTeX is to use a star to indicate some special version of a macro. Creating those with \newcommand is difficult, but it is easy with \NewDocumentCommand

\NewDocumentCommand\StarThenArg{sm}{%
  \IfBooleanTF#1
    {Use #2 with a star}
    {Use #2 without a star}%
}

Here, ‘s’ represents a star argument. You’ll see that it ends up as #1, while the mandatory argument is #2. You’ll also see that there needs to be a test to see if there is a star (\IfBooleanTF). This doesn’t mention stars as the test can be used for other things.

Summing up

There is more to xparse than I’ve mentioned here, but I hope that this is a useful flavour of what it can be used for. To get more flexibility there is a bit more to think about compared to \newcommand, but the overall consistency is hopefully worth it.

I’ve had a few chances in recent weeks to promote the xparse package. Regular readers will know that xparse is part of the efforts of the LaTeX3 Project, and is meant to provide a replacement for \newcommand, etc., for creating document macros. The promotions have come up where there are things that are hard to do with \newcommand but easy with xparse. One example was creating an optional argument for a macro but giving it in normal braces. With xparse, this is pretty straight forward

\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand\en{g}{%
  \IfNoValueTF{#1}{\epsilon}{\epsilon_{#1}}%
}
\begin{document}
\( \en \) and \( \en{stuff} \).
\end{document}

This is clear, and does not need any knowledge of the internals of the LaTeX3 approach. So I’m going to keep promoting xparse for day to day LaTeX users, even if they aren’t using much code. Hopefully this will make life easier for them, and for me, and will be a real benefit now from the LaTeX3 work.

My previous post mentioned some efforts to improve the performance of the siunitx parser. I’ve now committed an entirely new version of the parsing code to the repository. I’ve also done my best to speed up the rest of the package. The speed you see very much depends on the type of input involved. With my test file, I got a time of roughly 3 minutes using version 1 of siunitx, about 2 minutes before improving version 2, and about 1.5 minutes after. For comparison, doing no processing at all take the time down below 10 seconds for the same file (roughly 700 pages of repeated input). For the interested reader, an siunitx snapshot TDS-style zip is available.

One thing that I find interesting in all of this is that even before optimising the code, version 2 still worked faster than version 1, even though it does more things. A lot of that is because the pre-built looping material in expl3 does a much better job than my own attempts in version 1 of siunitx. Of course, good programmers will always use fast loops, but for the rest of us I think this shows how a sensible tool-kit can bring benefits “behind the scenes”.

It’s been a while since I gave an update on progress with siunitx version 2. I’ve had other things on the go, but am trying to make some progress with the code as I have lots of bugs which are fixed in the new version. Since the last snapshot, I’ve gone back over everything and moved internally to expl3 syntax. That makes my life a lot easier, as there are lots of tools available without needing to write them all for myself. It does mean that users/testers will need to have expl3 installed: you will need the most recent release, which should appear in TeX Live and MiKTeX soon for online installation, or can be downloaded from CTAN (look for expl3.tds.zip).

There are some notes and outstanding questions with the current snapshot: it is still some way from being finished. Things to be aware of or to think about:

• Places where units should be repeated are not working properly at the moment. That is the next thing I need to sort out: I’d hope it will be done soon (perhaps another snapshot in a couple of weeks)
• I’d set the defaults to use text mode for units and maths mode for numbers. This may not stay that way, see for example the results for \SI{1.23}{J.mol^{-1}.K^{-1}}. This is because printing units literally has to do the powers in the same font as the units.
• The options system is now using the LaTeX3 l3keys module. Following the pattern that looks likely for LaTeX3 work, I’ve given the options names divided into words using hyphens. I hope that they make sense.
• At present, complex numbers are always printed with the complex root (i) after the number. I need an option to reverse this, but can’t think of a good name.
• I’ve tried to simplify the various ‘trapambig…’ options into a single one (use-brackets). I’m not sure if this will work for everyone: do people need the ability to turn bracketing on and off in a more granulated way?

For those who want to test things out, you can get:

• The source (dtx file)
• The documentation (pdf file)
• The extracted package (sty file)
• A ready-to-install (tds) zip file

The next step for LaTeX3 development is to revise the two existing “xpackages” which are available to make a link between the code level and the user: xparse and template. Of the two, the xparse package is by far the easier to understand.

In LaTeX2e, you can either use the LaTeX method to create new commands:

\newcommand*\mycommand[2][]{%
  code goes here, using #1 and #2
}

or you can do things yourself using a mixture of TeX primitives and LaTeX internal functions (and ignoring the issue of default value):

\def\mycommand{%
  \@ifnextchar[{%
    \@mycommand
  }{%
    \@mycommand[]%
  }%
}
\def\@mycommand[#1]#2{%
  code goes here, using #1 and #2
}

This does not make for code which is easy to alter to reflect different input syntax (for example XML), or to change internal functions without knowing how the user input works. The idea of xparse is to separate out the user syntax from the internal code. Currently, exactly what tools need to be available is still being decided. The current version of xparse would expect the following syntax:

\DeclareDocumentCommand \mycommand { o m } {
  code goes here, using #1 and #2
}

The idea is that each argument is represented by a letter, for example o for an optional argument, m for a mandatory one or s for an optional star. This makes it possible it see immediately from the definition how many arguments are needed (one for each letter). It also means that the internal functions, which implement things, can be separated totally from the user part of the system. In that way, internal functions can have a fixed number of arguments, and leave xparse to supply them. So if the internal function needs to be changed, it does not matter how it is used, or vice versa.

That all sounds very good, but there are some outstanding issues. For example, handling verbatim arguments is not straight-forward. There are a couple of possible approaches to this. Either stick to a simple system, and accept that not everything can be done in the same way, or make the system more flexible but complicated. I’m currently in favour of the first approach: almost every user function is simple (especially as we have the ε-TeX extensions and so can \scantokens our way out of a lot of problems). I’d be interested to hear what other people think would be useful.

The LaTeX3 Project (of which I’m a member) have made a snapshot release of the current LaTeX3 code to CTAN today. This is in two parts:

• expl3, the base layer for LaTeX3, which provides a consistent programming interface for LaTeX3 on top of TeX (or rather, on top of pdfTeX, XeTeX or LuaTeX).
• xpackages, higher level interface ideas for LaTeX3.

expl3 is now reasonably stable: the code has been carefully reviewed and the Team are happy that what is there works well. There are gaps, and some of these are definitely on the “to do” list. On the other hand, the xpackages bundle is not so close to being finished. At the moment, it includes only the parts of the experimental code that look the closest to reaching some kind of conclusion. However, it is also the part that is actively being reviewed at the moment. So I expect changes!

I’m hoping that as expl3 gets distributed around the TeX world it will be possible to consider using it for coding new packages. The ideas in expl3 make coding much easier, over all, even if initially there is a lot to learn. Of course, delivering more of LaTeX3 will help in getting people interested as well. That is something I’m definitely keen on.

There is currently a very active thread on comp.text.tex about LuaTeX. Several related questions have come up, and it seems that some kind of summary would help me, at least.

Why did the LuaTeX team choose Lua?

Possibly the most frequently asked question with LuaTeX is why the team behind it chose Lua, as opposed to one of the other scripting languages available. The LuaTeX FAQ addresses this as question 1, so it’s obviously important. As I understand it, they looked at a number of possible languages for extending TeX, and decided that many of the “obvious” candidates (Java, Perl, Python, Ruby, Scheme) were not suitable. Not everyone is going to agree with that assessment, and some people are bound to feel that there was not sufficient consultation.

With LuaTeX already at version 0.40, it is progressing and will deliver. So unless a team can be got together to provided an alternative scripting system, it looks like Lua is what we will get. I don’t see there being enough experienced programmers with time on their hands to deliver a complete alternative at the moment.

How does LuaTeX relate to ConTeXt and LaTeX?

LuaTeX is an “engine”, like pdfTeX or XeTeX. So it provides certain functions, which might or might not get used. The end-user can use them directly, but this is really not helping most people as they don’t want to do this type of low-level work. So in the main it is down to either the TeX format (ConTeXt, LaTeX, …) or an add on (LaTeX package, ConTeXt module, etc.) to take advantage of them.

In the ConTeXt case, the entire format is being re-written as “Mark IV”, which will use a lot of Lua. This of course means things fundamentally change compared to early versions of ConTeXt, and ties it to a single engine.

In the LaTeX case, the format is written to work on plain TeX, no extensions at all. That is not about to change: essentially, LaTeX2e will never be “updated” in that sense. The current LaTeX3 plans don’t envisage requiring LuaTeX, although I’d hope that some low-level support will be included if LaTeX3 ever becomes a reality. There will, though, be LaTeX packages that use Lua: I’m sure that at some point there will be a fontspec-like package for LuaTeX, for example.

What is wrong with LaTeX2ε?

One issue that always comes up when future directions for TeX are discussed is whether LaTeX2ε needs to change, or whether it is okay as it is. Currently, if you know what you are doing then you can achieve a lot with LaTeX. There are a vast range of packages, and these cover very many of the things you could ever hope to do in TeX. So in a sense LaTeX works as it is. On the other hand, you have to know which packages to load, even to get some of the basics right (try making a new float type without loading any packages and using one of the base classes!). At the same time, things like UTF-8 text, system fonts and so on are not available using only the kernel.

A more fundamental issue is that LaTeX currently doesn’t do enough to provide structured data, and is not a good choice for dealing with XML input. This makes it hard to get data in and out, and is closely related to the fact that there is not enough separation of appearance and structure in the kernel and in add-on packages.

Neither of these points has escaped the notice of the LaTeX team. The question is whether a successor to LaTeX2ε will appear, and if it does whether it can succeed. There is a need to start from scratch with many things, meaning that a new LaTeX simply won’t work with most packages currently available. So the team have to deliver something that really works.

How can (La)TeX recruit new users?

One question that comes up here is what is a typical (La)TeX user anyway. Everyone has their own view on this: some people see TeX users mainly as mathematicians and physicists, other people point out that particularly with XeTeX available there is a lot of potential in the humanities.

Making life easier for new users is a priority for gaining new users. This means making it easy to install TeX (which both TeX Live and MiKTeX do, I think), making it easier to edit TeX files (the excellent TeXworks helps here), and making it easier to use TeX. It is, of course, the last one that is the problem. I think that we do need a new LaTeX as part of this. Removing the need to load a dozen packages and alter half a dozen internal macros to get reasonable output would benefit all LaTeX users. At the same time, something as simple as a generic thesis template could be made available: that would again help to “sell” LaTeX and by extension TeX as a whole.

LuaTeX has a role to play here. Using system fonts with pdfTeX is not easy, and XeTeX has helped a lot with this. LuaTeX provides this possibility and many more, and so we can hopefully look forward to not having to worry about loading system fonts at all. At the same time, it should be possible to do things similar to the current BibTeX and MakeIndex programs directly in the engine, but with full UTF-8 input. This is something that is long overdue, and again can’t hurt when it comes to selling (La)TeX.