Some TeX Developments

I’ve been working for a while on a method to provide a reasonably powerful method for creating keyval input for LaTeX3. This has been going under the working title “keys3”, but yesterday I took the plunge and added the code to the LaTeX3 development repository. For the moment, this can only be accessed via SVN, but if the rest of the team are happy with the idea, it will appear in the next snap shot that is sent to CTAN.

The ideas in the new module (now called l3keys) have been inspired by the pgfkeys package. By using keyval methods to create keys, the idea is to make life a lot easier for the programmer. However, things are somewhat modified compared to the pgfkeys package, mainly to try to keep the input syntax simple but powerful enough for most uses. For example, there are separate functions for creating keys and setting them, an idea that all other keyval packages use. So a typical setup block might look like:

\keys_define:nn { module } {
  key-one   .set:N     = \l_module_one_tl,   % Store input in variable
  key-one   .value_required:,
  key-two   .bool:N    = \l_module_two_bool, % Either true or false
  key-three .code:n    = { Code using an argument #1 },
  key-three .default:n = text
}

while keys are set using:

\keys_set:nn { module } {
  key-one = Value,
  key-two = true,
  key-three
}

Hopefully, the ideas are flexible and clear enough for people to get working with. One thing to notice: the key names are hyphenated. It looks like that will be the “style guide” suggestion for LaTeX3 keys. At some point, I’ll look at writing a TUGBoat article on how everything works.

I’ve been doing quite a bit of thinking about siunitx version 2. My original plan was to write everything in standard LaTeX2e code, but the more I thought about things the more it has not looked like such a good idea. siunitx needs a lot of programming tools, and these are almost all available in the experimental LaTeX3 code which is now approaching stability. At the same time, there are still some difficult issues to solve for siunitx, and I’m going to need some more support stuff, I think. So I’ve revised my ideas on how to progress.

Currently, I’m moving the code I have already re-written to use LaTeX3 conventions internally. I’m then going to try to solve the remaining issues for version 2. The aim is to have a package which will run on LaTeX2e, using LaTeX3 internals, so that I can eventually create a LaTeX3-only edition with the same interface ideas. I’m going to try to crack on with things, but it will take a while to solve the remaining issues!

The next step for LaTeX3 development is to revise the two existing “xpackages” which are available to make a link between the code level and the user: xparse and template. Of the two, the xparse package is by far the easier to understand.

In LaTeX2e, you can either use the LaTeX method to create new commands:

\newcommand*\mycommand[2][]{%
  code goes here, using #1 and #2
}

or you can do things yourself using a mixture of TeX primitives and LaTeX internal functions (and ignoring the issue of default value):

\def\mycommand{%
  \@ifnextchar[{%
    \@mycommand
  }{%
    \@mycommand[]%
  }%
}
\def\@mycommand[#1]#2{%
  code goes here, using #1 and #2
}

This does not make for code which is easy to alter to reflect different input syntax (for example XML), or to change internal functions without knowing how the user input works. The idea of xparse is to separate out the user syntax from the internal code. Currently, exactly what tools need to be available is still being decided. The current version of xparse would expect the following syntax:

\DeclareDocumentCommand \mycommand { o m } {
  code goes here, using #1 and #2
}

The idea is that each argument is represented by a letter, for example o for an optional argument, m for a mandatory one or s for an optional star. This makes it possible it see immediately from the definition how many arguments are needed (one for each letter). It also means that the internal functions, which implement things, can be separated totally from the user part of the system. In that way, internal functions can have a fixed number of arguments, and leave xparse to supply them. So if the internal function needs to be changed, it does not matter how it is used, or vice versa.

That all sounds very good, but there are some outstanding issues. For example, handling verbatim arguments is not straight-forward. There are a couple of possible approaches to this. Either stick to a simple system, and accept that not everything can be done in the same way, or make the system more flexible but complicated. I’m currently in favour of the first approach: almost every user function is simple (especially as we have the ε-TeX extensions and so can \scantokens our way out of a lot of problems). I’d be interested to hear what other people think would be useful.

The LaTeX3 Project (of which I’m a member) have made a snapshot release of the current LaTeX3 code to CTAN today. This is in two parts:

• expl3, the base layer for LaTeX3, which provides a consistent programming interface for LaTeX3 on top of TeX (or rather, on top of pdfTeX, XeTeX or LuaTeX).
• xpackages, higher level interface ideas for LaTeX3.

expl3 is now reasonably stable: the code has been carefully reviewed and the Team are happy that what is there works well. There are gaps, and some of these are definitely on the “to do” list. On the other hand, the xpackages bundle is not so close to being finished. At the moment, it includes only the parts of the experimental code that look the closest to reaching some kind of conclusion. However, it is also the part that is actively being reviewed at the moment. So I expect changes!

I’m hoping that as expl3 gets distributed around the TeX world it will be possible to consider using it for coding new packages. The ideas in expl3 make coding much easier, over all, even if initially there is a lot to learn. Of course, delivering more of LaTeX3 will help in getting people interested as well. That is something I’m definitely keen on.

Testing the current LaTeX3 code means worrying about what primitives are available. Recent releases of pdfTeX have included a number of new primitives, and the expl3 system currently uses \pdfstrcmp if it finds it. This has already caused one issue, as things were broken if it was not available. So I’ve been modifying the test system a little, to run the tests with pdfLaTeX and XeLaTeX. The “new” primitives are not available with XeTeX, so this is a good test of our code: it’s supposed to work with both. I’ve found one odd thing going on with the LateX3 code, but also one thing that seems to be a XeTeX snag.

The snag can be tracked down (thanks to Morten Høgholm) to a minimal case:

\message{\ifcat\par\relax T\else F\fi}

If you try this in TeX or pdfTeX, you get T in the log (as The TeXbook says you should). With XeTeX, you get F. Whether this is a bug or a feature, I don’t know. I’m sure Jonathan Kew will elaborate, but it reminds me of the need to test using different set ups. You can imagine some very odd errors appearing with this type of thing!

A while ago, I drafted an overview article on LaTeX3 for TUGBoat. Since then, there have been some key changes to the current LaTeX3 code, and so I’ve slightly revised the article. It is aimed as a very general overview of what is available at the moment, and is available as a PDF and in source form.

As I said earlier in the week, I’ve been revising my key3s package, which is an experimental keyval system in LaTeX3 syntax. I was initially aiming to keep things up to date with recent changes in LaTeX3, but the more I looked the more there was to do. So the new version is rather different to that of a week ago. I’ve dropped a number of things that I’d taken from pgfkeys (part of the pgf bundle) that, with thought, I’m not happy about. That is not to say that some things wouln’t re-appear: I still want to look at some of the new stuff in the development version of pgfkeys.

I’ll probably upload the package to CTAN over the weekend, but for the moment the interested can grab:

• The documentation (.pdf)
• The source file (.dtx)
• Docstrip extration file (.ins)
• Demonstration/test file (.tex)

A while ago, I did a bit of work on converting the pgfkeys package (part of pgf) to LaTeX3 syntax, and called the result keys3. Some things have changed in LaTeX3 since then, and I also have an outstanding list of things to think about adding to keys3. On top of that, I’ve spotted at least one bug (inherited from pgfkeys!). So the question is: is it worth fixing them? At the moment, keys3 might not go anywhere, so I’m not sure. Thoughts, anyone?

A recent announcement on comp.text.tex for a new package, silence, coincides with some discussion in the LaTeX3 team about how to handle messages for the user. The LaTeX3 stuff is currently looking at the low-level side of the area, whereas silence is dealing with things for the user.

The problem of “too much information” is clearly one that has attracted attention. An awful lot of what LaTeX prints is not interesting, most of the time. I’d say that a better model would be less, more targetted information as standard. A “developer” mode, printing more detail, is still needed but to be honest even then do many people care about some of the stuff that gets logged. Most of the time, I’d say no.

The silence approach (filtering on a per-package basis and also based on specific text in messages) is clearly very powerful but somewhat complicated. I’d hope that the LaTeX3 team will provide some filtering along with properly named messages (rather than the current situation where messages just appear directly in the code). Perhaps not quite as powerful as silence, but a lot better than at the moment.

There is quite a lot going on with the low level code for LaTeX3 at the moment. The number of commits to the code is ticking over nicely, as the code is revised and Will Robertson gets the test system written. Will has taken on a thankless task with this job, and I think is owed a debt of gratitude by everyone interested in LaTeX3.

One thing that is clear is that LaTeX3 (at the low level) is a programming language in itself, distinct from TeX. This is something of a risk, as it means that you cannot simply take what is done currently and convert it to the new system without thinking. On the other hand, the idea is to provide a system which makes programming easier and clearer, with some of the “features” of TeX hidden underneath a working LaTeX3 layer. The aim is that expansion and the somewhat odd methods for assignment to low-level TeX variables become something only the kernel team need to worry about.

It seems to me that there is a “window of opportunity” for the kernel team to show that something can be delivered, and that LaTeX3 as a programming language is the way to do this. The arrival of LuaTeX will bring real programming to TeX: Karl Berry has pointed out that this will bring programmers to the TeX world who would never consider writing serious (La)TeX code. So I think that LaTeX3 needs to show that the language is viable before widespread take-up of LuaTeX means that no-one is interested.

With the LaTeX3 low-level language approaching stability, I’d suggest the team need to make the next step and move on to document design and user macros. There are lots of ways this could be done, but an obvious one would be to produce a “microkernel”. By this, I mean something which can do the same as:

\documentclass{minimal}
\begin{document}
Hello World!
\end{document}

without using LaTeX2ε at all. I’d expect the user syntax to change a bit, but in essence the minimal LaTeX document is not going to change (at least, not if LaTeX3 is to succeed).

The experts will know that there is quite a lot of work to get to the test file I’ve suggested, and so going from where LaTeX3 is now to a working microkernel is non-trivial. However, it would be a good opportunity to demonstrate that real (usable) progress is happening, and would also avoid the problems associated with trying to build everything from the bottom up with no top level in sight. A microkernel would show that delivery is possible, and I hope raise interest in LaTeX3 from the current and potential development community.

Of course, a microkernel will still leave a lot to do. However, with something to build on I’d expect interest and ideas to accumulate and the new system to grow reasonably fast.