Some TeX Developments

A quick note to say that I see that Will Robertson has set up a blog ‘LaTeX Alive’. Will is the person behind the very useful fontspec package, which lets you choose system fonts easily when using one of the newer TeX engines (XeTeX and LuaTeX). Will is also very active in the LaTeX3 Project. I’ll be keeping an eye on his thoughts!

I see from the LuaTeX mailing list that version 0.45.0 has been released. There is the usual long list of new things and bug fixes, but some that caught my eye:

• \input and \openin now accept braced filenames;
• The new primitives \aligntab and \alignmark are aliases for the use of & and # in alignments;
• LuaTeX can now optionally use kpathsea to find lua require() files.

Regular expressions are very popular as a quick and powerful way to carry out searches and replacements in text of all sorts. Traditionally, TeX handles tokens and not strings or characters. This means that doing regex searches using TeX82 is pretty much impossible. To solve this, recent versions of pdfTeX adds the \pdfmatch primitive to allow real string matching inside TeX. The LuaTeX team have decided not to take all of the existing “new” primitives forward from pdfTeX, and as I understand it \pdfmatch will not be implemented in LuaTeX. However, Lua itself has regular expression matching, and so the functionality will still be around.

I’ve recently talked about adding new primitives to XeTeX, and you’ll see that \pdfmatch was not on the list for adding to XeTeX. The reason is that a XeTeX implementation would have to be slightly different from pdfTeX, as it is natively UTF-8, but also would be different to LuaTeX, as it would still be a TeX primitive and not a Lua function. So here “the prize wasn’t worth the winning”, in my opinion. As it is, using \pdfmatch is not widespread, and the idea of having three different regex methods inside TeX didn’t seem like a great idea!

Talking of regex implementations, I’ve been reading Programming in Lua, and also working with TeXworks to try to get syntax highlighting the way I like it. Both systems are slightly different, and it seems both are different from the Perl implementation. It seems that every time you want to use a regex system you have to read the manual to see which things are different from every other implementation!

There is currently a very active thread on comp.text.tex about LuaTeX. Several related questions have come up, and it seems that some kind of summary would help me, at least.

Why did the LuaTeX team choose Lua?

Possibly the most frequently asked question with LuaTeX is why the team behind it chose Lua, as opposed to one of the other scripting languages available. The LuaTeX FAQ addresses this as question 1, so it’s obviously important. As I understand it, they looked at a number of possible languages for extending TeX, and decided that many of the “obvious” candidates (Java, Perl, Python, Ruby, Scheme) were not suitable. Not everyone is going to agree with that assessment, and some people are bound to feel that there was not sufficient consultation.

With LuaTeX already at version 0.40, it is progressing and will deliver. So unless a team can be got together to provided an alternative scripting system, it looks like Lua is what we will get. I don’t see there being enough experienced programmers with time on their hands to deliver a complete alternative at the moment.

How does LuaTeX relate to ConTeXt and LaTeX?

LuaTeX is an “engine”, like pdfTeX or XeTeX. So it provides certain functions, which might or might not get used. The end-user can use them directly, but this is really not helping most people as they don’t want to do this type of low-level work. So in the main it is down to either the TeX format (ConTeXt, LaTeX, …) or an add on (LaTeX package, ConTeXt module, etc.) to take advantage of them.

In the ConTeXt case, the entire format is being re-written as “Mark IV”, which will use a lot of Lua. This of course means things fundamentally change compared to early versions of ConTeXt, and ties it to a single engine.

In the LaTeX case, the format is written to work on plain TeX, no extensions at all. That is not about to change: essentially, LaTeX2e will never be “updated” in that sense. The current LaTeX3 plans don’t envisage requiring LuaTeX, although I’d hope that some low-level support will be included if LaTeX3 ever becomes a reality. There will, though, be LaTeX packages that use Lua: I’m sure that at some point there will be a fontspec-like package for LuaTeX, for example.

What is wrong with LaTeX2ε?

One issue that always comes up when future directions for TeX are discussed is whether LaTeX2ε needs to change, or whether it is okay as it is. Currently, if you know what you are doing then you can achieve a lot with LaTeX. There are a vast range of packages, and these cover very many of the things you could ever hope to do in TeX. So in a sense LaTeX works as it is. On the other hand, you have to know which packages to load, even to get some of the basics right (try making a new float type without loading any packages and using one of the base classes!). At the same time, things like UTF-8 text, system fonts and so on are not available using only the kernel.

A more fundamental issue is that LaTeX currently doesn’t do enough to provide structured data, and is not a good choice for dealing with XML input. This makes it hard to get data in and out, and is closely related to the fact that there is not enough separation of appearance and structure in the kernel and in add-on packages.

Neither of these points has escaped the notice of the LaTeX team. The question is whether a successor to LaTeX2ε will appear, and if it does whether it can succeed. There is a need to start from scratch with many things, meaning that a new LaTeX simply won’t work with most packages currently available. So the team have to deliver something that really works.

How can (La)TeX recruit new users?

One question that comes up here is what is a typical (La)TeX user anyway. Everyone has their own view on this: some people see TeX users mainly as mathematicians and physicists, other people point out that particularly with XeTeX available there is a lot of potential in the humanities.

Making life easier for new users is a priority for gaining new users. This means making it easy to install TeX (which both TeX Live and MiKTeX do, I think), making it easier to edit TeX files (the excellent TeXworks helps here), and making it easier to use TeX. It is, of course, the last one that is the problem. I think that we do need a new LaTeX as part of this. Removing the need to load a dozen packages and alter half a dozen internal macros to get reasonable output would benefit all LaTeX users. At the same time, something as simple as a generic thesis template could be made available: that would again help to “sell” LaTeX and by extension TeX as a whole.

LuaTeX has a role to play here. Using system fonts with pdfTeX is not easy, and XeTeX has helped a lot with this. LuaTeX provides this possibility and many more, and so we can hopefully look forward to not having to worry about loading system fonts at all. At the same time, it should be possible to do things similar to the current BibTeX and MakeIndex programs directly in the engine, but with full UTF-8 input. This is something that is long overdue, and again can’t hurt when it comes to selling (La)TeX.

A recent announcement on the LuaTeX mailing list that LuaTeX has reached v0.40 has been expected for a while. One of the most notable, and widely discussed, changes is in the way new primitives are made available. As of 0.40, “out of the box” LuaTeX only provides one new primitive, \directlua. To get any primitives beyond those from TeX82, you then have to call Lua and get it to turn them on. The reason for this rather radical change is that it avoids any name clashes between new primitives and existing packages. So LuaTeX should, in principle, be able to replace pdfTeX as the engine of choice for most people. Of course, that is still some way off: LuaTeX is scheduled to reach version 1.0 in 2012.

Over the week-end, the copy of Programming in Lua that I’d ordered arrived at the bookshop. So I’ve been able to make at least a small start on the book, and get some of the ideas into my head. Having looked at the first 40 or so pages, I can see why Lua has been chosen for integration with TeX as LuaTeX. Some TeX concepts are also Lua concepts (such as the scope of variables and the ability to re-define anything), and so the fit between the two languages looks good. How that translates in practice I’ll have to see.

What I have found to date is that you really don’t want to be a beginner if you pick up Programming in Lua. The book starts with a lot of pretty complicated programming ideas, and without some background I’d be in trouble. Luckily I’ve picked up bits and pieces over the years (from Basic, C and Perl, amongst other things), and so I’m not totally lost. But for a book aimed at people starting off with the language, it is tough going.

The arrival of the book happily coincides with the release of LuaTeX 0.36.0, as announced on the LuaTeX mailing list. There are a number of changes aimed at making more changes in the future, but at least I can now begin to understand what some of them mean!

There has been quite a bit of dicussion recently on the LuaTeX mailing list about primitive names. The basic set of primitive names provided by TeX has already been extended by e-TeX, pdfTeX and XeTeX. While e-TeX primitives have names which express only the function (such as \detokenize), both pdfTeX and XeTeX have tended to add the engine name to all new primitives. So for example pdfTeX provides the \pdfprimitive function, which always carries out the orginial function of a name, even if it has later been redefined:

\let\everypar\undefined
% \everypar{Some stuff} % Gives an error
\pdfprimitive\everypar{Some stuff}

will, for example, add material to \everypar despite the redefinition.

In LuaTeX, the idea of adding an engine-specific prefix has been abandoned. The example above shows one reason why: \pdfprimitive makes more sense with the name \primitive (which is what LuaTeX calls it). The problem with this is that name clashes are marginally more likely without a prefix (for example, \expanded is used in ConTeXt as a macro, but this is planned as a primitive for pdfTeX 1.50).

On the other hand, LuaTeX is a fundamental break with previous engines, as the internal changes mean that it may give different line breaks and so on to earlier systems. Thus the argument for new, logical, primitive names is that moving to LuaTeX is not the same as moving from TeX to pdfTeX or pdfTeX to XeTeX. There will be changes, and the user (or package developer) will need to think about them before diving in.

Overall, I think the LuaTeX idea is the best in the long term. Primitives which have well specified names help everyone, and some times change is necessary. I’ll be interest to see if the \directlua primitive for actually using Lua ends up with a shorter name (just \lua, anyone?).

Progress on LuaTeX continues at an impressive pace. On the mailing list today I see that version 0.35.0 is available as a snapshot. The list of changes is, as you’d expect at this stage, quite technical and low level. Things are cetainly looking interesting, though.

There is quite a lot going on with the low level code for LaTeX3 at the moment. The number of commits to the code is ticking over nicely, as the code is revised and Will Robertson gets the test system written. Will has taken on a thankless task with this job, and I think is owed a debt of gratitude by everyone interested in LaTeX3.

One thing that is clear is that LaTeX3 (at the low level) is a programming language in itself, distinct from TeX. This is something of a risk, as it means that you cannot simply take what is done currently and convert it to the new system without thinking. On the other hand, the idea is to provide a system which makes programming easier and clearer, with some of the “features” of TeX hidden underneath a working LaTeX3 layer. The aim is that expansion and the somewhat odd methods for assignment to low-level TeX variables become something only the kernel team need to worry about.

It seems to me that there is a “window of opportunity” for the kernel team to show that something can be delivered, and that LaTeX3 as a programming language is the way to do this. The arrival of LuaTeX will bring real programming to TeX: Karl Berry has pointed out that this will bring programmers to the TeX world who would never consider writing serious (La)TeX code. So I think that LaTeX3 needs to show that the language is viable before widespread take-up of LuaTeX means that no-one is interested.

With the LaTeX3 low-level language approaching stability, I’d suggest the team need to make the next step and move on to document design and user macros. There are lots of ways this could be done, but an obvious one would be to produce a “microkernel”. By this, I mean something which can do the same as:

\documentclass{minimal}
\begin{document}
Hello World!
\end{document}

without using LaTeX2ε at all. I’d expect the user syntax to change a bit, but in essence the minimal LaTeX document is not going to change (at least, not if LaTeX3 is to succeed).

The experts will know that there is quite a lot of work to get to the test file I’ve suggested, and so going from where LaTeX3 is now to a working microkernel is non-trivial. However, it would be a good opportunity to demonstrate that real (usable) progress is happening, and would also avoid the problems associated with trying to build everything from the bottom up with no top level in sight. A microkernel would show that delivery is possible, and I hope raise interest in LaTeX3 from the current and potential development community.

Of course, a microkernel will still leave a lot to do. However, with something to build on I’d expect interest and ideas to accumulate and the new system to grow reasonably fast.

As LuaTeX progresses, the desire to use it in real documents increases. ConTeXt Mark IV is clearly the leading exponent of this, although it is still experimental and so perhaps not the best choice for day-to-day work. Karl Berry has said to me that he hopes that the inclusion of a “proper” programming language will bring new talent to (La)TeX. After all, some of the methods used to program TeX are complex to say the least.

This raises the question of how much to do in Lua and how much in TeX. Clearly, LuaTeX brings things to TeX that are impossible without it. So there it is simple: use Lua. However, what about things that can be done with either Lua or TeX? Part of this probably depends on the complexity of the different solutions (if it is easy in TeX, why do it in Lua, and vice versa). I’d also imagine that experiences TeX programmers will tend to stick to what they know, and favour TeX. On the other hand, new programmers coming to TeX because of Lua will favour the Lua route. I imagine that there will still be a lot more TeX than Lua code.