LaTeX3 as a low-level language

There is quite a lot going on with the low level code for LaTeX3 at the moment. The number of commits to the code is ticking over nicely, as the code is revised and Will Robertson gets the test system written. Will has taken on a thankless task with this job, and I think is owed a debt of gratitude by everyone interested in LaTeX3.

One thing that is clear is that LaTeX3 (at the low level) is a programming language in itself, distinct from TeX. This is something of a risk, as it means that you cannot simply take what is done currently and convert it to the new system without thinking. On the other hand, the idea is to provide a system which makes programming easier and clearer, with some of the “features” of TeX hidden underneath a working LaTeX3 layer. The aim is that expansion and the somewhat odd methods for assignment to low-level TeX variables become something only the kernel team need to worry about.

It seems to me that there is a “window of opportunity” for the kernel team to show that something can be delivered, and that LaTeX3 as a programming language is the way to do this. The arrival of LuaTeX will bring real programming to TeX: Karl Berry has pointed out that this will bring programmers to the TeX world who would never consider writing serious (La)TeX code. So I think that LaTeX3 needs to show that the language is viable before widespread take-up of LuaTeX means that no-one is interested.

With the LaTeX3 low-level language approaching stability, I’d suggest the team need to make the next step and move on to document design and user macros. There are lots of ways this could be done, but an obvious one would be to produce a “microkernel”. By this, I mean something which can do the same as:

\documentclass{minimal}
\begin{document}
Hello World!
\end{document}

without using LaTeX2ε at all. I’d expect the user syntax to change a bit, but in essence the minimal LaTeX document is not going to change (at least, not if LaTeX3 is to succeed).

The experts will know that there is quite a lot of work to get to the test file I’ve suggested, and so going from where LaTeX3 is now to a working microkernel is non-trivial. However, it would be a good opportunity to demonstrate that real (usable) progress is happening, and would also avoid the problems associated with trying to build everything from the bottom up with no top level in sight. A microkernel would show that delivery is possible, and I hope raise interest in LaTeX3 from the current and potential development community.

Of course, a microkernel will still leave a lot to do. However, with something to build on I’d expect interest and ideas to accumulate and the new system to grow reasonably fast.

10 thoughts on “LaTeX3 as a low-level language

  1. I really like to see latex3 make use of luatex. Looking at the progress of luatex and the team behind it, it is going strong. I am positive it will meet the goals set out in its roadmap and have a usable release in two years time. For latex 3 to have its programming interface, isn’t this doubling the effort of users/developers learning two languages?

    • I’ve said in other posts that I’d hope to see LuaTeX used if available in LaTeX3 for some things. However, there are several issues to bear in mind. First, the LaTeX2e kernel is, in all honesty, a mess. The basic work to update it is going to be done in TeX or not at all. Look at ConTeXt Mark IV: the low-level stuff is being done in Lua but not the majority of the code. Second, it will be years before most TeX installations have Lua available: I do some stuff for a publisher where e-TeX is not an option! Making LaTeX3 that “closed” would be a bad idea. Third, I’m not sure that “everything in Lua” is the best approach. There will always be a need to program in TeX if you are using TeX, and the question is then what is the best way. The team as a whole hope that LaTeX3 is, if not the ideal, then a lot closer than the current LaTeX2e approach. Finally, there is the time question. To throw away everything now is probably to write off LaTeX3 entirely: I just don’t see it happening if we try to re-write everything in Lua. I’d hope that there will be a more graduated approach to LaTeX3 than LaTeX2e (not just 3.0 but perhaps 3.1, 3.2, etc. with the possibility of changes to include more Lua use).

  2. I haven’t actually tried building a LaTeX3 kernel of my own, yet. If we add it to the Makefile then we can certainly keep it maintained. I believe that, ostensibly, we should have enough to compile a kernel that can’t do much but put boxes on the page 🙂

    Even if expl3 ends up being more useful for LaTeX2e package writers, I think you’re right that having a LaTeX3 kernel will help to support the notion that the code is Ready To Use (which it will be to a fair degree as soon as we start to freeze the internal function names).

  3. Certainly the current expl3 could be compiled into a format (if not today, with a few edits). However, that wouldn’t really be what I’m getting at. For me, at least, there is a strong need for a new LaTeX kernel. To create a document I need to load a lot of packages that I really shouldn’t need to think about. See how many templates show bad package use, for example (times, subfigure, epsfig, etc.). We can help developers all we like, but in the end it is the user experience that we need to focus on.

    I didn’t say, but in my minimal example I’d expect native UTF-8, T1 encoding “out of the box” and Latin Modern as the default fonts, as a minimum. I was also implying an output routine (which I suspect would initially be the LaTeX2e one in expl3, pending more improvements). I’d be then wanting to build on that: something like babel without needing to load a separate package, PDF metadata (hyperref), and so on. That is before you even think about anything that every document does not use (look at the supported modules in ConTeXt).

  4. I follow your posts on LaTeX3 with much interest, Joseph, and I must say that I got a little bit more excited about this project overall. However, this post got me baffled again. So is LaTeX3 going to be a competition to LuaTeX? What’s the point?

    Traditional TeX programming isn’t going to disappear anytime soon, although I wouldn’t miss it for one minute if it would. LuaTeX is coming soon and it’s coming strong. Do we need yet another way to program things?

    Now, I understand that many ideas (most of them?) in LaTeX3 predate LuaTeX, but I can’t shake the impression that the whole bottom-up approach to LaTeX2e successor misses the point. What really counts in this digital age is the ability to manipulate our documents in all sorts of ways (multiformat output, conversion, graphical frontends, etc.). For that we need a better document format that will allow to build better tools around it. LaTeX2e just doesn’t cut it, it never did. This is what hinders LaTeX the most, IMO. Your sources are quite useless without TeX, because that’s the only program that can truly understand them. But that makes your documents tied to paper output and this is just not enough any more.

    What’s lies beneath the format, i.e. the whole machinery to process and display/print your document, is of secondary importance—it can be replaced over time transparently to the user.

    Of course, I’m simplifying things a lot here. The format itself, even the best one, is useless without tools that work with it. But I would risk a statement here, that the majority of users couldn’t care less about what’s turns their sources into output, as long as they get functionality and output they need.

    From that perspective you could build a new system on top of the current one and over time replace/extend it bit by bit transparently to the user. Going bottom-up will result in providing the new document syntax as the very last element and therefore will delay the adoption of it by years or even decades.

    Perhaps, I’m missing a point here. I don’t know. But somehow I always arrive at the same conclusions, no matter how I try to look at this.

  5. This blog is great. Is there any good documentation on what comprises LaTeX3 beyond the summaries and “newsletters” posted over at http://www.latex-project.org ?

    It seems to me that LaTeX could advance much more quickly from a bazaar-type development model rather than the cathedral model it currently has. Granted there are some parts that are considered “research” and will take much more time, rather than just the clean re-implementation or re-architecting aspects, but every time I think about it, I think that the world is waiting for the development of the next-generation LaTeX system to be way more open than it currently is.

    While there is a predictable avoidance to design-by-committee, I think the benefits of a more open development process outweigh the downsides. Already, an svn repository is available and the mailing lists are public. Maybe my criticism of the openness of the project is unfounded, but one of the risks also here is that LaTeX3 would be eclipsed by some other competing system which comes out in the meantime, which the original post above seems to be suggesting.

  6. Tomek, thanks for the detailed comments. I’ll try to give you the detailed answers they deserve!

    On the LuaTeX issue, you have to remember that LaTeX3 can only do things that are already possible in (pdfe)TeX. So a lot of what is there is simply formalising what clever TeX programmers have done for many years (or less clever ones have found and copied!). On the other hand, the key point of LuaTeX is the things it brings to TeX that TeX cannot do. Yes, there are general programming tools in Lua but these alone are not what is exciting. It’s the stuff that only Lua can do that will get used. See what the ConTeXt people are up to with Mark IV: they are still using a lot of TeX, and Lua is coming in where only it will do.

    So what is the expl3 language for? Well, as I’ve said it’s a formalisation of what might be considered best TeX practice. By putting this in the kernel, everyone gets to use it. Things like more reliable Boolean switches and proper variables are really useful, and mean that the oddities of TeX expansion become less of a worry. I’d say that is a “good thing”.

    Looking at the input situation, Frank is very keen that we keep a focus on avoiding what LaTeX2e gets wrong here. The current situation ties the internal functions too much to the LaTeX input syntax, which makes changing the input method hard. So the LaTeX3 idea is that the internals are written without a fixed input method in mind, and then a translation layer converts input to internal syntax. That way, only the translation layer has to change to alter how input works. For example, look at how things like section are currently implemented. There is simply too much overlap between different parts of LaTeX2e.

    What I’ve suggested with a “microkernel” is very much what you are saying about changes over time. I don’t think you could start from LaTeX2e and “morph” into LaTeX3, because there are too many problems with the current syntax. Almost all real documents would break too earlier for this to be viable. So what I’m suggesting is getting something working that can do the very basics, and adding to it over time. As each part of the user interface was written, I’d aim to fix it but not the underlying implementation. I think this has to start with a LaTeX2e-like document, although I’d hope to add some kind of XML input as a parallel method quite quickly.

    I’m not saying this is necessarily going to happen. Whether LaTeX3 can deliver is still very much up in the air, for me at least. LuaTeX is important here because of momentum (if everyone is doing different things, many of them will not have critical mass).

  7. Tom, at the moment what is on the project website covers quite a bit of what there is. If you read the LaTeX-L list and the documents for the code (“source3” and “expl3” plus the xpackage stuff), you’ll know pretty much as much as I do!

    As you say, both the LaTeX-L list and the SVN are public. I’m on the project team because I said some half-sensible things on the list and had a go at writing a couple of small modules. I think anyone proposing sensible code/ideas will be welcome to help out. Once again, I feel what’s needed is a microkernel, showing how things should work. We might then see other interested people submitting single modules, quite possibly conversions of existing LaTeX packages to fit the new model. I’d hope that after a “review” these could be added to the kernel so that functionality could grow quite quickly. What I’m not sure about is whether we’ve got the resources to do this in a formalised way (with owners, bug databases, etc.). There will only ever be a small core team of volunteers. So a lot will always happen by e-mail between individuals.

  8. I’ve never heard L3 described as a programming language before. In fact, that one statement clarifies a great deal about L3. So, my (strong) suggestion is, if it’s a programming language, how about some documentation describing it *as* a programming language?

    The L3 documentation I have seen has always been either extremely general (so I end up with no real idea of what it does in practice) or extremely detailed (so I end up with no real idea of what the idea is). That’s why I asked you for the TUGboat article(s) to try to bridge that gap …

  9. Karl, thanks for the comment. What I’m trying to get at is that *part* of LaTeX3 is a programming language (“expl3”). This is the low-level stuff on which things like the xpackages are built. For the user, the low-level stuff should not matter (as other people have also pointed out).

    Of course you are right that something on “LaTeX3 as a programming language” is entirely necessary. The current re-factor means that some of what was true in the past is not any more. I’ll look at updating my “intro” article once things look a bit more settled, and get the revisions to you.

Leave a Reply