LaTeX3: Key points

A comment on one of my other posts raises the important issue of what the key targets are for LaTeX3. Only the team can answer this, but reading the project webpage, the mailing list and the code, you can get some ideas.

  1. A well-designed, consistent and documented coding system. This is the most complete part of the current code base. The expl3 module provides a lot of low-level coding methods, which deal much better with control of expansion than using plain TeX or LaTeX2e (no long \expandafter runs). As I’ve said in another post, there are ideas that you can take from this for current work.
  2. A much larger and more functional kernel. The current LaTeX kernel provides only a limited number of functions and customisation possibilities. This is one reason for the very large number of LaTeX packages around. The new kernel will need to cover a large amount of what goes on in packages under LaTeX2e. This is much more the ConTeXt model, with the core team providing most of the basics.
  3. Internal macros taking a clear and fixed number of arguments. Currently, a lot of coding and design decisions are mixed together. User macros take optional arguments and various complex ways are used to send these through to the underlying functions. The idea for LaTeX3 is that there will be a “glue” layer, which converts from user to internal syntax. At the internal level, each function will have a strictly fixed set of arguments, and the glue layer will provide those from user input.
  4. Separation of design from code (and day-to-day use). The current kernel has design decisions all over the place: in the kernel, in classes, in packages and in documents. The LaTeX3 team are aiming for a system where functions are general, and make no assumptions about design. The glue layer described in (3) then brings in design decisions, which again are separated from user macros. Exact details are still some way off (although template shows a way of doing things).
  5. User syntax decoupled from internal design. It seems likely that the main user “interface” to LaTeX3 will remain a file starting \documentclass and ending \end{document}. However, the ideas in (3) and (4) mean that it should be possible to use a different glue layer to typeset completely different user input with the LaTeX3 kernel. This should allow LaTeX3 to take on entirely different, but structured, input formats (most obviously XML-based).
  6. New ideas for complex layouts. A lot of work is going into areas such as grid typesetting and complex float handling. This is intimately bound up with the output routine, and so LaTeX3 will provide a totally revised system in this area.

There are a lot of challenges there, and progress in different areas is variable. A lot of the work, of course, is in the thinking stage rather than writing the code. I’d say that all of these ideas are good things, and so it only remains to implement them all!

11 thoughts on “LaTeX3: Key points

  1. I should probably have mentioned the breqn package as well here. This does very clever things with maths, and I suspect will feature in LaTeX3.

  2. That’s pretty interesting.

    The question is, however, why reinvent the wheel – Hans has already written ConTeXt…

    And more seriously: exactly as you stated, the real problem is: what are the design goals of LaTeX? Currently (as of 2e), LaTeX is fundamentally flawed: it *claims* to be a “document preparation system”, but in fact it *is* a “scientific paper preparation system”. It would be ok with me if it were clearly stated – but it’s not. Instead, LaTeX has numerous hacks and dirty tricks allowing it to produce other kinds of documents (mostly implemented as packages). This results in a feeling that “you can do everything in LaTeX, but you get a feeling that it’s somehow awkward”. Yes it is – because usually it’s just the wrong tool (unless you write a paper on maths, when it’s very nice).

    But maybe I’m wrong. Maybe it would be refreshing to have some real competition. Maybe LaTeX3 will be redesigned so heavily that it will really be a “document preparation system”.

    Remeber, however, that ConTeXt has two things LaTeX doesn’t (or am I wrong?): they have lua and apparently they have cured themselves from the “backward-compatibility-syndrome” (with MkIV). This might make ConTeXt a very tough competitor…

    • My take is that ConTeXt and LaTeX have different aims. ConTeXt is rather more like plain TeX, in the sense that each document is independent. So ConTeXt is great for arbitrary document design: if you’re creating a one-off book, there are good arguments for using ConTeXt. On the other hand, LaTeX is more concerned with “reusable” document design, hence the document class idea. So I’d see the two as more complementary than in direct competition. Also in the aim vein, the ConTeXt development model (small team, everything done by them) would probably not work so well if everyone using LaTeX made the switch. Being able to contribute code in an ad hoc sense is one of the reasons LaTeX is popular (although more control would be very welcome here!).
      I can’t say that I know enough about writing “serious” works outside of science to comment on how easy or difficult it is in LaTeX2e. My own experience writing chemistry reports (papers are done in Word) is that the kernel still needs to be supplemented to work well. So I’d say that the LaTeX3 kernel will need to be a lot bigger to be useful without lots of bolt ons, for all subject areas. I’d be interested to know what the “top issues” are.
      On the back-compatibility issue, it’s not stated anywhere officially, but it seems pretty clear that LaTeX3 won’t work with LaTeX2e documents without editing. So I’d say that the message is as understood for LaTeX as for ConTeXt. As to Lua, it doesn’t look like LaTeX3 will use it in the kernel, but I’d hope that the team will be a bit more proactive in planning beyond LaTeX3 once it is released. I’d hope for “3.5” work to start as 3 is finalised, and to use Lua. But that might just be me!

  3. Joseph also asked for comments on the LaTeX3 list; until I start my own writings on LaTeX3 on a website somewhere, I’ll have to reply in that medium:

    Here’s what I wrote:

    * * *

    Hi Joseph,

    I think I’ve mentioned to you before that our thinking here is very
    closely aligned; in my opinion you’ve done a good job covering the
    “big vague issues” that I expect LaTeX3 to address in time.

    The most interesting part for me is your point #2 — designing an
    improved, consistent interface for functionality that spans what’s
    described in The LaTeX Companion and then some.

    I think I might have mentioned it before but I think it would be a
    good idea to examine projects like Gellmu and see if we can coerce or
    pervert the existing LaTeX document syntax into validating XML; I
    believe that we can make it easier to write LaTeX by hand (since I
    assume that people will always want to do this) by decreasing the
    number of “special characters” and by eliminating the space-gobbling
    macro that takes no arguments (which is useful for macro writers but
    error-prone for document authors). (E.g., forcing users to write
    something like `LaTeX;` rather than `LaTeX{}` or `LaTeX ` or
    `{LaTeX}`.)

    I don’t have much to say about LaTeX3 vs. ConTeXt besides the fact
    that I’ve hardly used ConTeXt, and it does seem like many of the goals
    of the LaTeX project have already been implemented in ConTeXt; it’s
    possibly only critical mass that prevents it being a worthy successor
    to LaTeX2e. On the other hand, I think it’s good for us to approach
    similar problems independently to explore different possibilities. As
    Joseph commented, I hope that ConTeXt Mk IV and LaTeX3 could be
    complements rather than direct competitors to each other.

    Staying on this point for a second, one big philosophical different
    between the two is that once the LaTeX3 document model is designed it
    will no doubt remain rigidly backwards compatible for many many years,
    whereas ConTeXt may continue to evolve and continue to break backwards
    compatibility to various degrees (although this may well not be the
    case after Mk IV is complete).

    Hypothetically speaking, I might expect a strict LaTeX3 document class
    to be eligible for becoming an ISO standard, whereas ConTeXt is not
    being designed, as far as I know, to naturally fit into this sort of
    model of rigidity.

    Meanwhile, better get back to work so that I can spend some other time
    on the expl3 code so we can start working on all these things that we
    keep talking about 🙂

    Will

  4. I have extensively used both LaTeX and ConTeXt, but have not kept up to date with LaTeX3. I agree with Marcin: ConTeXt already solves the problem. Specifically, taking your points

    1. It has a well designed and consistent document coding system. Now with lua in the picture, a lot of stuff is *much* easier. We may not need an explicit auxillary file anymore!

    2. I doubt you can get a larger kernel than ConTeXt 😉

    3. ConTeXt does not plan to do this. I am not sure how useful it is. Most programming languages to not specify the airity of the function in the function name.

    4. I guess this is true for almost all markup based language (except CSS, which is a mess).

    5. ConTeXt already has excellent support for XML. Now all the XML parsing is done using lpeg (a lua) library, and you can specify both tree and stream based rules.

    6. ConTeXt does grid layout and column sets. However, making any change in the output routine still requires a lot of work. (There is a module called streams, which makes things a bit easier. But changing the output routine is never going to be easy in ConTeXt).

    I disagree with you on the aims of ConTeXt. You can have ConTeXt modules which are similar to LaTeX classes and packages. If you want a design for a journal, write a module for that. The same way you do in LaTeX. Ok, in LaTeX most of the time someone else has written the package for you. But someone has to do it. Someone will have to do it in LaTeX3.

    A module can do any ad-doc thing that the module writer wants. Just like LaTeX packages. There are fewer ConTeXt modules, simply because Hans can code faster than you can request features.

    I believe that the only thing that provides backward compatibility is plain TeX. If backward compatibility means that an old document compiles correctly, then to a large extent, both latex and context are b/c. If b/c means that you get the same output, the neither is. Sure, the latex kernel seldom changes, but you will alsways use some packages in latex, and packages change all the time. There goes b/c. In fact, Hans is very particular about b/c. Often there are ways to do things more cleanly, but he has refused to change many things saying that it would affect b/c.

    It is really sad that LaTeX3 will not support lua. There are many deficiencies in TeX (e.g., automatic linebreaking of equations, syntax highlighing for verbatim text, sorting for bibliography, index, and table of contents) which are much easier to do in lua than in plain TeX. At least, once luatex is out, these things could be done via latex packages.

  5. Mahajan, thanks for the very detailed comment, which I think deserves and equally detailed reply! First off, I should say that there are some parts of the ConTeXt user interface that I’m not so keen on. So I’d like the option of LaTeX3 at some point 🙂

    On the coding syntax/naming, I’d agree that including things about the arguments of a function in the function name is unusual. I was a bit sceptical myself at first, but I think this is useful due to the nature of the TeX language. An obvious example is expansion. With plain TeX, you need to do things like defMyMacro but expandafterdefcsname MyMacroendcsname to do a name definition. Using LaTeX3, the same is done with def:NpnMyMacro and def:cpn{MyMacro}. Leaving aside the “pn” part (which may change), the def:N versus def:c idea is one I like.

    Th point about a consistent language is that in LaTeX2e and ConTeXt you use a mix of TeX primitives and format macros to programme. The LaTeX3 idea is that *everything* is given a LaTeX interface (and name). The primitives tend to have syntax inconsistent with higher-level functions, and you are still left needing to learn two languages (TeX + LaTeX or TeX + ConTeXt).

    On the usemodule point, you are correct that I missed this possibility out. The LaTeX3 team are very keen to *really* separate functions and design. The problem with usemodule is that there is no separation: a module can provide new functionality or alter design settings. LaTeX2e does better than LaTeX2.09 on this, but is still very poor. I’d expect LaTeX3 to do much better.

    Writing the code is obviously a problem (after all, LaTeX3 was started before LaTeX2e came out!). I suspect that LaTeX3 will be bigger than ConTeXt (given the general “we need everything in the LaTeX Companion as a starting point” design brief).

    On Lua, I’m not sure what will happen. At present there are no plans to require Lua for LaTeX3, at least partly because it will take years to propagate around. (I’ve recently had problems with a publisher where the e-TeX extensions are not available.) I’d think that this may change: I’d like to see Lua used if available (I’ve suggested that LaTeX3.5 might be an idea for that).

    Finally, there are no guarantees for LaTeX3. At some point, if the team cannot deliver, people will move to ConTeXt, no question. I’m certainly trying to improve my ConTeXt skills.

  6. I have rarely seen expandafterdefcsname MyMacro endcsname being used. In LaTeX, one normally uses @namedef{MyMacro} (spelling?). In ConTeXt, setvalue{MyMacro}. But I agree that a consistent interface is easier to use. The trouble with both LaTeX and ConTeXt is that there is little documentation of the programming extensions that they provide. To be fair, the documentation is present for both, but it is hard to find. To be successful, LaTeX3 will need to provide something similar to the TeXbook, for new users to understand the programming interface.

    What is the distinction between functionality and design. Suppose I am writing a LaTeX class/ConTeXt module for a journal, which requires the user to specify keywords. Suppose the default class does not provide a keyword environment. Is adding keyword env part of the style or new functionality?

    The thing that worries me most about LaTeX3 is how well it will be tested before it is released. ConTeXt has more of a rolling release philosophy, and right now there are quite a few users who are actually using Luatex+MKIV. So, bugs get caught and corrected. LaTeX3 needs to move to a stage where users can use it.

    The other concern is whether the big TeX publishers (like IEEE, SIAM, Springer, AMS) change to LaTeX3 in their workflow. This is right now a problem with ConTeXt. Which means, even if I like ConTeXt, but know that the paper that I am working on will finally need to be submitted in LaTeX2e, I will use LaTeX2e. The same will be true of LaTeX3.

  7. My example was perhaps not the best. Better would have been edef (there is no @nameedef in LaTeX: I don’t know about ConTeXt). The more general point is that this applies across LaTeX3 (rather too many places to go through them all).

    You are quite right about documentation. That seems to me to be something that the LaTeX team are aware of. The current “source3.pdf” file is a start (for the low-level stuff), but I’d agree that both LaTeX2e and ConTeXt suffer from the lack of an “official” book (or at least book-as-PDF) on the programming side of things.

    I’d also agree that there will always be a somewhat blurred line when new functions open up new possibilities for design. On that, I have not idea how the team will tackle things.

    For using LaTeX3, the only part that is currently usable is the low level programming stuff. That seems to be reasonably robust, and the team are working on an automated test system at the moment. But as yet there is not enough “on top” of that to see how things will work. For example, if you look at my xnotes2bib package (my notes2bib package re-implemented in LaTeX3), only the low-level stuff is in LaTeX3: the higher level stuff is still LaTeX2e.

    The final point you make is of course the biggest issue of all. On that, “critical mass” is important, and it’s not clear that LaTeX3 can get that. But LaTeX2e can’t go on for ever. So at some point something has to give if the LaTeX team do not deliver (we all move to ConTeXt, we all have to use Word => Quark/InDesign, something else as yet unknown).

  8. IMO LaTeX3 and (pure) ConTeXt are not going in the right direction. As Will pointed out, writing things with backslashes is error prone (LaTeX) and having unstructured section{…} in your document is a one way. The backends are only good for programmers or people that know how to ask a programmer for help. So the direction should be XML (or some other kind of structured content).

    Instead of writing LaTeX3 which is only another interface for unstructured, error prone input, we should provide XML interfaces for high quality typesetting. ConTeXt has (I’ve hardly used it) the ability to have XML-only documents. And I am currently writing an XML only typesetting system.

  9. Hello Patrick,

    Sorry it’s taken a few days to get back on this: busy times.

    There are a lot of different things that people want from typesetting systems. One of them is, as you say, highly structured data processing. That makes sense at a publisher working from a database (which is what a lot of journals do), and support at least from LaTeX needs to be better. As you say, ConTeXt does handle this. My own view is that TeX is the wrong way to do things here: a parser for XML is much easier to write in other ways, with output to the typesetting end from that process. Of course, this would be helped by a more structured LaTeX typesetting system.

    However, other people use LaTeX differently. There are a lot of people who want to be able simply to type documents and have them look good without any database back end. I can only speak for myself, but there is no way I’d favour typing documents in XML over typing them in some kind of more flexible markup (LaTeX , markdown, HTML, etc.). For those people XML is not directly relevant.

    One idea that has been suggested is that LaTeX (as an input syntax) should be converted first to XML then to a typesetting language. That still needs a more structured LaTeX, and I suspect is far from trivial. I’m not ruling it out, thought.

    One question is always the amount of time available. The ConTeXt team have put a lot of commercial effort in, and they’ve still not moved to a pure XML interior (or indeed a TeX-free parser). So I’m not sure that LaTeX is likely to either: I think Hans and Taco will have considered this.

    Joseph

Leave a Reply