Sorry it’s taken a few days to get back on this: busy times.

There are a lot of different things that people want from typesetting systems. One of them is, as you say, highly structured data processing. That makes sense at a publisher working from a database (which is what a lot of journals do), and support at least from LaTeX needs to be better. As you say, ConTeXt does handle this. My own view is that TeX is the wrong way to do things here: a parser for XML is much easier to write in other ways, with output to the typesetting end from that process. Of course, this would be helped by a more structured LaTeX typesetting system.

However, other people use LaTeX differently. There are a lot of people who want to be able simply to type documents and have them look good without any database back end. I can only speak for myself, but there is no way I’d favour typing documents in XML over typing them in some kind of more flexible markup (LaTeX , markdown, HTML, etc.). For those people XML is not directly relevant.

One idea that has been suggested is that LaTeX (as an input syntax) should be converted first to XML then to a typesetting language. That still needs a more structured LaTeX, and I suspect is far from trivial. I’m not ruling it out, thought.

One question is always the amount of time available. The ConTeXt team have put a lot of commercial effort in, and they’ve still not moved to a pure XML interior (or indeed a TeX-free parser). So I’m not sure that LaTeX is likely to either: I think Hans and Taco will have considered this.

Joseph

Instead of writing LaTeX3 which is only another interface for unstructured, error prone input, we should provide XML interfaces for high quality typesetting. ConTeXt has (I’ve hardly used it) the ability to have XML-only documents. And I am currently writing an XML only typesetting system.

You are quite right about documentation. That seems to me to be something that the LaTeX team are aware of. The current “source3.pdf” file is a start (for the low-level stuff), but I’d agree that both LaTeX2e and ConTeXt suffer from the lack of an “official” book (or at least book-as-PDF) on the programming side of things.

I’d also agree that there will always be a somewhat blurred line when new functions open up new possibilities for design. On that, I have not idea how the team will tackle things.

For using LaTeX3, the only part that is currently usable is the low level programming stuff. That seems to be reasonably robust, and the team are working on an automated test system at the moment. But as yet there is not enough “on top” of that to see how things will work. For example, if you look at my xnotes2bib package (my notes2bib package re-implemented in LaTeX3), only the low-level stuff is in LaTeX3: the higher level stuff is still LaTeX2e.

The final point you make is of course the biggest issue of all. On that, “critical mass” is important, and it’s not clear that LaTeX3 can get that. But LaTeX2e can’t go on for ever. So at some point something has to give if the LaTeX team do not deliver (we all move to ConTeXt, we all have to use Word => Quark/InDesign, something else as yet unknown).

What is the distinction between functionality and design. Suppose I am writing a LaTeX class/ConTeXt module for a journal, which requires the user to specify keywords. Suppose the default class does not provide a keyword environment. Is adding keyword env part of the style or new functionality?

The thing that worries me most about LaTeX3 is how well it will be tested before it is released. ConTeXt has more of a rolling release philosophy, and right now there are quite a few users who are actually using Luatex+MKIV. So, bugs get caught and corrected. LaTeX3 needs to move to a stage where users can use it.

The other concern is whether the big TeX publishers (like IEEE, SIAM, Springer, AMS) change to LaTeX3 in their workflow. This is right now a problem with ConTeXt. Which means, even if I like ConTeXt, but know that the paper that I am working on will finally need to be submitted in LaTeX2e, I will use LaTeX2e. The same will be true of LaTeX3.

On the coding syntax/naming, I’d agree that including things about the arguments of a function in the function name is unusual. I was a bit sceptical myself at first, but I think this is useful due to the nature of the TeX language. An obvious example is expansion. With plain TeX, you need to do things like \def\MyMacro but \expandafter\def\csname MyMacro\endcsname to do a name definition. Using LaTeX3, the same is done with \def:Npn\MyMacro and \def:cpn{MyMacro}. Leaving aside the “pn” part (which may change), the \def:N versus \def:c idea is one I like.

Th point about a consistent language is that in LaTeX2e and ConTeXt you use a mix of TeX primitives and format macros to programme. The LaTeX3 idea is that *everything* is given a LaTeX interface (and name). The primitives tend to have syntax inconsistent with higher-level functions, and you are still left needing to learn two languages (TeX + LaTeX or TeX + ConTeXt).

On the \usemodule point, you are correct that I missed this possibility out. The LaTeX3 team are very keen to *really* separate functions and design. The problem with \usemodule is that there is no separation: a module can provide new functionality or alter design settings. LaTeX2e does better than LaTeX2.09 on this, but is still very poor. I’d expect LaTeX3 to do much better.

Writing the code is obviously a problem (after all, LaTeX3 was started before LaTeX2e came out!). I suspect that LaTeX3 will be bigger than ConTeXt (given the general “we need everything in the LaTeX Companion as a starting point” design brief).

On Lua, I’m not sure what will happen. At present there are no plans to require Lua for LaTeX3, at least partly because it will take years to propagate around. (I’ve recently had problems with a publisher where the e-TeX extensions are not available.) I’d think that this may change: I’d like to see Lua used if available (I’ve suggested that LaTeX3.5 might be an idea for that).

Finally, there are no guarantees for LaTeX3. At some point, if the team cannot deliver, people will move to ConTeXt, no question. I’m certainly trying to improve my ConTeXt skills.

1. It has a well designed and consistent document coding system. Now with lua in the picture, a lot of stuff is *much* easier. We may not need an explicit auxillary file anymore!

2. I doubt you can get a larger kernel than ConTeXt

3. ConTeXt does not plan to do this. I am not sure how useful it is. Most programming languages to not specify the airity of the function in the function name.

4. I guess this is true for almost all markup based language (except CSS, which is a mess).

5. ConTeXt already has excellent support for XML. Now all the XML parsing is done using lpeg (a lua) library, and you can specify both tree and stream based rules.

6. ConTeXt does grid layout and column sets. However, making any change in the output routine still requires a lot of work. (There is a module called streams, which makes things a bit easier. But changing the output routine is never going to be easy in ConTeXt).

I disagree with you on the aims of ConTeXt. You can have ConTeXt modules which are similar to LaTeX classes and packages. If you want a design for a journal, write a module for that. The same way you do in LaTeX. Ok, in LaTeX most of the time someone else has written the package for you. But someone has to do it. Someone will have to do it in LaTeX3.

A module can do any ad-doc thing that the module writer wants. Just like LaTeX packages. There are fewer ConTeXt modules, simply because Hans can code faster than you can request features.

I believe that the only thing that provides backward compatibility is plain TeX. If backward compatibility means that an old document compiles correctly, then to a large extent, both latex and context are b/c. If b/c means that you get the same output, the neither is. Sure, the latex kernel seldom changes, but you will alsways use some packages in latex, and packages change all the time. There goes b/c. In fact, Hans is very particular about b/c. Often there are ways to do things more cleanly, but he has refused to change many things saying that it would affect b/c.

It is really sad that LaTeX3 will not support lua. There are many deficiencies in TeX (e.g., automatic linebreaking of equations, syntax highlighing for verbatim text, sorting for bibliography, index, and table of contents) which are much easier to do in lua than in plain TeX. At least, once luatex is out, these things could be done via latex packages.

http://permalink.gmane.org/gmane.comp.tex.latex.latex3/1366

Here’s what I wrote:

* * *

Hi Joseph,

I think I’ve mentioned to you before that our thinking here is very
closely aligned; in my opinion you’ve done a good job covering the
“big vague issues” that I expect LaTeX3 to address in time.

The most interesting part for me is your point #2 — designing an
improved, consistent interface for functionality that spans what’s
described in The LaTeX Companion and then some.

I think I might have mentioned it before but I think it would be a
good idea to examine projects like Gellmu and see if we can coerce or
pervert the existing LaTeX document syntax into validating XML; I
believe that we can make it easier to write LaTeX by hand (since I
assume that people will always want to do this) by decreasing the
number of “special characters” and by eliminating the space-gobbling
macro that takes no arguments (which is useful for macro writers but
error-prone for document authors). (E.g., forcing users to write
something like `\LaTeX;` rather than `\LaTeX{}` or `\LaTeX\ ` or
`{\LaTeX}`.)

I don’t have much to say about LaTeX3 vs. ConTeXt besides the fact
that I’ve hardly used ConTeXt, and it does seem like many of the goals
of the LaTeX project have already been implemented in ConTeXt; it’s
possibly only critical mass that prevents it being a worthy successor
to LaTeX2e. On the other hand, I think it’s good for us to approach
similar problems independently to explore different possibilities. As
Joseph commented, I hope that ConTeXt Mk IV and LaTeX3 could be
complements rather than direct competitors to each other.

Staying on this point for a second, one big philosophical different
between the two is that once the LaTeX3 document model is designed it
will no doubt remain rigidly backwards compatible for many many years,
whereas ConTeXt may continue to evolve and continue to break backwards
compatibility to various degrees (although this may well not be the
case after Mk IV is complete).

Hypothetically speaking, I might expect a strict LaTeX3 document class
to be eligible for becoming an ISO standard, whereas ConTeXt is not
being designed, as far as I know, to naturally fit into this sort of
model of rigidity.

Meanwhile, better get back to work so that I can spend some other time
on the expl3 code so we can start working on all these things that we
keep talking about

Will

The question is, however, why reinvent the wheel – Hans has already written ConTeXt…

And more seriously: exactly as you stated, the real problem is: what are the design goals of LaTeX? Currently (as of 2e), LaTeX is fundamentally flawed: it *claims* to be a “document preparation system”, but in fact it *is* a “scientific paper preparation system”. It would be ok with me if it were clearly stated – but it’s not. Instead, LaTeX has numerous hacks and dirty tricks allowing it to produce other kinds of documents (mostly implemented as packages). This results in a feeling that “you can do everything in LaTeX, but you get a feeling that it’s somehow awkward”. Yes it is – because usually it’s just the wrong tool (unless you write a paper on maths, when it’s very nice).

But maybe I’m wrong. Maybe it would be refreshing to have some real competition. Maybe LaTeX3 will be redesigned so heavily that it will really be a “document preparation system”.

Remeber, however, that ConTeXt has two things LaTeX doesn’t (or am I wrong?): they have lua and apparently they have cured themselves from the “backward-compatibility-syndrome” (with MkIV). This might make ConTeXt a very tough competitor…