Some TeX Developments

Coding in the TeX world

Lua, TeX, LaTeX and ConTeXt

with 14 comments

There is currently a very active thread on comp.text.tex about LuaTeX. Several related questions have come up, and it seems that some kind of summary would help me, at least.

Why did the LuaTeX team choose Lua?

Possibly the most frequently asked question with LuaTeX is why the team behind it chose Lua, as opposed to one of the other scripting languages available. The LuaTeX FAQ addresses this as question 1, so it’s obviously important. As I understand it, they looked at a number of possible languages for extending TeX, and decided that many of the “obvious” candidates (Java, Perl, Python, Ruby, Scheme) were not suitable. Not everyone is going to agree with that assessment, and some people are bound to feel that there was not sufficient consultation.

With LuaTeX already at version 0.40, it is progressing and will deliver. So unless a team can be got together to provided an alternative scripting system, it looks like Lua is what we will get. I don’t see there being enough experienced programmers with time on their hands to deliver a complete alternative at the moment.

How does LuaTeX relate to ConTeXt and LaTeX?

LuaTeX is an “engine”, like pdfTeX or XeTeX. So it provides certain functions, which might or might not get used. The end-user can use them directly, but this is really not helping most people as they don’t want to do this type of low-level work. So in the main it is down to either the TeX format (ConTeXt, LaTeX, …) or an add on (LaTeX package, ConTeXt module, etc.) to take advantage of them.

In the ConTeXt case, the entire format is being re-written as “Mark IV”, which will use a lot of Lua. This of course means things fundamentally change compared to early versions of ConTeXt, and ties it to a single engine.

In the LaTeX case, the format is written to work on plain TeX, no extensions at all. That is not about to change: essentially, LaTeX2e will never be “updated” in that sense. The current LaTeX3 plans don’t envisage requiring LuaTeX, although I’d hope that some low-level support will be included if LaTeX3 ever becomes a reality. There will, though, be LaTeX packages that use Lua: I’m sure that at some point there will be a fontspec-like package for LuaTeX, for example.

What is wrong with LaTeX2ε?

One issue that always comes up when future directions for TeX are discussed is whether LaTeX2ε needs to change, or whether it is okay as it is. Currently, if you know what you are doing then you can achieve a lot with LaTeX. There are a vast range of packages, and these cover very many of the things you could ever hope to do in TeX. So in a sense LaTeX works as it is. On the other hand, you have to know which packages to load, even to get some of the basics right (try making a new float type without loading any packages and using one of the base classes!). At the same time, things like UTF-8 text, system fonts and so on are not available using only the kernel.

A more fundamental issue is that LaTeX currently doesn’t do enough to provide structured data, and is not a good choice for dealing with XML input. This makes it hard to get data in and out, and is closely related to the fact that there is not enough separation of appearance and structure in the kernel and in add-on packages.

Neither of these points has escaped the notice of the LaTeX team. The question is whether a successor to LaTeX2ε will appear, and if it does whether it can succeed. There is a need to start from scratch with many things, meaning that a new LaTeX simply won’t work with most packages currently available. So the team have to deliver something that really works.

How can (La)TeX recruit new users?

One question that comes up here is what is a typical (La)TeX user anyway. Everyone has their own view on this: some people see TeX users mainly as mathematicians and physicists, other people point out that particularly with XeTeX available there is a lot of potential in the humanities.

Making life easier for new users is a priority for gaining new users. This means making it easy to install TeX (which both TeX Live and MiKTeX do, I think), making it easier to edit TeX files (the excellent TeXworks helps here), and making it easier to use TeX. It is, of course, the last one that is the problem. I think that we do need a new LaTeX as part of this. Removing the need to load a dozen packages and alter half a dozen internal macros to get reasonable output would benefit all LaTeX users. At the same time, something as simple as a generic thesis template could be made available: that would again help to “sell” LaTeX and by extension TeX as a whole.

LuaTeX has a role to play here. Using system fonts with pdfTeX is not easy, and XeTeX has helped a lot with this. LuaTeX provides this possibility and many more, and so we can hopefully look forward to not having to worry about loading system fonts at all. At the same time, it should be possible to do things similar to the current BibTeX and MakeIndex programs directly in the engine, but with full UTF-8 input. This is something that is long overdue, and again can’t hurt when it comes to selling (La)TeX.

Written by Joseph Wright

May 21st, 2009 at 8:06 am

Posted in General

Tagged with , , , ,

14 Responses to 'Lua, TeX, LaTeX and ConTeXt'

Subscribe to comments with RSS or TrackBack to 'Lua, TeX, LaTeX and ConTeXt'.

  1. Your approach described in the c.t.t. i.e., gradually adopting LuaTeX is most viable compared to the rest. I think what’s so frustrating about LaTeX3 is that there is not even experimental releases where with ConTeXt and LuaTeX is doing this on a regular basis. Users are excited about it. I know a few people even typeset their PhD thesis in it despite its beta status.

    Leo

    21 May 09 at 6:10 pm

  2. There is experimental LaTeX3 stuff on CTAN, but at the moment what you won’t find is a format you can actually use. Why? There isn’t one! I think we probably need something, even if “Hello World” is the limit of its abilities. Something for me to pursue over the coming weeks, perhaps.

    Joseph Wright

    21 May 09 at 8:31 pm

  3. I also do not like Lua and think the LuaTeX team chose it because they just liked lua. I asked Jonathan Kew if perhaps he is going to put a programming language in XeTeX like what luatex does and he said lua is not my favourite programming language too but for TeX, the best language to use is Lua.

    I just had a question about LaTeX3. In your very recent post, you talk about LaTeX3 in a way that audience will feel latex3 may not even appear as the next version of latex. is that true? I am personally a Plain TeX user because I have got used to it and it is heaps finer than ConTeXt and LaTeX but for the latex packages that I have, I do some LaTeX programming so it would be very good if the current packages can work with LaTeX3 as we have heaps of latex2e packages and rewriting them based on latex3 is not a good idea at all. It is true that ConTeXt is a modern macro package comparing to latex but I think latex is more popular. There are heaps of mathematicians who do not care and do not have time to follow TeX developments, they are using latex for a long time and they are still happy with it and they would not change to ConTeXT because that means another learning curve.

    Vafa Khalighi

    22 May 09 at 6:04 am

  4. Hi Joseph,

    Thank you for your response. I agree with you. With that and more regular releases, then it will give people the impression LaTeX3 is alive and a user community will follow. Unlike in the 80s, these days users have many options to choose their software toolchain, and most people tend to move towards active ones for one thing better support.

    BTW, the LuaTeX team was able to secure funding for their development. What’s the situation with LaTeX3?

    Has this Initial LaTeX3 microkernel (http://www.tug.org/gsoc/) been accepted?

    Leo

    Leo

    22 May 09 at 11:43 am

  5. Vafa,

    Sorry if I come across as negative. The point I’m aiming at is that *at the moment* there is no experimental LaTeX3 format as such. There are a number of packages which the team are trying to build on. I hope this will lead to a working LaTeX3, but at the moment things are not in the same situation as say LuaTeX or ConTeXt Mark IV. They both have a lot of material already done, and we can all be confident they will deliver. I working on LaTeX3 with the exception it will work: otherwise I’d rather be wasting my time. I just can’t say *today* that things are as clear as they are for other current projects. If things go the way I hope, my assessment in 6 months to a years time will be different.

    Joseph Wright

    22 May 09 at 12:41 pm

  6. Leo,

    The LaTeX3 fund certainly contains some money, but as I understand it not at the level where it would pay for someone to work full time on LaTeX3. LuaTeX has gained funding from various sources, I think, with lots of small parcels adding up to a reasonable amount. Even if there were the money, I’m not sure if there is anyone “available” to develop LaTeX3 full time at the moment!

    On my microkernel idea, TUG was not picked by Google this year. There are some issues with getting something together, but I’d hope that if we can get the next phase of LaTeX3 sorted then we may be in a position to produce a kernel. It won’t do much to start with!

    Joseph Wright

    22 May 09 at 12:44 pm

  7. @Leo, if read my argumentation on c.t.t. for making LuaTeX an explicit dependency for LaTeX3, you will see that this is not about coding in Lua at all. I give more explanation in the thread but in short, if you release LaTeX 3.0 with minimal requirements of pdfTeX, then you make a commitment to support that even if you decide to switch to LuaTeX in the future. I see this as a decision to accumulate legacy baggage that will only drag on you later on. This is not the end of the world, but I think it’s good to realize that fact.

    In the context of what’s wrong with LaTeX2e I think that by far the bigest problem is that LaTeX2e as a document delivery format is not structured enough and too much tied to TeX. This lies at the base of many other problems, one of them is getting content out of LaTeX sources. The example given at c.t.t. of “Optics Express” journal urging users to switch to Word is just frightening. It’s a clear sign that LaTeX is on its way out from the publishing industry. Once that happens, it will not make its way back no matter what. That would be a truly serious blow to the LaTeX community.

    Tomek

    22 May 09 at 2:01 pm

  8. Joseph,

    The situation with LaTeX3 now reminds me of GNU hurd http://www.gnu.org/software/hurd/hurd.html. It is been under development longer than LaTeX3 and now is almost completely lost with the emerging of Linux ;)

    Leo

    Leo

    22 May 09 at 6:12 pm

  9. Tomek,

    Which engine latex3 is based on is invisible to users. I think it is more important to get something out of the door. I am only interested in that because I am not technical enough to speak about the merits of each engine although I sense LuaTeX is a good candidate.

    Leo

    Leo

    22 May 09 at 6:22 pm

  10. Leo,

    I can see the HURD analogy, but things are a bit different there as there is Linux. The comparison doesn’t quite add up in the TeX world: ConTeXt is more complementary to LaTeX than Linux is to HURD.

    I’d also agree that getting something out of the door is the priority. That doesn’t mean that engine ideas are not worth discussion. Hence my suggestion that a planned move to newer engines “in the future” is the best compromise: deliver something, then move over time but with this flagged in advance.

    Joseph Wright

    23 May 09 at 8:55 am

  11. Dealing with xml input is one of the places where lua really shines (compared to TeX82). You can easily write parsers in lua. ConTeXt already has two different parsers for xml, and it is relatively straight forward to typeset xml documents directly. Some users use xml as the input format, and some are typing to write a style for the xml generated by open office.

    Aditya Mahajan

    24 May 09 at 7:08 am

  12. XML input in not such a problem. It is the other way around that gives a lot of headaches. Some people advocate XML as an alternative/successor to LaTeX, but its verbosity is a show stopper for many.

    As a side note, one interresting example w.r.t. XML verbosity is RELAX NG:

    http://en.wikipedia.org/wiki/RELAX_NG

    It has a compact syntax that is isomorphic to its XML representation.

    Tomek

    24 May 09 at 10:32 am

  13. I’ve always thought that XML is fine as an exchange format, but that I’m not sure about it as an input format for end users. The LaTeX input format is much “nicer” to write, I’d say. So I’d agree that the challenge is to define a version of LaTeX which works for LaTeX and is easy to convert to well-formed XML. I’d guess that needs to be done using LaTeX itself, suggesting some kind of LaTeX-to-XML two-way functionality for LaTeX3.

    Joseph Wright

    26 May 09 at 6:03 pm

  14. […] Wollmilchsau” one could say in German (for a collection of advantages of that approach see here). On the professional level that means that the rebuild engine could have been made much more […]

Leave a Reply