Some TeX Developments

I posted yesterday that v0.4.0 of TeXworks has been released: this marks a new stable branch in the code. I also pointed out that there are official Windows and Ubuntu builds. I’m now pleased to see that there is also an unofficial build for the Mac, at least for 64-bit Intel systems (like mine!). Thanks to Jjgod Jiang for doing this: I’ve posted before about the issues building TeXworks on the Mac. So I’m now happily using the first new Mac build for over a year.

TeX users on Linux tend to use the packaged TeX installation provided by there distro, rather than using the standard TeX Live installer and managing things themselves. Ubuntu is one of the more popular Linux distros, and there has been an issue for a while that they had stuck with TeX Live 2007, despite TeX Live 2008 and then TeX Live 2009 being released.

The latest Ubuntu release, 10.04 ‘Lucid Lynx’, finally moves to TeX Live 2009. This finally makes it easy to get TeX Live in a reasonable up to date version:

sudo apt-get install texlive

This grabs a subset of the complete TeX Live 2009, but seems to include quite a good selection (a bit like the MiKTeX basic installation on Windows, but I think a bit bigger). TeXworks is also on the list (at least the stable version), so if you don’t want to compile it from the source you can do

sudo apt-get install texworks

and be ready to go. Of course, you might still need to grab a few more bits of TeX Live (for example, XeTeX is not included in the standard selection). However, it’s definitely an improvement of the earlier situation.

I see on the TeXworks mailing list that Jonathan Kew has posted new builds of the ‘bleeding edge’ experimental code. For those of us who use them (and find them pretty stable) this will add a number of refinements, from the look of the change log.

TeXworks: Experimental LaTeX-errors interface

The latest builds of TeXworks include support for scripting. There are not a lot of scripts just yet, but one that already looks good is LaTeX error message highlighting.

This already seems to work pretty well, and it’s one of the few things I miss from my previous editor of choice (WinEdt). It can already jump to the line in question, when you choose the appropriate line in the listing.

Of course, this is only the first step in getting full scripting support for TeXworks. But things look good to me: the extra complexity to the interface is minimal, and it really opens up the possibilities for more advanced users.

Jonathan Kew has posted new binaries for TeXworks, with the idea that these are a release candidate for version 0.1 (which will be in TeX Live 2009 on Windows). Over the past few builds, we’ve got an icon, installer and the ability to associate .tex files with TeXworks. So things already look pretty good to me. I’m not sure what exactly is planned for v0.2, but somewhere along the line the plan is to include scripting, so perhaps that’s the next big item to add. I’m already using TeXworks for my day-to-day work, so don’t be put off by the low version number.

Both MiKTeX and TeX Live have new versions in the offing. I’ve been testing out both MiKTeX 2.8 and TeX Live 2009, to keep up to date with what is happening. In the past, I’ve tended to stick with MiKTeX as it is designed for Windows, and so can make some platform-specific decisions and be more focussed. However, the TeX Live team have done a lot of work to make TeX Live usable across platforms, and there are advantages to that approach.

Looking through the feature lists, a lot of the new features are common to the two systems, for example:

• TeXworks installed as a distribution-maintained editor.
• XeTeX version 0.9995 (which includes the new primitives that the LaTeX3 team asked for).
• Some \write18 functions enabled without turning on full \write18 support: this is used to allow “safe” functions.

There are, of course, also differences. For example, only TeX Live includes LuaTeX at present. I also notice that MiKTeX 2.8 is adding the full path of files to the log, whereas in the past you got the relative path. I’m not so sure this is a good idea: it makes things rather wordy, and also the log will vary between systems: not so great. On the other hand, MiKTeX 2.8 does provide user-specific texmf directories. For multi-user systems, this is a real bonus: you can use the auto-install system without needing to be the Administrator.

As I said, I’ve tended to use MiKTeX to date as it’s been the best “fit” on Windows. The latest version of TeX Live makes this a pretty tight call, I think. If you are happy installing a full TeX system (which I do), then there is very little in it. MiKTeX still has the edge for small installations, as the auto-install system really pays off there.

Jonathan Kew has produced new binaries for TeXworks for both both Windows and MacOS X. The ZIP file now contains everything you need to download in one place, which makes installation that bit easier.

On a somewhat related matter, I see that MiKTeX 2.8 is going to include TeXworks, including setting up file associations (.tex and .sty). As TeX Live 2009 is also coming with TeXworks, getting hold of the programme should become very easy for end users. This can only be a good thing.

Regular expressions are very popular as a quick and powerful way to carry out searches and replacements in text of all sorts. Traditionally, TeX handles tokens and not strings or characters. This means that doing regex searches using TeX82 is pretty much impossible. To solve this, recent versions of pdfTeX adds the \pdfmatch primitive to allow real string matching inside TeX. The LuaTeX team have decided not to take all of the existing “new” primitives forward from pdfTeX, and as I understand it \pdfmatch will not be implemented in LuaTeX. However, Lua itself has regular expression matching, and so the functionality will still be around.

I’ve recently talked about adding new primitives to XeTeX, and you’ll see that \pdfmatch was not on the list for adding to XeTeX. The reason is that a XeTeX implementation would have to be slightly different from pdfTeX, as it is natively UTF-8, but also would be different to LuaTeX, as it would still be a TeX primitive and not a Lua function. So here “the prize wasn’t worth the winning”, in my opinion. As it is, using \pdfmatch is not widespread, and the idea of having three different regex methods inside TeX didn’t seem like a great idea!

Talking of regex implementations, I’ve been reading Programming in Lua, and also working with TeXworks to try to get syntax highlighting the way I like it. Both systems are slightly different, and it seems both are different from the Perl implementation. It seems that every time you want to use a regex system you have to read the manual to see which things are different from every other implementation!

There is currently a very active thread on comp.text.tex about LuaTeX. Several related questions have come up, and it seems that some kind of summary would help me, at least.

Why did the LuaTeX team choose Lua?

Possibly the most frequently asked question with LuaTeX is why the team behind it chose Lua, as opposed to one of the other scripting languages available. The LuaTeX FAQ addresses this as question 1, so it’s obviously important. As I understand it, they looked at a number of possible languages for extending TeX, and decided that many of the “obvious” candidates (Java, Perl, Python, Ruby, Scheme) were not suitable. Not everyone is going to agree with that assessment, and some people are bound to feel that there was not sufficient consultation.

With LuaTeX already at version 0.40, it is progressing and will deliver. So unless a team can be got together to provided an alternative scripting system, it looks like Lua is what we will get. I don’t see there being enough experienced programmers with time on their hands to deliver a complete alternative at the moment.

How does LuaTeX relate to ConTeXt and LaTeX?

LuaTeX is an “engine”, like pdfTeX or XeTeX. So it provides certain functions, which might or might not get used. The end-user can use them directly, but this is really not helping most people as they don’t want to do this type of low-level work. So in the main it is down to either the TeX format (ConTeXt, LaTeX, …) or an add on (LaTeX package, ConTeXt module, etc.) to take advantage of them.

In the ConTeXt case, the entire format is being re-written as “Mark IV”, which will use a lot of Lua. This of course means things fundamentally change compared to early versions of ConTeXt, and ties it to a single engine.

In the LaTeX case, the format is written to work on plain TeX, no extensions at all. That is not about to change: essentially, LaTeX2e will never be “updated” in that sense. The current LaTeX3 plans don’t envisage requiring LuaTeX, although I’d hope that some low-level support will be included if LaTeX3 ever becomes a reality. There will, though, be LaTeX packages that use Lua: I’m sure that at some point there will be a fontspec-like package for LuaTeX, for example.

What is wrong with LaTeX2ε?

One issue that always comes up when future directions for TeX are discussed is whether LaTeX2ε needs to change, or whether it is okay as it is. Currently, if you know what you are doing then you can achieve a lot with LaTeX. There are a vast range of packages, and these cover very many of the things you could ever hope to do in TeX. So in a sense LaTeX works as it is. On the other hand, you have to know which packages to load, even to get some of the basics right (try making a new float type without loading any packages and using one of the base classes!). At the same time, things like UTF-8 text, system fonts and so on are not available using only the kernel.

A more fundamental issue is that LaTeX currently doesn’t do enough to provide structured data, and is not a good choice for dealing with XML input. This makes it hard to get data in and out, and is closely related to the fact that there is not enough separation of appearance and structure in the kernel and in add-on packages.

Neither of these points has escaped the notice of the LaTeX team. The question is whether a successor to LaTeX2ε will appear, and if it does whether it can succeed. There is a need to start from scratch with many things, meaning that a new LaTeX simply won’t work with most packages currently available. So the team have to deliver something that really works.

How can (La)TeX recruit new users?

One question that comes up here is what is a typical (La)TeX user anyway. Everyone has their own view on this: some people see TeX users mainly as mathematicians and physicists, other people point out that particularly with XeTeX available there is a lot of potential in the humanities.

Making life easier for new users is a priority for gaining new users. This means making it easy to install TeX (which both TeX Live and MiKTeX do, I think), making it easier to edit TeX files (the excellent TeXworks helps here), and making it easier to use TeX. It is, of course, the last one that is the problem. I think that we do need a new LaTeX as part of this. Removing the need to load a dozen packages and alter half a dozen internal macros to get reasonable output would benefit all LaTeX users. At the same time, something as simple as a generic thesis template could be made available: that would again help to “sell” LaTeX and by extension TeX as a whole.

LuaTeX has a role to play here. Using system fonts with pdfTeX is not easy, and XeTeX has helped a lot with this. LuaTeX provides this possibility and many more, and so we can hopefully look forward to not having to worry about loading system fonts at all. At the same time, it should be possible to do things similar to the current BibTeX and MakeIndex programs directly in the engine, but with full UTF-8 input. This is something that is long overdue, and again can’t hurt when it comes to selling (La)TeX.