Some TeX Developments

A quick note to say that I see that Will Robertson has set up a blog ‘LaTeX Alive’. Will is the person behind the very useful fontspec package, which lets you choose system fonts easily when using one of the newer TeX engines (XeTeX and LuaTeX). Will is also very active in the LaTeX3 Project. I’ll be keeping an eye on his thoughts!

There have been a few queries recently about using my chemstyle package with XeTeX. The problems arise when people attempt to use the \schemeref macro, which is defined by the ‘low-level’ chemscheme package, which is loaded by chemstyle. For those of you not familiar with the package, it’s for chemistry graphics (which are usually called ‘schemes’), and is for putting in reference numbers automatically.

The problems arise because chemstyle ultimately relies on psfrag for the graphic manipulations. To the best of my knowledge, psfrag can’t be made to work with XeTeX, which means that neither can my \schemeref macro. I’ve just uploaded a version of chemstyle to CTAN which says this, and issues a warning if used with XeTeX. I do hope that helps a little.

I get a few e-mails about siunitx and the micro symbol. People tend to be surprised that the symbol ‘sticks’ to a look very much like Computer Modern. The reason is that picking a proper upright (not italic) μ is not so easy in TeX.You don’t get one in Computer Modern, so siunitx takes one from the TS1 (text support) set in the absence of a better plan. I’ve set up some auto-detection for a few obvious alternatives (such as the upgreek package), but that doesn’t really work for XeTeX users.

XeTeX users are likely to load system fonts, and I’d hope be using UTF-8 input. That makes it hard to auto-detect what they are doing, but should make life easier for them to get things right. A lot of more comprehensive fonts include Greek letters in the main font, so getting the μ right is simple:

\sisetup{
  mathsmu = \text{μ},
  textmu  = μ
}

or for people testing version 2 of siunitx:

\sisetup{
  maths-micro = \text{μ},
  text-micro  = μ
}

There may be a bit of testing required: this will not work if, for example, you are using the Latin Modern font.

Both MiKTeX and TeX Live have new versions in the offing. I’ve been testing out both MiKTeX 2.8 and TeX Live 2009, to keep up to date with what is happening. In the past, I’ve tended to stick with MiKTeX as it is designed for Windows, and so can make some platform-specific decisions and be more focussed. However, the TeX Live team have done a lot of work to make TeX Live usable across platforms, and there are advantages to that approach.

Looking through the feature lists, a lot of the new features are common to the two systems, for example:

• TeXworks installed as a distribution-maintained editor.
• XeTeX version 0.9995 (which includes the new primitives that the LaTeX3 team asked for).
• Some \write18 functions enabled without turning on full \write18 support: this is used to allow “safe” functions.

There are, of course, also differences. For example, only TeX Live includes LuaTeX at present. I also notice that MiKTeX 2.8 is adding the full path of files to the log, whereas in the past you got the relative path. I’m not so sure this is a good idea: it makes things rather wordy, and also the log will vary between systems: not so great. On the other hand, MiKTeX 2.8 does provide user-specific texmf directories. For multi-user systems, this is a real bonus: you can use the auto-install system without needing to be the Administrator.

As I said, I’ve tended to use MiKTeX to date as it’s been the best “fit” on Windows. The latest version of TeX Live makes this a pretty tight call, I think. If you are happy installing a full TeX system (which I do), then there is very little in it. MiKTeX still has the edge for small installations, as the auto-install system really pays off there.

The XeTeX page on the TUG website has been overhauled. It now has a set of useful links about XeTeX and using it. Well worth a visit. Unsurprisingly, there are a lot of XeLaTeX links, but there should be something for everyone. I particularly like the showcase: some of these would be very hard with traditional TeX!

Regular expressions are very popular as a quick and powerful way to carry out searches and replacements in text of all sorts. Traditionally, TeX handles tokens and not strings or characters. This means that doing regex searches using TeX82 is pretty much impossible. To solve this, recent versions of pdfTeX adds the \pdfmatch primitive to allow real string matching inside TeX. The LuaTeX team have decided not to take all of the existing “new” primitives forward from pdfTeX, and as I understand it \pdfmatch will not be implemented in LuaTeX. However, Lua itself has regular expression matching, and so the functionality will still be around.

I’ve recently talked about adding new primitives to XeTeX, and you’ll see that \pdfmatch was not on the list for adding to XeTeX. The reason is that a XeTeX implementation would have to be slightly different from pdfTeX, as it is natively UTF-8, but also would be different to LuaTeX, as it would still be a TeX primitive and not a Lua function. So here “the prize wasn’t worth the winning”, in my opinion. As it is, using \pdfmatch is not widespread, and the idea of having three different regex methods inside TeX didn’t seem like a great idea!

Talking of regex implementations, I’ve been reading Programming in Lua, and also working with TeXworks to try to get syntax highlighting the way I like it. Both systems are slightly different, and it seems both are different from the Perl implementation. It seems that every time you want to use a regex system you have to read the manual to see which things are different from every other implementation!

There has been a bit more work on the idea of adding primitives to XeTeX to match those available in pdfTeX.The list of pdfTeX primitives which look interesting has grown slightly, and now reads:

• \ifincsname
• \ifpdfprimitive
• \pdfprimitive
• \pdfshellescape
• \pdfstrcmp

At the same time, it would be useful to include the “extended” version of \vadjust which pdfTeX makes available. This is something that has been asked about before, and as with the rest of the changes the main issue is not the idea of doing it but the time for actual implementation.

The real need to have \pdfstrcmp available for LaTeX3 work means that some effort has actually gone into this. I’ve got no experience with either Pascal or the WEB format, but I’ve managed but dint of determination to get something passable to Jonathan Kew. There will need to be some adjustments, as XeTeX works with UTF-8 internally, which pdfTeX does not do. However, I’m hopeful that we will see new primitives in XeTeX soon.

Quite how the primitives will be named is still to be decided. The existing \pdf... naming does not really make sense with these non-PDF related functions. So they could end up as \XeTeX... or may just be given generic names. I’m leaving that to Jonathan!

XeTeX has, over the past few years, made using TeX with multiple fonts and UTF-8 input easy. The work-flow using XeTeX is very much more accessible than the routes needed using pdfTeX or TeX82. So I’m sure that many people, like me, use XeTeX whenever they want to use arbitrary fonts or to write anything which doesn’t use western European characters.

XeTeX is based on ε-TeX, which means it has a number of primitives which were not present in TeX82, but are present in ε-TeX itself or in pdfTeX (which also includes the ε-TeX primitives). However, ε-TeX was finalised over ten years ago, and since then the pdfTeX team have added a number of new primitives, many related directly to PDF output. At the same time, XeTeX includes its own new or extended primitive functions, in this case focussed on UTF input. For the most part this does not concern people as things work fine.

Recently, there has been some testing of the current LaTeX3 code with XeTeX (and older versions of pdfTeX, which don’t have all of the newer primitives). LaTeX3 requires the ε-TeX extensions, which are as I said available with any modern TeX engine. However, when it’s available LaTeX3 also uses the \pdstrcmp primitive: this is only present in newer versions of pdfTeX. For those people not familiar with \pdstrcmp, it allows you to do string comparisons of text (not token comparisons), and in an expandable manner. This is very useful, and much better than doing things without it; with no \pdfstrcmp, comparisons are not expandable. It became clear that there is a danger of some things working when using newer versions of pdfTeX, but failing with older ones or with XeTeX. Older versions of pdfTeX is one thing (the advice can simply be “sorry, you’ll have to update your pdfTeX”), but failing with XeTeX is simply no acceptable. After a bit of discussion, the best solution seemed to be to talk to Jonathan Kew about getting a very small number of “new” pdfTeX primitives into XeTeX.

At the moment, things are still under discussion, but the list of additional primitives is going to be small (somewhere between 2 and 5 seems likely). I think it’s giving nothing away to say that \pdfstrcmp is one that really is needed (although the name might be an issue!). Another likely candidate is \ifincsname, which looks handy and also not too complex to implement. There are a few other suggestions, but I’m not sure just yet what will be really needed, as opposed to nice to have. What is clear is that this is a one-off request. Once these small gaps are filled, LaTeX3 will not be using other primitives for general functions. I’m not sure how long it will take to finalise things, both for the team to agree on what is needed and for Jonathan Kew to do the hard work, but I’d imagine weeks not longer.

Testing the current LaTeX3 code means worrying about what primitives are available. Recent releases of pdfTeX have included a number of new primitives, and the expl3 system currently uses \pdfstrcmp if it finds it. This has already caused one issue, as things were broken if it was not available. So I’ve been modifying the test system a little, to run the tests with pdfLaTeX and XeLaTeX. The “new” primitives are not available with XeTeX, so this is a good test of our code: it’s supposed to work with both. I’ve found one odd thing going on with the LateX3 code, but also one thing that seems to be a XeTeX snag.

The snag can be tracked down (thanks to Morten Høgholm) to a minimal case:

\message{\ifcat\par\relax T\else F\fi}

If you try this in TeX or pdfTeX, you get T in the log (as The TeXbook says you should). With XeTeX, you get F. Whether this is a bug or a feature, I don’t know. I’m sure Jonathan Kew will elaborate, but it reminds me of the need to test using different set ups. You can imagine some very odd errors appearing with this type of thing!

There has been quite a bit of dicussion recently on the LuaTeX mailing list about primitive names. The basic set of primitive names provided by TeX has already been extended by e-TeX, pdfTeX and XeTeX. While e-TeX primitives have names which express only the function (such as \detokenize), both pdfTeX and XeTeX have tended to add the engine name to all new primitives. So for example pdfTeX provides the \pdfprimitive function, which always carries out the orginial function of a name, even if it has later been redefined:

\let\everypar\undefined
% \everypar{Some stuff} % Gives an error
\pdfprimitive\everypar{Some stuff}

will, for example, add material to \everypar despite the redefinition.

In LuaTeX, the idea of adding an engine-specific prefix has been abandoned. The example above shows one reason why: \pdfprimitive makes more sense with the name \primitive (which is what LuaTeX calls it). The problem with this is that name clashes are marginally more likely without a prefix (for example, \expanded is used in ConTeXt as a macro, but this is planned as a primitive for pdfTeX 1.50).

On the other hand, LuaTeX is a fundamental break with previous engines, as the internal changes mean that it may give different line breaks and so on to earlier systems. Thus the argument for new, logical, primitive names is that moving to LuaTeX is not the same as moving from TeX to pdfTeX or pdfTeX to XeTeX. There will be changes, and the user (or package developer) will need to think about them before diving in.

Overall, I think the LuaTeX idea is the best in the long term. Primitives which have well specified names help everyone, and some times change is necessary. I’ll be interest to see if the \directlua primitive for actually using Lua ends up with a shorter name (just \lua, anyone?).