Some TeX Developments

One issue a lot of people find confusing with (La)TeX is the rules about which types of graphic files work with which engines.  EPS files are fine when going via the DVI route, but do not work with direct PDF creation. The solution is to turn the EPS files in PDFs, and the problem goes away. However, there is then the question of how to do the conversion.

For most documents, having to convert every file by hand is not a sensible choice. The next nearest thing is the epstopdf package, which will do the same thing but from within a LaTeX run. However, it needs \write18 enabled, and this is not always desirable. More importantly, a lot of people who struggle with the graphics problem do not know how to turn on \write18 anyway.  A good way around has been added to the latest version of TeX Live, which is currently in the final testing stages. TeX Live 2009 has some restricted \write18 functions enabled as standard, and also has a version of epstopdf “built in”. The result is that EPS files are automatically converted to PDF files, in a transparent manner. Of course, this only happens if the PDF does not also exist! At the moment, this feature is not in MiKTeX 2.8, so it is one reason to favour TeX Live 2009 even on Windows.

There are places where epstopdf will not help: for example, when using psfrag or pstricks. There, the best solution will either be auto-pst-pdf or pstool. Both are written by Will Robertson, and both need \write18 enabled to work. pstool is more efficient (it only re-creates graphics as needed), but for some cases on auto-pst-pdt will work.  Will has documented both packages very well, so the best way to learn about them is to have a read of the documentation.

I’ve just send a new version of my notes2bib package to CTAN. notes2bib lets you include the text of notes in the body of a file, but have them appear in the bibliography: for chemists, this is pretty common. The new version is a re-working of the existing code plus some ideas I explored with an experimental version called xnotes2bib. I’ve re-jigged the options to make them more descriptive, and some of the macros have been renamed. My original choices were not always the best. The biggest change, though, is internally, where I’ve recoded everything using expl3, the coding base for LaTeX3. That means that users will need to install a couple of support packages, but I hope means that the code should make a bit more sense (at least to me!). The only way expl3 will get tested is if people use is, so I’m prepared to have a go and hope everything works. So far, all looks good.

Programming complex functions in TeX tends to involve a lot of manipulation of variables. At the low level, TeX doesn’t provide a lot of variable types, and so most things are done using macros. This means that a lot of array or record-type work is done by using large numbers of macros and giving them appropriate names. This is fast, as there is no need to recover data from a larger structure, but doesn’t make for easy to read code.

One aim of LaTeX3 is to provide better programming tools built on TeX, and this includes higher-level variable support. If you take a look at the expl3 manual, you’ll find that there is support for ideas such as property lists an sequence stacks. These are, of course, implemented using macro or token registers, but provide some of the more abstract ideas that other programming languages provide for variables. The pay-off is in speed: there will always be a cost to adding layers on top of the TeX basics.

I’ve been worrying about how to use the new structures as I work on siunitx version 2. My original code uses lots of macros to store things, but the result is that the code is difficult to read and even harder to change. My earlier attempts at improving the code stuck with this approach, but rationalised it into something more systematic. Better than version 1, but I’ve not been entirely happy with the results. I’m now trying again, this time using the high-level variables of LaTeX3 to do things. There is going to be a cost in speed, but I hope that making the code readable will be reward enough. I’m also spotting ways to improve the flow of code, so that there should be less to actually do in the new version. Looking to the future, making the code more maintainable should more than compensate for a little loss of speed. In the end, I’m sure I’ll be asked for more features, and having data structures I can rely on is essential for doing that.

If you take a look at the development version of siunitx, you will find that I’m getting on with moving the code over to using LaTeX3 coding internally.  That does not mean that the package will not work using LaTeX2ε: everything still works nicely in a normal LaTeX document! However, it does mean that I can use the new coding ideas to make my life easier, and hopefully the code more robust and a little faster.

I’ve also moved to using my own LaTeX3 keyval package for option handling. I recently added it to the main LaTeX3 code base, so I know that it will be available if LaTeX3 is! Following the pattern for other LaTeX3 keyval settings, I’m changing the original key naming plan somewhat. I’ll probably post more detail on this once things are working a little better. The idea is to use more descriptive key names that then current version of siunitx, for example:

\sisetup{
  group-digits         = true,
  round-mode           = places,
  round-places         = 3,
  separate-uncertainty = true
}

There will be a delay of a week as I’m on holiday, then I hope to get the experimental code doing almost everything that the current release version does. There is going to be a bigger delay as I sort out one idea that has been requested several times. I want to add some floating point stuff, but to do that I need to write support for that for LaTeX3. I have some ideas, but it will take a while to actually do the work.

I’ve been working for a while on a method to provide a reasonably powerful method for creating keyval input for LaTeX3. This has been going under the working title “keys3”, but yesterday I took the plunge and added the code to the LaTeX3 development repository. For the moment, this can only be accessed via SVN, but if the rest of the team are happy with the idea, it will appear in the next snap shot that is sent to CTAN.

The ideas in the new module (now called l3keys) have been inspired by the pgfkeys package. By using keyval methods to create keys, the idea is to make life a lot easier for the programmer. However, things are somewhat modified compared to the pgfkeys package, mainly to try to keep the input syntax simple but powerful enough for most uses. For example, there are separate functions for creating keys and setting them, an idea that all other keyval packages use.  So a typical setup block might look like:

\keys_define:nn { module } {
  key-one   .set:N     = \l_module_one_tl,   % Store input in variable
  key-one   .value_required:,
  key-two   .bool:N    = \l_module_two_bool, % Either true or false
  key-three .code:n    = { Code using an argument #1 },
  key-three .default:n = text
}

while keys are set using:

\keys_set:nn { module } {
  key-one = Value,
  key-two = true,
  key-three
}

Hopefully, the ideas are flexible and clear enough for people to get working with. One thing to notice: the key names are hyphenated. It looks like that will be the “style guide” suggestion for LaTeX3 keys. At some point, I’ll look at writing a TUGBoat article on how everything works.

I’ve been doing quite a bit of thinking about siunitx version 2. My original plan was to write everything in standard LaTeX2e code, but the more I thought about things the more it has not looked like such a good idea.  siunitx needs a lot of programming tools, and these are almost all available in the experimental LaTeX3 code which is now approaching stability. At the same time, there are still some difficult issues to solve for siunitx, and I’m going to need some more support stuff, I think. So I’ve revised my ideas on how to progress.

Currently, I’m moving the code I have already re-written to use LaTeX3 conventions internally. I’m then going to try to solve the remaining issues for version 2. The aim is to have a package which will run on LaTeX2e, using LaTeX3 internals, so that I can eventually create a LaTeX3-only edition with the same interface ideas. I’m going to try to crack on with things, but it will take a while to solve the remaining issues!

The next step for LaTeX3 development is to revise the two existing “xpackages” which are available to make a link between the code level and the user: xparse and template. Of the two, the xparse package is by far the easier to understand.

In LaTeX2e, you can either use the LaTeX method to create new commands:

\newcommand*\mycommand[2][]{%
  code goes here, using #1 and #2
}

or you can do things yourself using a mixture of  TeX primitives and LaTeX internal functions (and ignoring the issue of default value):

\def\mycommand{%
  \@ifnextchar[{%
    \@mycommand
  }{%
    \@mycommand[]%
  }%
}
\def\@mycommand[#1]#2{%
  code goes here, using #1 and #2
}

This does not make for code which is easy to alter to reflect different input syntax (for example XML), or to change internal functions without knowing how the user input works. The idea of xparse is to separate out the user syntax from the internal code. Currently, exactly what tools need to be available is still being decided. The current version of xparse would expect the following syntax:

\DeclareDocumentCommand \mycommand { o m } {
  code goes here, using #1 and #2
}

The idea is that each argument is represented by a letter, for example o for an optional argument, m for a mandatory one or s for an optional star. This makes it possible it see immediately from the definition how many arguments are needed (one for each letter). It also means that the internal functions, which implement things, can be separated totally from the user part of the system. In that way, internal functions can have a fixed number of arguments, and leave xparse to supply them. So if the internal function needs to be changed, it does not matter how it is used, or vice versa.

That all sounds very good, but there are some outstanding issues. For example, handling verbatim arguments is not straight-forward. There are a couple of possible approaches to this. Either stick to a simple system, and accept that not everything can be done in the same way, or make the system more flexible but complicated. I’m currently in favour of the first approach: almost every user function is simple (especially as we have the ε-TeX extensions and so can \scantokens our way out of a lot of problems). I’d be interested to hear what other people think would be useful.

The LaTeX3 Project (of which I’m a member) have made a snapshot release of the current LaTeX3 code to CTAN today. This is in two parts:

• expl3, the base layer for LaTeX3, which provides a consistent programming interface for LaTeX3 on top of TeX (or rather, on top of pdfTeX, XeTeX or LuaTeX).
• xpackages, higher level interface ideas for LaTeX3.

expl3 is now reasonably stable: the code has been carefully reviewed and the Team are happy that what is there works well. There are gaps, and some of these are definitely on the “to do” list. On the other hand, the xpackages bundle is not so close to being finished. At the moment, it includes only the parts of the experimental code that look the closest to reaching some kind of conclusion. However, it is also the part that is actively being reviewed at the moment. So I expect changes!

I’m hoping that as expl3 gets distributed around the TeX world it will be possible to consider using it for coding new packages. The ideas in expl3 make coding much easier, over all, even if initially there is a lot to learn. Of course, delivering more of LaTeX3 will help in getting people interested as well. That is something I’m definitely keen on.

Testing the current LaTeX3 code means worrying about what primitives are available. Recent releases of pdfTeX have included a number of new primitives, and the expl3 system currently uses \pdfstrcmp if it finds it. This has already caused one issue, as things were broken if it was not available. So I’ve been modifying the test system a little, to run the tests with pdfLaTeX and XeLaTeX. The “new” primitives are not available with XeTeX, so this is a good test of our code: it’s supposed to work with both. I’ve found one odd thing going on with the LateX3 code, but also one thing that seems to be a XeTeX snag.

The snag can be tracked down (thanks to Morten Høgholm) to a minimal case:

\message{\ifcat\par\relax T\else F\fi}

If you try this in TeX or pdfTeX, you get T in the log (as The TeXbook says you should). With XeTeX, you get F. Whether this is a bug or a feature, I don’t know. I’m sure Jonathan Kew will elaborate, but it reminds me of the need to test using different set ups. You can imagine some very odd errors appearing with this type of thing!

A while ago, I drafted an overview article on LaTeX3 for TUGBoat.  Since then, there have been some key changes to the current LaTeX3 code, and so I’ve slightly revised the article.  It is aimed as a very general overview of what is available at the moment, and is available as a PDF and in source form.