Some TeX Developments

As a chemist, one of the things I want to do is track compound numbers (which are normally given as bold numbers, 1, 2, etc.). The traditional way to do that is by hand, which works but does require some concentration. Recent versions of ChemDraw have included an add-in for Word to do things automatically, and of course there is LaTeX support for the same idea.

In LaTeX there is a choice between two packages for tracking what is what. First, there is the bpchem package. It provides for the idea of subdivisions, so you can have 1a, 1b, 1c and so forth. However, I find the interface in bpchem is a bit awkward. The alternative is the chemcompounds package. It has a very easy to use approach to tracking, but does not have built-in support for subdivisions. So I’ve been working on how to achieve this easily in some stuff I’m writing at the moment. It turns out to be quite easy when you think about it.

The first stage is of course to load the package.

\usepackage[noimplicit]{chemcompounds}

I’ve decided to go with the option to turn off automatically creating new compound references, which means I have to declare each one separately. This requires a block of declarations in the preamble, but I actually find this easier than doing things ad hoc. The subdivisions I want are all about R groups (chemists will understand!). So I’ve started by setting up some simple R group letters (I have a family of compounds, and so it makes sense to use the same letter for the same R group in each case):

\declarecompound[a]{Mes}
\declarecompound[b]{iPr}

Hopefully you can see how this works: the optional argument sets up the label that will print, and the mandatory one is the label I’ll use to refer to the compound.
Then I need to set up the general compounds (the ones that will be 1, 2 and so on). I can let chemcompounds do the numbering, so this is easy:

\declarecompound{imidazole}
\declarecompound{pincer:salt}
\declarecompound{pincer:carbene}

The last stage in the preamble is to create the subdivided compounds. Rather than have to track the numbers and letter myself, I’ve found that I can simply refer back to the existing labels:

\declarecompound[\compound{imidazole}\compound{Mes}]
  {imidazole:Mes}
\declarecompound[\compound{imidazole}\compound{iPr}]
  {imidazole:iPr}
\declarecompound[\compound{pincer:salt}\compound{Mes}]
  {pincer:salt:Mes}
\declarecompound[\compound{pincer:salt}\compound{iPr}]
  {pincer:salt:iPr}
\declarecompound[\compound{pincer:carbene}\compound{Mes}]
  {pincer:carbene:Mes}
\declarecompound[\compound{pincer:carbene}\compound{iPr}]
  {pincer:carbene:iPr}

In the document body, things are now very easy. I just use the \compound macro. So for the general case I’ll have

\compound{imidazole}

(printing say 4) whereas for a single case I might have

\compound{imidazole:Mes}

(printing say 4a). This keeps my source easy to follow (I don’t have to remember numbers and letters, only labels), and avoids mistakes on my part.

I work hard on my LaTeX packages, and try to get things to work well for the user with clear interfaces. However, there is a down side to this: I get asked to do more things! For example, I’ve recently released version 2 of siunitx. This builds on the work from version 1, which itself was designed as an improvement to various earlier unit packages for LaTeX. I did remove a few features when I moved from version 1 to version 2, but in the main each new version of siunitx has added ‘more stuff’ to the package. However, I’ve also got organised with tracking bugs and feature requests using BitBucket. The resulting list of open issues is already quite frightening (at least for me). I’ve been keeping up with the bugs in the new release (I try to deal with them within a few days of being reported), so the list is all made up of feature requests, and almost all of those are new things. So the package being successful results in more work for me, not less. I’m not complaining: I want things to work. It’s just worth bearing in mind!

It’s well-known that TeX is good at integer arithmetic, and does not provide any primitive functions for real number calculations. Experienced TeX programmers will know that you can make use of dimen registers to do real number calculations at speed, at the cost of accuracy (as TeX truncates dimensions at five decimal places). However, this is not exactly ideal and can be a bit awkward to use.

An alternative for LaTeX users is to use the fp package (which has been around since LaTeX 2.09). This is a powerful package, but can be rather slow. Part of the reason is that it allows 18 digits either side of the decimal point: a wide range of numbers, but almost certainly overkill in most cases. For applications that really need large numbers (such as pgfplots) input using floating points (such as 1.23e20) is probably better. Floating point calculations make life a bit more complex again, but can be done in TeX (see for example pgfmath).

For LaTeX3 I’ve been working on what would be a sensible approach: real number functions are needed in various places. At the moment, the plan is to support fixed-point numbers with nine digits either side of the decimal place, so up to 999999999.999999999. That should be enough for most application, and at some stage supporting floating-point numbers might well be added. To date there is only basic arithmetic (add, subtract, multiply an divide) in the new code, but the plan is to add trigonometric and logarithmic functions. I’m sure that there will be other functions to add: I’ll be interested to see what is asked for.

One area that I’ve been working is overall performance compared to the fp package. The biggest single gain is of course moving from 18 plus 18 digits to 9 plus 9, which makes quite a big difference on it’s own. However, there are various places inside the code where there are opportunities to save time. First, as LaTeX3 requires ε-TeX I’ve exploited the \numexpr primitive where it makes a difference (mainly where it cuts down on the number of assignments needed). At the same time, there are places where using delimited macros is faster than actually doing mathematics! The exact performance gains depend on what exactly you are doing, but it is possible to draw some comparisons. Doing lots of repeated calculations it’s possible to get some feel for the difference between fp and the new LaTeX3 module. On my system 100 000 additions take 31.6 s with fp and 6.0 s with l3fp, while for 20 000 divisions it takes 64.8 s with fp and 4.1 s using l3fp. Quite some speed enhancement, and I think enough to justify using the new code!

Following the release of version 2 of siunitx, you might have noticed that there have been rapid minor updates (v2.0a, v2.0b and v2.0c). I thought I’d just say that most of this is to make sure that the version that ends up in TeX Live 2010 is as good as possible. There are the inevitable bugs to sort, especially with compatibility with version 1, and I want to get things working as well as possible. I suspect there will be a few more releases in quick succession!

A quick note to say that I see that Will Robertson has set up a blog ‘LaTeX Alive’. Will is the person behind the very useful fontspec package, which lets you choose system fonts easily when using one of the newer TeX engines (XeTeX and LuaTeX). Will is also very active in the LaTeX3 Project. I’ll be keeping an eye on his thoughts!

After many months of work, I’m pleased to announce that I’ve just sent version 2 of siunitx to CTAN. Many readers will be familiar with the package and some of the development process. Here, I’ve put together a summary as ‘release notes’ for the new version.

A comprehensive (SI) units package

Typesetting values with units requires care to ensure that the combined mathematical meaning of the value plus unit combination is clear. In particular, the SI units system lays down a consistent set of units with rules on how these are to be used. However, different countries and publishers have differing
conventions on the exact appearance of numbers (and units).

The siunitx package provides a set of tools for authors to typeset numbers and units in a consistent way. The package has an extended set of configuration options which make it possible to follow varying typographic conventions with the same input syntax. The package includes automated processing of numbers and
units, and the ability to control tabular alignment of numbers.

Version 2

Over the past two years siunitx has developed to include many features not originally foreseen when development began. While it has been possible to add a range of new features, some of the underlying limitation of the version 1 code have made this difficult. At the same time, renewed effort by the LaTeX Team on the development of LaTeX3, and in particular the expl3 programming system, has offered a more robust method to create the internal structure of siunitx. As a result, version 2 of siunitx has been almost completely re-written internally.

As well as fixing a number of bugs and limitations in the original release, version 2 is also much better written to work quickly. As a result, most users should see performance enhancements with this new release of siunitx.

As part of the revision of siunitx, the option system and user macros have been completely re-thought. The options now have longer, descriptive names and also a much clearer range of input values. The options which in version 1 took either a key word or a literal value have been replaced by ones which take literals only: in some cases this means that advice has been added to the documentation on how to get particular output effects.

Moving from version 1 to version 2

Depending on how you use siunitx, there may be very little to do to move to version 2. The new version includes a compatibility support file, meaning that loading siunitx using:

\usepackage[load-configurations = version-1]{siunitx}

should mean that existing documents compile with very few changes.

There are some changes to standard settings between version 1 and version 2, which may lead to some alterations in documents. At the same time, a small number of the features of siunitx version 1 which I feel did not work cleanly have been dropped. At present, some of these are scheduled to be re-examined for
inclusion in later releases of siunitx.

While there is a back-compatibility layer for users upgrading, it is strongly recommended that documents are updated to use the new option names and functions. The new approach has been chosen as it is an improvement on the previous version, and in the longer term this layer may be removed.

Installation

Most users will obtain siunitx as part of their TeX distribution. MiKTeX 2.8 should include siunitx version 2 after a short delay (a few days after CTAN upload). For TeX Live users, there will be a slight delay as the package will appear in updated form in TeX Live 2010 but not TeX Live 2009 (which is frozen).

For users who wish to install siunitx themselves, the package is available as a pre-extracted zip file, siunitx.tds.zip. Simply unzip this in your local texmf directory and run ‘texhash’ to update the database of file locations. Version 2 of siunitx requires up to date versions of the LaTeX3 packages expl3 and xpackages. These are also available from CTAN in ready to install format (as expl3.tds.zip and xpackages.tds.zip), and can be installed in the same way if necessary.

If you want to unpack the dtx yourself, running ‘tex siunitx.dtx’ will extract the package whereas ‘latex siunitx.dtx’ will extract it and also typeset the documentation. Typesetting the documentation requires a number of packages in addition to those needed to use siunitx. These should all be available in a complete TeX Live 2010 or MiKTeX 2.8 installation.

Development code and bug database

In order to help users see what is happening, and also to allow me to work efficiently, the development code for siunitx is available on the code hosting site BitBucket.

You can download the very latest code from there: of course, this may or may not work properly depending on exactly what I have added to the code.

The BitBucket site includes an issue tracker, where you can report bugs or make feature requests. I also add bugs to the database from e-mails I get from users. Filling in the bug database helps to make sure that I do not forget things, and also helps other users see what issues are known.

If you want to contribute code to siunitx, you can of course send patches directly to me. Alternatively, the code is hosted using the revision control system Mercurial, which was chosen as it is decentralised and is easy to install on a range of operating systems (I use MacOS X, Windows XP, Windows 7 and Ubuntu!). I’m happy to explain to potential contributors how Mercurial works for developing siunitx.

Roadmap for future releases

The bug database already includes a number of feature requests which are marked to be looked at for version 2.1. The current intention is that the next few months will be devoted to bug fixes in this release (v2.0), with moves to add features for v2.1 beginning in the late summer. I anticipate that v2.1 will be released toward the end of 2010.

It is likely that not all of the features currently marked as to be looked at for v2.1 will be fully working by the time it is released. At the same time, there are some longer term areas which may also need attention. Version 2.2 of siunitx is therefore planned, but with no current list of features marked for inclusion. This version is likely to appear in Spring 2011.

One longer term aim is to include LuaTeX support in siunitx, so that the entire package can work much more rapidly with LuaTeX than when using TeX macros alone. This is not likely to happen until next year (2011), but is in the bug database and is part of the longer term development plan for siunitx.

The internals of siunitx

Currently, the only documented interface to any of the functionality of siunitx is via the key-value control system and functions described in the manual. The internal code of the package is not documented, and there is therefore no guarantee of stability of internal functions. While it is common for users to have to modify the internals of LaTeX2e packages as part of their documents, this is not good programming practice and is not encouraged for siunitx, or indeed in general.

If there is a user function that you require that is not available using the documented tools, please either e-mail or report a bug in the database. One of the general aims of siunitx is to provide a proper documented interface for all of the
functions of the package. I am therefore very happy to add interfaces to internal processes as necessary.

Programmers should note that siunitx is coded using the LaTeX3 ‘expl3′ programming system. This looks somewhat different to traditional TeX or LaTeX programming. Details of the programming environment are documented as part of the expl3 bundle. Currently, none of the internal functions or interfaces are documented, and so are not meant for use outside of siunitx. Other programmers wanting to make use of internal siunitx functions are encouraged to get in contact with me. This will enable me to ensure that the parts of siunitx which are needed by others are documented and are not changed without consultation.

Following on from my last post, I thought it might be useful to give some simple example of how the xparse package works and why it’s useful. I want to do this to show end users of LaTeX how it can replace \newcommand, so the example will not involve anything too complex, code-wise.

First, why would you want to use xparse’s \NewDocumentCommand in place of \newcommand? First, \NewDocumentCommand can make macros that take a mixture of arguments that \newcommand cannot. With \newcommand, you can make a macro that takes a number of mandatory arguments, or ones where the first argument is option and in square brackets, but that is it. Anything else then needs the use of TeX programming or internal LaTeX macros: not really helpful for end users. The second thing is that \newcommand macros are not ‘robust’. This shows up where you need to \protect things, which can be very confusing. Macros created with \NewDocumentCommand are robust, and this means that they work more reliably.

I’m going to illustrate moving from \newcommand to \NewDocumentCommand with a series of simple examples. For all of them, you need to load the xparse package:

\usepackage{xparse}

Macros with no arguments

The simplest type of macro is one with no arguments at all. This isn’t going to show off xparse very much but is is a starting point. The traditional method to do this is

\newcommand\NoArgs{Text to insert}

which becomes

\NewDocumentCommand\NoArgs{}{Text to insert}

That does not look too bad, I hope. Notice that I’ve got an empty argument in the xparse case: this is where the arguments are listed, and with \NewDocumentCommand there always has to be a list of arguments, even if it is empty. That’s a contrast with the \newcommand approach, where we only need to mention arguments when there are any.

One or more mandatory arguments

The most common type of argument for a macro is a mandatory one. With \newcommand, we’d give a number of arguments to use:

\newcommand\OneArg[1]{Text using #1}
\newcommand\TwoArgs[2]{Text using #1 and #2}

\NewDocumentCommand is a bit different. Since it can work with different types of argument, each one is give separately as a letter. A mandatory argument is ‘m’, so we’d need

\NewDocumentCommand\OneArg{m}{Text using #1}
\NewDocumentCommand\TwoArgs{mm}{Text using #1 and #2}

This is still pretty similar to \newcommand: the useful stuff starts when life gets a little more complicated.

One of more optional (square brackets) arguments

To really get something clever out of xparse, the arguments need to be a little more varied than I’ve show so far. Let’s look at optional arguments, which LaTeX puts in square brackets. If I want the first argument to be optional, then LaTeX can help me

\newcomand\OneOptOfTwo[2][]{Text with #2 and perhaps #1}
\newcomand\OneOptOfThree[3][]{Text with #2, #3 and perhaps #1}

If I want anything else, I’m on my own (so no more \newcommand examples!). First, let’s do the two example using xparse. An optional argument in square brackets, which works like a \newcommand one, is ‘O’ followed by {}:

\NewDocumentCommand\OneOptOfTwo{O{}m}%
  {Text with #2 and perhaps #1}
\NewDocumentCommand\OneOptOfTwo{O{}mm}%
{Text with #2, #3 and perhaps #1}

How about two optional arguments? It’s pretty obvious:

\NewDocumentCommand\TwoOptOfThree{O{}O{}m}%
  {Text with #3 and perhaps #1 and #2}

What if we want something as a default value for the optional argument? With \newcommand, that would be

\newcommand\OneOptWithDefault[2][default]%
  {Text using #1 (could be the default) and #2}

which would become

\NewDocumentCommand\OneOptWithDefault{O{default}m}%
  {Text using #1 (could be the default) and #2}

The same idea applies to each optional argument: whatever is in the braces after the O is the default value.

More complex optional arguments

You might be wondering why we need the ‘{}c after ‘O’ when there is no default value: why not just ‘o’? Well, there is ‘o’ as well. Unlike \newcommand, \NewDocumentCommand can tell the difference between an option argument that is not given and one that is empty. To do that, it provides a test to see if the argument is empty:

\NewDocumentCommand\OneOptOfTwoWithTest{om}{%
  \IfNoValueTF{#1}
    {Do stuff with #2 only}
    {Do stuff with #1 and #2}%
}

Don’t worry if you forget to do the test: the special marker that is used here will simply print ‘-NoValue-’ as a reminder!

Two types of optional argument

Sometimes you might want two different optional arguments, and be able to tell which is which. This can be done by using something other than square brackets, often using angle brackets (‘<’ and ‘>’). We can do that using the letter ‘d’ (or ‘D’ if we give a default).

\NewDocumentCommand\TwoTypesOfOpt{D<>{}O{}m}%
  {Text using #1, #2 and #3}

What input syntax does this make? Let’s look at some examples

\TwoTypesOfOpt{text}             % One mandatory
\TwoTypesOfOpt[text]{text}       % A normal optional
\TwoTypesOfOpt<text>{text}       % A special optional
\TwoTypesOfOpt<text>[text]{text} % Both optionals

How did that work? The first two characters after the ‘D’ are used to find the optional argument, so in this case ‘<’ and ‘>’.

Finding stars or other special markers

Another common idea in LaTeX is to use a star to indicate some special version of a macro. Creating those with \newcommand is difficult, but it is easy with \NewDocumentCommand

\NewDocumentCommand\StarThenArg{sm}{%
  \IfBooleanTF#1
    {Use #2 with a star}
    {Use #2 without a star}%
}

Here, ‘s’ represents a star argument. You’ll see that it ends up as #1, while the mandatory argument is #2. You’ll also see that there needs to be a test to see if there is a star (\IfBooleanTF). This doesn’t mention stars as the test can be used for other things.

Summing up

There is more to xparse than I’ve mentioned here, but I hope that this is a useful flavour of what it can be used for. To get more flexibility there is a bit more to think about compared to \newcommand, but the overall consistency is hopefully worth it.

I’ve had a few chances in recent weeks to promote the xparse package. Regular readers will know that xparse is part of the efforts of the LaTeX3 Project, and is meant to provide a replacement for \newcommand, etc., for creating document macros. The promotions have come up where there are things that are hard to do with \newcommand but easy with xparse. One example was creating an optional argument for a macro but giving it in normal braces. With xparse, this is pretty straight forward

\documentclass{article}
\usepackage{xparse}
\NewDocumentCommand\en{g}{%
  \IfNoValueTF{#1}{\epsilon}{\epsilon_{#1}}%
}
\begin{document}
\( \en \) and \( \en{stuff} \).
\end{document}

This is clear, and does not need any knowledge of the internals of the LaTeX3 approach. So I’m going to keep promoting xparse for day to day LaTeX users, even if they aren’t using much code. Hopefully this will make life easier for them, and for me, and will be a real benefit now from the LaTeX3 work.

Over the past few months I’ve been working a new version of siunitx with completely re-written internals. This is now at the point where I hope that it is usable for most people, but before replaced version 1 some testing is needed. This is what this beta (testing) release is for. There are some important notes for people testing, which I’ll run through below. For the impatient, you can get:

• A ready to install .zip file
• The documentation
• The source (dtx) file

Release notes

The code used in siunitx relies on the LaTeX3 Project packages expl3 and xpackages. You will need the latest versions of both of these to test siunitx 2: they can both be downloaded from CTAN or installed using the update facilities in TeX Live or MiKTeX.

Version 2 of siunitx renames most of the package options to make them more informative. The old names are available by using:

\usepackage[load-configurations = version-1]{siunitx}

The new option names are intended to make it easier to continue to expand siunitx without having completely opaque option names. At the same time, the nature of some of the options has been changed. This means that there are no longer any ‘magic’ keywords, which have caused confusion in the past.

There will be no significant features added to siunitx version 2 between the beta version and the production release (probably in June). The aim is to get the inevitable bugs in the current code found and fixed, which is best done while not making other changes.

A small number of features from version 1 of siunitx are not present in version 2. The features removed have never worked as well as I would like, and so I felt it was better to remove them and rework them later if needed. If this causes severe problems for users then some of these decisions may of course be reversed.

I have had a large number of feature requests for siunitx, and only some of these have been added to version 2 at present. This is partly as I have a limited amount of time, and need to get siunitx 2 to release within a reasonable time. At the same time, some of the feature requests are very specialist and I need to consider which of these fall within the scope of the package. That said, I intend to work on adding more features to siunitx after the full release of version 2.0. More details on this will be posted here in the future.

Feedback on any aspect of siunitx version 2 is very welcome: joseph.wright@morningstar2.co.uk.

As many readers will know I’ve been working on version 2 of siunitx for some time. There are always more feature requests, but at some stage I have to actually release something. I’ve now got some code that fixes a lot of bugs and annoyances in the current release and provides a better platform for new features, and so I want to move to releasing it.

The plan is to fix the known bugs in the development code and add as many new features as possible by the 25th of the month, and then to freeze development for a beta (testing) release. Depending on feedback, I’ll then aim to fix the bugs in the beta and go for a first production edition of version 2 in June. I’m not going to add any new features between the beta and production releases, but will start on them again once the production version is out. There is a feature tracker on the BerliOS site, which is a good place to put things so I don’t forget about them.