Some TeX Developments

After many months of work, I’m pleased to announce that I’ve just sent version 2 of siunitx to CTAN. Many readers will be familiar with the package and some of the development process. Here, I’ve put together a summary as ‘release notes’ for the new version.

A comprehensive (SI) units package

Typesetting values with units requires care to ensure that the combined mathematical meaning of the value plus unit combination is clear. In particular, the SI units system lays down a consistent set of units with rules on how these are to be used. However, different countries and publishers have differing
conventions on the exact appearance of numbers (and units).

The siunitx package provides a set of tools for authors to typeset numbers and units in a consistent way. The package has an extended set of configuration options which make it possible to follow varying typographic conventions with the same input syntax. The package includes automated processing of numbers and
units, and the ability to control tabular alignment of numbers.

Version 2

Over the past two years siunitx has developed to include many features not originally foreseen when development began. While it has been possible to add a range of new features, some of the underlying limitation of the version 1 code have made this difficult. At the same time, renewed effort by the LaTeX Team on the development of LaTeX3, and in particular the expl3 programming system, has offered a more robust method to create the internal structure of siunitx. As a result, version 2 of siunitx has been almost completely re-written internally.

As well as fixing a number of bugs and limitations in the original release, version 2 is also much better written to work quickly. As a result, most users should see performance enhancements with this new release of siunitx.

As part of the revision of siunitx, the option system and user macros have been completely re-thought. The options now have longer, descriptive names and also a much clearer range of input values. The options which in version 1 took either a key word or a literal value have been replaced by ones which take literals only: in some cases this means that advice has been added to the documentation on how to get particular output effects.

Moving from version 1 to version 2

Depending on how you use siunitx, there may be very little to do to move to version 2. The new version includes a compatibility support file, meaning that loading siunitx using:

\usepackage[load-configurations = version-1]{siunitx}

should mean that existing documents compile with very few changes.

There are some changes to standard settings between version 1 and version 2, which may lead to some alterations in documents. At the same time, a small number of the features of siunitx version 1 which I feel did not work cleanly have been dropped. At present, some of these are scheduled to be re-examined for
inclusion in later releases of siunitx.

While there is a back-compatibility layer for users upgrading, it is strongly recommended that documents are updated to use the new option names and functions. The new approach has been chosen as it is an improvement on the previous version, and in the longer term this layer may be removed.

Installation

Most users will obtain siunitx as part of their TeX distribution. MiKTeX 2.8 should include siunitx version 2 after a short delay (a few days after CTAN upload). For TeX Live users, there will be a slight delay as the package will appear in updated form in TeX Live 2010 but not TeX Live 2009 (which is frozen).

For users who wish to install siunitx themselves, the package is available as a pre-extracted zip file, siunitx.tds.zip. Simply unzip this in your local texmf directory and run ‘texhash’ to update the database of file locations. Version 2 of siunitx requires up to date versions of the LaTeX3 packages expl3 and xpackages. These are also available from CTAN in ready to install format (as expl3.tds.zip and xpackages.tds.zip), and can be installed in the same way if necessary.

If you want to unpack the dtx yourself, running ‘tex siunitx.dtx’ will extract the package whereas ‘latex siunitx.dtx’ will extract it and also typeset the documentation. Typesetting the documentation requires a number of packages in addition to those needed to use siunitx. These should all be available in a complete TeX Live 2010 or MiKTeX 2.8 installation.

Development code and bug database

In order to help users see what is happening, and also to allow me to work efficiently, the development code for siunitx is available on the code hosting site BitBucket.

You can download the very latest code from there: of course, this may or may not work properly depending on exactly what I have added to the code.

The BitBucket site includes an issue tracker, where you can report bugs or make feature requests. I also add bugs to the database from e-mails I get from users. Filling in the bug database helps to make sure that I do not forget things, and also helps other users see what issues are known.

If you want to contribute code to siunitx, you can of course send patches directly to me. Alternatively, the code is hosted using the revision control system Mercurial, which was chosen as it is decentralised and is easy to install on a range of operating systems (I use MacOS X, Windows XP, Windows 7 and Ubuntu!). I’m happy to explain to potential contributors how Mercurial works for developing siunitx.

Roadmap for future releases

The bug database already includes a number of feature requests which are marked to be looked at for version 2.1. The current intention is that the next few months will be devoted to bug fixes in this release (v2.0), with moves to add features for v2.1 beginning in the late summer. I anticipate that v2.1 will be released toward the end of 2010.

It is likely that not all of the features currently marked as to be looked at for v2.1 will be fully working by the time it is released. At the same time, there are some longer term areas which may also need attention. Version 2.2 of siunitx is therefore planned, but with no current list of features marked for inclusion. This version is likely to appear in Spring 2011.

One longer term aim is to include LuaTeX support in siunitx, so that the entire package can work much more rapidly with LuaTeX than when using TeX macros alone. This is not likely to happen until next year (2011), but is in the bug database and is part of the longer term development plan for siunitx.

The internals of siunitx

Currently, the only documented interface to any of the functionality of siunitx is via the key-value control system and functions described in the manual. The internal code of the package is not documented, and there is therefore no guarantee of stability of internal functions. While it is common for users to have to modify the internals of LaTeX2e packages as part of their documents, this is not good programming practice and is not encouraged for siunitx, or indeed in general.

If there is a user function that you require that is not available using the documented tools, please either e-mail or report a bug in the database. One of the general aims of siunitx is to provide a proper documented interface for all of the
functions of the package. I am therefore very happy to add interfaces to internal processes as necessary.

Programmers should note that siunitx is coded using the LaTeX3 ‘expl3′ programming system. This looks somewhat different to traditional TeX or LaTeX programming. Details of the programming environment are documented as part of the expl3 bundle. Currently, none of the internal functions or interfaces are documented, and so are not meant for use outside of siunitx. Other programmers wanting to make use of internal siunitx functions are encouraged to get in contact with me. This will enable me to ensure that the parts of siunitx which are needed by others are documented and are not changed without consultation.

I’ve been working on siunitx version 2, getting the code to the point where I’d hope it works for most things that the current release does. Over the last couple of weeks I hope I’ve sorted out the table issues which tend to be a problem in version 1, plus “mopped up” a few outstanding odds and ends. So it feels like an appropriate point for a public snap shot of the code. As progress has been good, I’m calling this one alpha 1.

At this stage, there are likely to be some bugs and other annoyances in the code. However, I hope that enough works for people to risk taking the plunge and trying it out. For those willing to try out siunitx v2.0, you can get:

• The ready to install TDS-style zip file
• The package documentation
• The source file (dtx)

You’ll need to have up to date installations of both expl3 and the xpackages to try out the code, as internally the new code uses the LaTeX3 internal syntax. The biggest change that users should see from version 1 is that I’ve re-thought the option names. They are mainly longer, but more informative, in the new code. Improvements to the names I’ve picked are of course welcome.

At this stage, I’m still working on adding more ideas to the code. So there are some omissions in the release that I know are there and am intending to sort out. Of course, different users have different priorities for improvement. That said, there are bound to be things which simply are broken, things I’ve forgotten and the odd item that I’m currently planning not to carry forward from version 1. Feedback by as comments here, by e-mail or at the BerliOS site is very welcome.

The current development plan is to see how much is wrong with the alpha code, and if necessary to have some more alpha-status snap shots. I’d then hope to have a beta some time in the spring (perhaps April or May), with a full release currently in my mind for June or July. Exactly how this will work out depends on other projects: I’d like, for example, to have some floating-point tools in siunitx, but for that I need to write them for LaTeX3. The feature list for the release certainly isn’t fixed, and I’d expect that once v2.0 is out there will continue to be more ideas to add on.

My previous post mentioned some efforts to improve the performance of the siunitx parser. I’ve now committed an entirely new version of the parsing code to the repository. I’ve also done my best to speed up the rest of the package. The speed you see very much depends on the type of input involved. With my test file, I got a time of roughly 3 minutes using version 1 of siunitx, about 2 minutes before improving version 2, and about 1.5 minutes after. For comparison, doing no processing at all take the time down below 10 seconds for the same file (roughly 700 pages of repeated input). For the interested reader, an siunitx snapshot TDS-style zip is available.

One thing that I find interesting in all of this is that even before optimising the code, version 2 still worked faster than version 1, even though it does more things. A lot of that is because the pre-built looping material in expl3 does a much better job than my own attempts in version 1 of siunitx. Of course, good programmers will always use fast loops, but for the rest of us I think this shows how a sensible tool-kit can bring benefits “behind the scenes”.

I had an e-mail today about using siunitx when there are a lot of calls to the package. As you might expect, things can get a bit slow, and the person who contacted me felt that things get rather too slow. There are differences between the current release version of siunitx and the development code (version 2), and I’ve also added a few features to help speed things up where appropriate using version 2. So I thought I’d put a bit of information on the comparison in the public domain.

First, a baseline is not to use siunitx at all, and to simply test everything by hand. For that, I tried the simple test file:

\documentclass{article}
\usepackage{siunitx,xparse}
\ExplSyntaxOn
\DeclareDocumentCommand \repeated { m m }{
  \prg_replicate:nn {#1} {#2}
}
\ExplSyntaxOff
\begin{document}

\repeated{10000}{$1.23\,\text{m}$ }

\end{document}

This repeats the same text 10 000 times: boring but handy for testing. Using the command-line time program, I get an overall time of 1.714 s for this.

A very slight change of the file lets me test with siunitx version 1:

\documentclass{article}
\usepackage{siunitx,xparse}
\ExplSyntaxOn
\DeclareDocumentCommand \repeated { m m }{
  \prg_replicate:nn {#1} {#2}
}
\ExplSyntaxOff
\begin{document}

\repeated{10000}{\SI{1.23}{\metre} }

\end{document}

With the latest release version of siunitx (v1.3g), I get a time of 80.878 s for this on the same system.

In siunitx version 2, I’ve recoded all of the loops and parsing code, and so things are faster using the standard settings: 58.944 s. With the very latest code (SVN 243), I’ve included two options to make things move faster: parse-numbers and parse-units. Of course, these do mean that you get less of the power of siunitx. But for many people they might be useful. Turning both parsing systems off, the time needed for the test file drops to 14.975 s (just turning off the number parser gives a time of 18.803 s).

I may take another look at trying to improve the performance of the number parser. The problem is at least in part that making the code faster will either mean making some of it less powerful or, more likely, a lot harder to read and maintain. I hope that for most people, most of the time, the performance is acceptable. Of course, at some point I’ll try to do some Lua-based code for the parsers, at least. But that won’t help for most users now.

Ultimately, there is a limit to how fast things can work. Whether the performance hit of using siunitx is worthwhile is something is down to users. I think it’s worth it, as the better logic in the mark-up more than makes up for the extra time required. But then I would say that!

I get a few e-mails about siunitx and the micro symbol. People tend to be surprised that the symbol ‘sticks’ to a look very much like Computer Modern. The reason is that picking a proper upright (not italic) μ is not so easy in TeX.You don’t get one in Computer Modern, so siunitx takes one from the TS1 (text support) set in the absence of a better plan. I’ve set up some auto-detection for a few obvious alternatives (such as the upgreek package), but that doesn’t really work for XeTeX users.

XeTeX users are likely to load system fonts, and I’d hope be using UTF-8 input. That makes it hard to auto-detect what they are doing, but should make life easier for them to get things right. A lot of more comprehensive fonts include Greek letters in the main font, so getting the μ right is simple:

\sisetup{
  mathsmu = \text{μ},
  textmu  = μ
}

or for people testing version 2 of siunitx:

\sisetup{
  maths-micro = \text{μ},
  text-micro  = μ
}

There may be a bit of testing required: this will not work if, for example, you are using the Latin Modern font.

I’m continuing to work on version 2 of siunitx, and the code has now reached the point where the basic macros (\num, \SI, \si, \numrange and \SIrange) work at least as well as the version one code. There are probably still some bugs, but I’m using the new code for my own work and at the moment all seems good. The internal improvements mean that while there are still things to add this should not be too hard.

If you want to try things out, as before you can grab things here:

• The source (dtx file)
• The documentation (pdf file)
• The extracted package (sty file)
• A ready-to-install (tds) zip file

What is on the list next is tackling the tables issues. That is going to be hard work, as there are some complicated things to sort out. So I will probably add a few bits and pieces to the rest of the code at the same time.

As always, feedback by e-mail or to the BerliOS site is very welcome.

It’s been a while since I gave an update on progress with siunitx version 2. I’ve had other things on the go, but am trying to make some progress with the code as I have lots of bugs which are fixed in the new version. Since the last snapshot, I’ve gone back over everything and moved internally to expl3 syntax. That makes my life a lot easier, as there are lots of tools available without needing to write them all for myself. It does mean that users/testers will need to have expl3 installed: you will need the most recent release, which should appear in TeX Live and MiKTeX soon for online installation, or can be downloaded from CTAN (look for expl3.tds.zip).

There are some notes and outstanding questions with the current snapshot: it is still some way from being finished. Things to be aware of or to think about:

• Places where units should be repeated are not working properly at the moment. That is the next thing I need to sort out: I’d hope it will be done soon (perhaps another snapshot in a couple of weeks)
• I’d set the defaults to use text mode for units and maths mode for numbers. This may not stay that way, see for example the results for \SI{1.23}{J.mol^{-1}.K^{-1}}. This is because printing units literally has to do the powers in the same font as the units.
• The options system is now using the LaTeX3 l3keys module. Following the pattern that looks likely for LaTeX3 work, I’ve given the options names divided into words using hyphens. I hope that they make sense.
• At present, complex numbers are always printed with the complex root (i) after the number. I need an option to reverse this, but can’t think of a good name.
• I’ve tried to simplify the various ‘trapambig…’ options into a single one (use-brackets). I’m not sure if this will work for everyone: do people need the ability to turn bracketing on and off in a more granulated way?

For those who want to test things out, you can get:

• The source (dtx file)
• The documentation (pdf file)
• The extracted package (sty file)
• A ready-to-install (tds) zip file

I had a bug report for siunitx recently, concerning handling of the ohm symbol when the MathTime Pro package has made \Omega slanted. As many of you will know, MathTime Pro is commercial, and so I was not totally sure I’d be able to sort the issue out. I dropped the suppliers, PCTeX, a quick e-mail to see if they’d let me see the code for bug-fixing purposes, more in hope than expectation. I was pleasantly surprised to get an e-mail back the next morning, saying that this was no problem and even letting me download the fonts themselves for testing. A ‘pat on the back’ to PCTeX: very good service indeed.

I’ve just released version 1.3 of siunitx to CTAN. Although the changes are not large in number, they are important. First, I’ve altered the default code for \micro (μ), as the previous version did not look great: an important change, hence the bump of version number. Second, the packaging has been altered to make life easier all round. You can now extract the package and typeset in one go from the .dtx file using LaTeX, or just extract it using TeX. Overall, the effect is much cleaner than the old version. Of course, there are some bug fixes too.

I’ve just sent a minor update of the release version of siunitx to CTAN. There was a clash with the fourier package, to do with maths fonts. This is all due to the lack of an upright μ in Computers Modern: a continued pain. For the moment, all is sorted again! One note: I’ve done the release here using TeX Live 2009, rather than my more usual MiKTeX 2.7. Everything seems to have worked, but any feedback is of course welcome!