Some TeX Developments

I work hard on my LaTeX packages, and try to get things to work well for the user with clear interfaces. However, there is a down side to this: I get asked to do more things! For example, I’ve recently released version 2 of siunitx. This builds on the work from version 1, which itself was designed as an improvement to various earlier unit packages for LaTeX. I did remove a few features when I moved from version 1 to version 2, but in the main each new version of siunitx has added ‘more stuff’ to the package. However, I’ve also got organised with tracking bugs and feature requests using BitBucket. The resulting list of open issues is already quite frightening (at least for me). I’ve been keeping up with the bugs in the new release (I try to deal with them within a few days of being reported), so the list is all made up of feature requests, and almost all of those are new things. So the package being successful results in more work for me, not less. I’m not complaining: I want things to work. It’s just worth bearing in mind!

Following the release of version 2 of siunitx, you might have noticed that there have been rapid minor updates (v2.0a, v2.0b and v2.0c). I thought I’d just say that most of this is to make sure that the version that ends up in TeX Live 2010 is as good as possible. There are the inevitable bugs to sort, especially with compatibility with version 1, and I want to get things working as well as possible. I suspect there will be a few more releases in quick succession!

After many months of work, I’m pleased to announce that I’ve just sent version 2 of siunitx to CTAN. Many readers will be familiar with the package and some of the development process. Here, I’ve put together a summary as ‘release notes’ for the new version.

A comprehensive (SI) units package

Typesetting values with units requires care to ensure that the combined mathematical meaning of the value plus unit combination is clear. In particular, the SI units system lays down a consistent set of units with rules on how these are to be used. However, different countries and publishers have differing
conventions on the exact appearance of numbers (and units).

The siunitx package provides a set of tools for authors to typeset numbers and units in a consistent way. The package has an extended set of configuration options which make it possible to follow varying typographic conventions with the same input syntax. The package includes automated processing of numbers and
units, and the ability to control tabular alignment of numbers.

Version 2

Over the past two years siunitx has developed to include many features not originally foreseen when development began. While it has been possible to add a range of new features, some of the underlying limitation of the version 1 code have made this difficult. At the same time, renewed effort by the LaTeX Team on the development of LaTeX3, and in particular the expl3 programming system, has offered a more robust method to create the internal structure of siunitx. As a result, version 2 of siunitx has been almost completely re-written internally.

As well as fixing a number of bugs and limitations in the original release, version 2 is also much better written to work quickly. As a result, most users should see performance enhancements with this new release of siunitx.

As part of the revision of siunitx, the option system and user macros have been completely re-thought. The options now have longer, descriptive names and also a much clearer range of input values. The options which in version 1 took either a key word or a literal value have been replaced by ones which take literals only: in some cases this means that advice has been added to the documentation on how to get particular output effects.

Moving from version 1 to version 2

Depending on how you use siunitx, there may be very little to do to move to version 2. The new version includes a compatibility support file, meaning that loading siunitx using:

\usepackage[load-configurations = version-1]{siunitx}

should mean that existing documents compile with very few changes.

There are some changes to standard settings between version 1 and version 2, which may lead to some alterations in documents. At the same time, a small number of the features of siunitx version 1 which I feel did not work cleanly have been dropped. At present, some of these are scheduled to be re-examined for
inclusion in later releases of siunitx.

While there is a back-compatibility layer for users upgrading, it is strongly recommended that documents are updated to use the new option names and functions. The new approach has been chosen as it is an improvement on the previous version, and in the longer term this layer may be removed.

Installation

Most users will obtain siunitx as part of their TeX distribution. MiKTeX 2.8 should include siunitx version 2 after a short delay (a few days after CTAN upload). For TeX Live users, there will be a slight delay as the package will appear in updated form in TeX Live 2010 but not TeX Live 2009 (which is frozen).

For users who wish to install siunitx themselves, the package is available as a pre-extracted zip file, siunitx.tds.zip. Simply unzip this in your local texmf directory and run ‘texhash’ to update the database of file locations. Version 2 of siunitx requires up to date versions of the LaTeX3 packages expl3 and xpackages. These are also available from CTAN in ready to install format (as expl3.tds.zip and xpackages.tds.zip), and can be installed in the same way if necessary.

If you want to unpack the dtx yourself, running ‘tex siunitx.dtx’ will extract the package whereas ‘latex siunitx.dtx’ will extract it and also typeset the documentation. Typesetting the documentation requires a number of packages in addition to those needed to use siunitx. These should all be available in a complete TeX Live 2010 or MiKTeX 2.8 installation.

Development code and bug database

In order to help users see what is happening, and also to allow me to work efficiently, the development code for siunitx is available on the code hosting site BitBucket.

You can download the very latest code from there: of course, this may or may not work properly depending on exactly what I have added to the code.

The BitBucket site includes an issue tracker, where you can report bugs or make feature requests. I also add bugs to the database from e-mails I get from users. Filling in the bug database helps to make sure that I do not forget things, and also helps other users see what issues are known.

If you want to contribute code to siunitx, you can of course send patches directly to me. Alternatively, the code is hosted using the revision control system Mercurial, which was chosen as it is decentralised and is easy to install on a range of operating systems (I use MacOS X, Windows XP, Windows 7 and Ubuntu!). I’m happy to explain to potential contributors how Mercurial works for developing siunitx.

Roadmap for future releases

The bug database already includes a number of feature requests which are marked to be looked at for version 2.1. The current intention is that the next few months will be devoted to bug fixes in this release (v2.0), with moves to add features for v2.1 beginning in the late summer. I anticipate that v2.1 will be released toward the end of 2010.

It is likely that not all of the features currently marked as to be looked at for v2.1 will be fully working by the time it is released. At the same time, there are some longer term areas which may also need attention. Version 2.2 of siunitx is therefore planned, but with no current list of features marked for inclusion. This version is likely to appear in Spring 2011.

One longer term aim is to include LuaTeX support in siunitx, so that the entire package can work much more rapidly with LuaTeX than when using TeX macros alone. This is not likely to happen until next year (2011), but is in the bug database and is part of the longer term development plan for siunitx.

The internals of siunitx

Currently, the only documented interface to any of the functionality of siunitx is via the key-value control system and functions described in the manual. The internal code of the package is not documented, and there is therefore no guarantee of stability of internal functions. While it is common for users to have to modify the internals of LaTeX2e packages as part of their documents, this is not good programming practice and is not encouraged for siunitx, or indeed in general.

If there is a user function that you require that is not available using the documented tools, please either e-mail or report a bug in the database. One of the general aims of siunitx is to provide a proper documented interface for all of the
functions of the package. I am therefore very happy to add interfaces to internal processes as necessary.

Programmers should note that siunitx is coded using the LaTeX3 ‘expl3′ programming system. This looks somewhat different to traditional TeX or LaTeX programming. Details of the programming environment are documented as part of the expl3 bundle. Currently, none of the internal functions or interfaces are documented, and so are not meant for use outside of siunitx. Other programmers wanting to make use of internal siunitx functions are encouraged to get in contact with me. This will enable me to ensure that the parts of siunitx which are needed by others are documented and are not changed without consultation.

Over the past few months I’ve been working a new version of siunitx with completely re-written internals. This is now at the point where I hope that it is usable for most people, but before replaced version 1 some testing is needed. This is what this beta (testing) release is for. There are some important notes for people testing, which I’ll run through below. For the impatient, you can get:

• A ready to install .zip file
• The documentation
• The source (dtx) file

Release notes

The code used in siunitx relies on the LaTeX3 Project packages expl3 and xpackages. You will need the latest versions of both of these to test siunitx 2: they can both be downloaded from CTAN or installed using the update facilities in TeX Live or MiKTeX.

Version 2 of siunitx renames most of the package options to make them more informative. The old names are available by using:

\usepackage[load-configurations = version-1]{siunitx}

The new option names are intended to make it easier to continue to expand siunitx without having completely opaque option names. At the same time, the nature of some of the options has been changed. This means that there are no longer any ‘magic’ keywords, which have caused confusion in the past.

There will be no significant features added to siunitx version 2 between the beta version and the production release (probably in June). The aim is to get the inevitable bugs in the current code found and fixed, which is best done while not making other changes.

A small number of features from version 1 of siunitx are not present in version 2. The features removed have never worked as well as I would like, and so I felt it was better to remove them and rework them later if needed. If this causes severe problems for users then some of these decisions may of course be reversed.

I have had a large number of feature requests for siunitx, and only some of these have been added to version 2 at present. This is partly as I have a limited amount of time, and need to get siunitx 2 to release within a reasonable time. At the same time, some of the feature requests are very specialist and I need to consider which of these fall within the scope of the package. That said, I intend to work on adding more features to siunitx after the full release of version 2.0. More details on this will be posted here in the future.

Feedback on any aspect of siunitx version 2 is very welcome: joseph.wright@morningstar2.co.uk.

As many readers will know I’ve been working on version 2 of siunitx for some time. There are always more feature requests, but at some stage I have to actually release something. I’ve now got some code that fixes a lot of bugs and annoyances in the current release and provides a better platform for new features, and so I want to move to releasing it.

The plan is to fix the known bugs in the development code and add as many new features as possible by the 25th of the month, and then to freeze development for a beta (testing) release. Depending on feedback, I’ll then aim to fix the bugs in the beta and go for a first production edition of version 2 in June. I’m not going to add any new features between the beta and production releases, but will start on them again once the production version is out. There is a feature tracker on the BerliOS site, which is a good place to put things so I don’t forget about them.

I’ve been working on siunitx version 2, getting the code to the point where I’d hope it works for most things that the current release does. Over the last couple of weeks I hope I’ve sorted out the table issues which tend to be a problem in version 1, plus “mopped up” a few outstanding odds and ends. So it feels like an appropriate point for a public snap shot of the code. As progress has been good, I’m calling this one alpha 1.

At this stage, there are likely to be some bugs and other annoyances in the code. However, I hope that enough works for people to risk taking the plunge and trying it out. For those willing to try out siunitx v2.0, you can get:

• The ready to install TDS-style zip file
• The package documentation
• The source file (dtx)

You’ll need to have up to date installations of both expl3 and the xpackages to try out the code, as internally the new code uses the LaTeX3 internal syntax. The biggest change that users should see from version 1 is that I’ve re-thought the option names. They are mainly longer, but more informative, in the new code. Improvements to the names I’ve picked are of course welcome.

At this stage, I’m still working on adding more ideas to the code. So there are some omissions in the release that I know are there and am intending to sort out. Of course, different users have different priorities for improvement. That said, there are bound to be things which simply are broken, things I’ve forgotten and the odd item that I’m currently planning not to carry forward from version 1. Feedback by as comments here, by e-mail or at the BerliOS site is very welcome.

The current development plan is to see how much is wrong with the alpha code, and if necessary to have some more alpha-status snap shots. I’d then hope to have a beta some time in the spring (perhaps April or May), with a full release currently in my mind for June or July. Exactly how this will work out depends on other projects: I’d like, for example, to have some floating-point tools in siunitx, but for that I need to write them for LaTeX3. The feature list for the release certainly isn’t fixed, and I’d expect that once v2.0 is out there will continue to be more ideas to add on.

My previous post mentioned some efforts to improve the performance of the siunitx parser. I’ve now committed an entirely new version of the parsing code to the repository. I’ve also done my best to speed up the rest of the package. The speed you see very much depends on the type of input involved. With my test file, I got a time of roughly 3 minutes using version 1 of siunitx, about 2 minutes before improving version 2, and about 1.5 minutes after. For comparison, doing no processing at all take the time down below 10 seconds for the same file (roughly 700 pages of repeated input). For the interested reader, an siunitx snapshot TDS-style zip is available.

One thing that I find interesting in all of this is that even before optimising the code, version 2 still worked faster than version 1, even though it does more things. A lot of that is because the pre-built looping material in expl3 does a much better job than my own attempts in version 1 of siunitx. Of course, good programmers will always use fast loops, but for the rest of us I think this shows how a sensible tool-kit can bring benefits “behind the scenes”.

I had an e-mail today about using siunitx when there are a lot of calls to the package. As you might expect, things can get a bit slow, and the person who contacted me felt that things get rather too slow. There are differences between the current release version of siunitx and the development code (version 2), and I’ve also added a few features to help speed things up where appropriate using version 2. So I thought I’d put a bit of information on the comparison in the public domain.

First, a baseline is not to use siunitx at all, and to simply test everything by hand. For that, I tried the simple test file:

\documentclass{article}
\usepackage{siunitx,xparse}
\ExplSyntaxOn
\DeclareDocumentCommand \repeated { m m }{
  \prg_replicate:nn {#1} {#2}
}
\ExplSyntaxOff
\begin{document}

\repeated{10000}{$1.23\,\text{m}$ }

\end{document}

This repeats the same text 10 000 times: boring but handy for testing. Using the command-line time program, I get an overall time of 1.714 s for this.

A very slight change of the file lets me test with siunitx version 1:

\documentclass{article}
\usepackage{siunitx,xparse}
\ExplSyntaxOn
\DeclareDocumentCommand \repeated { m m }{
  \prg_replicate:nn {#1} {#2}
}
\ExplSyntaxOff
\begin{document}

\repeated{10000}{\SI{1.23}{\metre} }

\end{document}

With the latest release version of siunitx (v1.3g), I get a time of 80.878 s for this on the same system.

In siunitx version 2, I’ve recoded all of the loops and parsing code, and so things are faster using the standard settings: 58.944 s. With the very latest code (SVN 243), I’ve included two options to make things move faster: parse-numbers and parse-units. Of course, these do mean that you get less of the power of siunitx. But for many people they might be useful. Turning both parsing systems off, the time needed for the test file drops to 14.975 s (just turning off the number parser gives a time of 18.803 s).

I may take another look at trying to improve the performance of the number parser. The problem is at least in part that making the code faster will either mean making some of it less powerful or, more likely, a lot harder to read and maintain. I hope that for most people, most of the time, the performance is acceptable. Of course, at some point I’ll try to do some Lua-based code for the parsers, at least. But that won’t help for most users now.

Ultimately, there is a limit to how fast things can work. Whether the performance hit of using siunitx is worthwhile is something is down to users. I think it’s worth it, as the better logic in the mark-up more than makes up for the extra time required. But then I would say that!

I get a few e-mails about siunitx and the micro symbol. People tend to be surprised that the symbol ‘sticks’ to a look very much like Computer Modern. The reason is that picking a proper upright (not italic) μ is not so easy in TeX.You don’t get one in Computer Modern, so siunitx takes one from the TS1 (text support) set in the absence of a better plan. I’ve set up some auto-detection for a few obvious alternatives (such as the upgreek package), but that doesn’t really work for XeTeX users.

XeTeX users are likely to load system fonts, and I’d hope be using UTF-8 input. That makes it hard to auto-detect what they are doing, but should make life easier for them to get things right. A lot of more comprehensive fonts include Greek letters in the main font, so getting the μ right is simple:

\sisetup{
  mathsmu = \text{μ},
  textmu  = μ
}

or for people testing version 2 of siunitx:

\sisetup{
  maths-micro = \text{μ},
  text-micro  = μ
}

There may be a bit of testing required: this will not work if, for example, you are using the Latin Modern font.

I’m continuing to work on version 2 of siunitx, and the code has now reached the point where the basic macros (\num, \SI, \si, \numrange and \SIrange) work at least as well as the version one code. There are probably still some bugs, but I’m using the new code for my own work and at the moment all seems good. The internal improvements mean that while there are still things to add this should not be too hard.

If you want to try things out, as before you can grab things here:

• The source (dtx file)
• The documentation (pdf file)
• The extracted package (sty file)
• A ready-to-install (tds) zip file

What is on the list next is tackling the tables issues. That is going to be hard work, as there are some complicated things to sort out. So I will probably add a few bits and pieces to the rest of the code at the same time.

As always, feedback by e-mail or to the BerliOS site is very welcome.