Some TeX Developments

Before the series on programming LaTeX3 can really get started, it’s going to be important to establish some background, basic concepts and indeed what the aims are. So in this post I’m going cover some of these issues: we won’t be seeing any code just yet! The approach I’m aiming to take is to bring in concepts as they are needed: this may mean a few simplifications in the beginning to allow ideas to be developed.

LaTeX3: What is available now?

The very first thing to cover is what the current status of LaTeX3 is, and what the aim of this series is. Anyone following LaTeX3 development will know that at the moment it’s not ready for creating documents independent of LaTeX2e. What is available now is a programming layer: l3kernel. At the same time, one of the aims with LaTeX3 is to clearly separate out programming, design decisions and actually using LaTeX. So what I will be covering here is programming. At the same time, I’ll aim to highlight concepts which are not necessarily tied to LaTeX3 programming but which the LaTeX3 Project feel are part of the overall aims of LaTeX3 development.

The target audience

I have two distinct audiences in mind in writing this series. The first is experienced (La)TeX programmers who want to see what ideas LaTeX3 introduces. These people will be familiar with many basic TeX concepts, and will want to see the relationship between what they are used to and the ‘LaTeX3 way’. The second group is experience LaTeX2e users who want to learn to program LaTeX, and have decided to miss out learning to program TeX first. It’s important that the latter group are included: another key aim for LaTeX3 is to provide a complete set of documentation and support without having to say ‘read The TeXbook’ as a requirement to make progress.

What both of these groups have in common is lots of experience with LaTeX2e. So I’m going to expect familiarity with LaTeX2e’s user syntax, concepts and so on. So that will very much be the baseline: I do hope that the more experienced LaTeX programmers will bear with me.

Requirements

As I’ve indicated, programming LaTeX3 currently means works on top of LaTeX2e. So to get started you need a LaTeX2e installation, which for most people means either TeX Live or MiKTeX. Most of the code in the programming layer of LaTeX3 has been moving to a stable situation for some time, but there are refinements going on all of the time. As a result, I’ll be assuming that readers have the latest CTAN releases of l3kernel and l3packages installed. That can be done by downloading them from CTAN directly, or using the package managers in TeX Live 2011 or MiKTeX 2.9.

Development of LaTeX3 has attracted interest from other TeX programmers for a while. One of the big barriers to new entrants is that programming LaTeX3 is distinct from programming LaTeX2e or plain TeX. So what is needed is a ‘Programming LaTeX3’ guide. The problem is getting one written: these things take time, and what to write is also something of a challenge.

To make a start on tackling this, I thought it would be useful to write a series of short blog posts, taking one area of LaTeX3 at a time and looking at it from the point of view of beginner in programming LaTeX3. The idea is that by keeping things short I can divide the problem into manageable chunks (both for readers and for me), and get feedback on each part before taking on the next one. If I make decent progress, I’ll then have some material to edit into something like an article for TUGBoat.

Now, to do a reasonable job I will have to cover some things I’ve looked at before: sorry if it turns out to be repetitive in places. I’m planning to start by looking at how you can actually start programming LaTeX3 today, covering the idea of ‘LaTeX3 in 2e’, for example. Then it will be on to the basics of the language, before we even get to creating any macros. Ideas for topics to cover are very welcome!

There was a question recently on the TeX.sx site about font selection and LaTeX3. At the moment, there is not a LaTeX3 font system set up, and there are issues outstanding, so this is not something with a single answer. What I can do, though, is look at what seems likely and what some of the areas to consider are.

(New) Font Selection Scheme

TeX’s font mechanism is pretty basic. There is no relationship between one text font and another: they are all simply set up using the \font primitive. So with plain TeX

{\bf Some {\it text}}

will have ‘Some’ in bold, but ‘text’ in mid-weight italics. LaTeX2e introduced the ‘New Font Selection Scheme’ (NFSS), which provides a method for managing fonts in a way that is likely to be more logical for the user. Thus

{\bfseries Some {\itshape text}}

will have the inner text both bold and italic. At the same time, the NFSS provides a system for loading font files in an organised way and substituting fonts when a particular shape combination is unavailable.

Over all, the NFSS is one the key successes of LaTeX2e compared with LaTeX2.09. There are also a lot of existing .fd files about for using fonts with LaTeX2e, and supporting those is important. So something like the NFSS is definitely needed: the ‘New’ is rather anachronistic nowadays, so the working title is just FSS.

The NFSS is not perfect, and so LaTeX3′s FSS cannot be simply a clone of NFSS. Perhaps the most common complaint about the NFSS is that \textsc is treated as a shape, which makes it impossible to combine it with \itshape to have italic small caps. Other areas which need addressing are for example flexible sizing and proportional/fixed width numbers for tables. This is all evolutionary, and so the plan is to port the existing NFSS first, tidy it up to fit better with LaTeX3 coding approaches, then add new abilities.

Font face loading

The second area to think about is loading fonts in the first place. The traditional LaTeX2e approach to this to set up a small(ish) package to select a font family, for example lmodern or mathptmx, which will then use the NFSS to load the appropriate TeX font files. For users of XeTeX or LuaTeX, the standard method is to use the fontspec package, which provides an interface between the extended \font primitives in these engines and the NFSS.

There are a few things to think about here. First, while XeTeX and LuaTeX can load system fonts directly, pdfTeX cannot. Secondly, even if you are using XeTeX or LuaTeX access to traditional TeX fonts cannot be ignored. There is a lot of MetaFont material on CTAN which is not available in any other format, so simply dropping support for these is not an option.

What I feel we need is a single font-loading interface at the user level which is capable of dealing with these requirements. Clearly, fontspec is going to provide inspiration on how to proceed, but some mechanism for working with pdfTeX will also be needed. My personal take on this is we’ll need a mapping layer, which will mean that at the user level you choose a font by name (as you would in a GUI application), and which then does the appropriate translation to the engine layer.

There are also math mode fonts to worry about. OpenType maths fonts are very much in development, but that doesn’t help with pdfTeX and again does not cover all cases. So again we need to continue to support TeX’s traditional math mode fonts. That will probably be the last part of this particular jigsaw to be tackled, simply because it’s the one with the least clear path at present.

One of the questions that comes up from time to time is what the ‘roadmap’ is for LaTeX3 development. While there is not an official plan, I certainly have some ideas on what I’d like to see addressed in a concrete way. This is all rather flexible, but I’ll try to outline some areas for attention.

Short term

The short term is about things which LaTeX3 does not critically depend on, and for which a lot of work is already done. This is very much an ongoing area:  small improvements spread across a range of topics. These items all fit into the concept of ‘LaTeX3 in LaTeX2e’, where this material can be loaded without affecting existing documents.

Floating point expressions

The current floating point implementation in LaTeX3 is based on carrying out assignments, and is pretty simple-minded in the way values are stored. Bruno Le Floch is working on an improved approach, which will allow floating point expressions to be parsed in an expandable way. This will be very handy for doing things like box rotations, but also in the future for Tikz-like drawing capabilities.

Extending key–value concepts

The current l3keys module is based loosely on pgfkeys. One area that is not implemented is the idea of a ‘key path’. Using a path model for setting keys has some advantages, so I’d like to add something like the ability to do

\keys_set:n
  {
    /module .cd ,
    key-a = value ,
    key-b .meta = { key-a = setting , /other-module/key = value }
  }

LaTeX3 niggles

There are a number of issues that have come up as LaTeX3 has seen more use. Fixing these is an ongoing problem, and we should probably look to deal with a few over the coming weeks. The supply is not going to run out!

Documentation improvements

A key aim for LaTeX3 is to document all of the material which is intended for others to use. Getting documentation right is hard, and so another ongoing area is to improve the documentation. We’ve already done a bit of work on this, but it’s certainly not complete.

Medium term

The medium term is about bigger ideas, but ones where there is some code written and we can see that results will appear within the next few months. These areas need discussion as much as writing code. Here, we move away from ‘LaTeX3 in LaTeX2e’ to code which will break existing documents unless they are adapted.

The galley

The galley is the vertical area into which text and other content is places. Galley management in LaTeX2e is basic, and this leads to odd effects with vertical spacing. For example, try

\end{itemize}
\vspace{3 pt}
\begin{itemize}

and see that you get 12 pt of space! We’ve already got three galley implementations, and are now working on a fourth! So the issue here is deciding what is the best approach: the different versions all have different levels of complexity. Hopefully we can resolve these ideas into a single working module within the next couple of months.

Font selection

The ‘New Font Selection Scheme’ in LaTeX2e is a pretty good approach to selecting fonts. So the first task here is more-or-less to convert the code to LaTeX3 syntax. There are then a few items to address, such as how to deal with small caps. Loading fonts also needs to be fitted in here: some combination of fontspec (for XeTeX and LuaTeX) with the existing pdfTeX mechanisms is going to be needed, and it’s a question of how to do this.

Long term

The long-term is again about big ideas, but here ones where the code is not yet written. Some of the concepts are sorted, but beyond that there is work to do.

The output routine

The biggest single challenge for LaTeX3 at an implementation level is getting the output routine right. Here, we have xor, an approach written by Frank Mittelbach. This works, but it’s in need of a serious overhaul as the code has developed over many years. So at the moment it’s rather a mix of LaTeX2e and LaTeX3 stuff. We’ve also got to discuss how the output routine should work, addressing things like flowing text across frames, grid typesetting and so on. This is all hard work.

Beyond the output routine

Once we have all of the above addressed, then we need to look at getting a format created that can actually typeset material. That means building the code level, galley, fonts and output routine into a stand-alone format. The result won’t be impressive, as it won’t have much in the way of mark-up abilities. However, it will allow testing without LaTeX2e. That’s the stage where we’ll need to add things like sectioning, float management, and all of the other stuff that can be done by LaTeX at the moment.

In the latest LaTeX3 News, the Project announced that a mirror of the development code is available on GitHub. Since then, there has been a definite increase in using expl3 (the LaTeX3 programming language), and we’ve had a number of very good questions on the LaTeX3 mailing list.

The programming layer of LaTeX3 works well in the main, and so it’s very pleasing to see other people starting to pick up on using it. Of course, there are some rough edges, and one result of the questions is that these have to be addressed. At the same time, wider use means that there is a greater need to make sure that the code is stable enough. Getting the balance right between making desirable changes and ensuring that other peoples code does not break is not easy! So there is more work to do with greater use.

At the same time, there is the question of a wider ‘road map’. For the programming layer, there are a few outstanding issues, for example a broader string module and an expandable floating point unit (both under way). However, most of the ‘big issues’ are outside of the base layer: we need support for a new format. There’s quite a lot to talk about there, and I’ll come back to it in another post in a day or so.

The LaTeX3 Project have recently updated the organisation of the various LaTeX3-based packages on CTAN. This means that the older expl3 and xpackages need to be replaced by l3kernel and l3packages. Unfortunately, this seems to confuse MiKTeX, which does not pick up the need to install the new material. So MiKTeX users will need to do this by hand in the MiKTeX Package Manager. This should be a passing problem, but does seem to be causing some confusion for MiKTeX users.

There was a question on the {TeX} Q&A site recently about compiling a document reading

\documentclass{minimal}
\document
a
\enddocument

This doesn’t work: I’ve explained why as my answer to the question. I also mentioned that in LaTeX3 I’d expect this won’t work. Here, I want to look at that in a bit more detail.

First, the background. When LaTeX2e creates an environment foo, what actually happens is two macros called \foo and \endfoo are defined (hence the original question). When you do

\begin{foo}
...
\end{foo}

LaTeX looks for the \foo macro, and if it finds it starts a group and inserts \foo. At the end of the environment, LaTeX will use \endfoo if it exists, but this is not required.

There are some problems with this scheme. First, it’s possible to abuse the system, as something like

\begin{emph}
...
\end{emph}

will not raise an error even though \emph was never intended to be used as the start of an environment. (LaTeX2e does not require that \endemph exists here: it’s only the start of an environment which must be defined.) Secondly, due to this mixing it’s not possible to have a foo environment and independent \foo macro: the two are tied together. Finally, as \endfoo is ‘special’ \newcommand won’t let you create macros which start \end....

For LaTeX3, we want the approach to be different. Document commands and document environments should be independent of one another, and so xparse defines a pair of special internal macros when you use

\NewDocumentEnvironment{foo}...

This is totally independent of a macro called \foo, so the plan is that you’ll be able to have an environment called foo and a separate \foo command, or indeed one called \endfoo.

At present, most people will be using xparse with LaTeX2e. That means we still need to follow what LaTeX2e does. So at the moment you’re not going to see the change: messing with the LaTeX2e mechanism looks like a very bad idea. However, for a native LaTeX3 format the clash will disappear.

A quick note to say that I’ve just uploaded to CTAN updates for expl3, xpackages and siunitx. There are some low-level changes which mean that these have to be updated together. So if you grab one of them, grab them all!

It’s well-known that TeX is good at integer arithmetic, and does not provide any primitive functions for real number calculations. Experienced TeX programmers will know that you can make use of dimen registers to do real number calculations at speed, at the cost of accuracy (as TeX truncates dimensions at five decimal places). However, this is not exactly ideal and can be a bit awkward to use.

An alternative for LaTeX users is to use the fp package (which has been around since LaTeX 2.09). This is a powerful package, but can be rather slow. Part of the reason is that it allows 18 digits either side of the decimal point: a wide range of numbers, but almost certainly overkill in most cases. For applications that really need large numbers (such as pgfplots) input using floating points (such as 1.23e20) is probably better. Floating point calculations make life a bit more complex again, but can be done in TeX (see for example pgfmath).

For LaTeX3 I’ve been working on what would be a sensible approach: real number functions are needed in various places. At the moment, the plan is to support fixed-point numbers with nine digits either side of the decimal place, so up to 999999999.999999999. That should be enough for most application, and at some stage supporting floating-point numbers might well be added. To date there is only basic arithmetic (add, subtract, multiply an divide) in the new code, but the plan is to add trigonometric and logarithmic functions. I’m sure that there will be other functions to add: I’ll be interested to see what is asked for.

One area that I’ve been working is overall performance compared to the fp package. The biggest single gain is of course moving from 18 plus 18 digits to 9 plus 9, which makes quite a big difference on it’s own. However, there are various places inside the code where there are opportunities to save time. First, as LaTeX3 requires ε-TeX I’ve exploited the \numexpr primitive where it makes a difference (mainly where it cuts down on the number of assignments needed). At the same time, there are places where using delimited macros is faster than actually doing mathematics! The exact performance gains depend on what exactly you are doing, but it is possible to draw some comparisons. Doing lots of repeated calculations it’s possible to get some feel for the difference between fp and the new LaTeX3 module. On my system 100 000 additions take 31.6 s with fp and 6.0 s with l3fp, while for 20 000 divisions it takes 64.8 s with fp and 4.1 s using l3fp. Quite some speed enhancement, and I think enough to justify using the new code!

After many months of work, I’m pleased to announce that I’ve just sent version 2 of siunitx to CTAN. Many readers will be familiar with the package and some of the development process. Here, I’ve put together a summary as ‘release notes’ for the new version.

A comprehensive (SI) units package

Typesetting values with units requires care to ensure that the combined mathematical meaning of the value plus unit combination is clear. In particular, the SI units system lays down a consistent set of units with rules on how these are to be used. However, different countries and publishers have differing
conventions on the exact appearance of numbers (and units).

The siunitx package provides a set of tools for authors to typeset numbers and units in a consistent way. The package has an extended set of configuration options which make it possible to follow varying typographic conventions with the same input syntax. The package includes automated processing of numbers and
units, and the ability to control tabular alignment of numbers.

Version 2

Over the past two years siunitx has developed to include many features not originally foreseen when development began. While it has been possible to add a range of new features, some of the underlying limitation of the version 1 code have made this difficult. At the same time, renewed effort by the LaTeX Team on the development of LaTeX3, and in particular the expl3 programming system, has offered a more robust method to create the internal structure of siunitx. As a result, version 2 of siunitx has been almost completely re-written internally.

As well as fixing a number of bugs and limitations in the original release, version 2 is also much better written to work quickly. As a result, most users should see performance enhancements with this new release of siunitx.

As part of the revision of siunitx, the option system and user macros have been completely re-thought. The options now have longer, descriptive names and also a much clearer range of input values. The options which in version 1 took either a key word or a literal value have been replaced by ones which take literals only: in some cases this means that advice has been added to the documentation on how to get particular output effects.

Moving from version 1 to version 2

Depending on how you use siunitx, there may be very little to do to move to version 2. The new version includes a compatibility support file, meaning that loading siunitx using:

\usepackage[load-configurations = version-1]{siunitx}

should mean that existing documents compile with very few changes.

There are some changes to standard settings between version 1 and version 2, which may lead to some alterations in documents. At the same time, a small number of the features of siunitx version 1 which I feel did not work cleanly have been dropped. At present, some of these are scheduled to be re-examined for
inclusion in later releases of siunitx.

While there is a back-compatibility layer for users upgrading, it is strongly recommended that documents are updated to use the new option names and functions. The new approach has been chosen as it is an improvement on the previous version, and in the longer term this layer may be removed.

Installation

Most users will obtain siunitx as part of their TeX distribution. MiKTeX 2.8 should include siunitx version 2 after a short delay (a few days after CTAN upload). For TeX Live users, there will be a slight delay as the package will appear in updated form in TeX Live 2010 but not TeX Live 2009 (which is frozen).

For users who wish to install siunitx themselves, the package is available as a pre-extracted zip file, siunitx.tds.zip. Simply unzip this in your local texmf directory and run ‘texhash’ to update the database of file locations. Version 2 of siunitx requires up to date versions of the LaTeX3 packages expl3 and xpackages. These are also available from CTAN in ready to install format (as expl3.tds.zip and xpackages.tds.zip), and can be installed in the same way if necessary.

If you want to unpack the dtx yourself, running ‘tex siunitx.dtx’ will extract the package whereas ‘latex siunitx.dtx’ will extract it and also typeset the documentation. Typesetting the documentation requires a number of packages in addition to those needed to use siunitx. These should all be available in a complete TeX Live 2010 or MiKTeX 2.8 installation.

Development code and bug database

In order to help users see what is happening, and also to allow me to work efficiently, the development code for siunitx is available on the code hosting site BitBucket.

You can download the very latest code from there: of course, this may or may not work properly depending on exactly what I have added to the code.

The BitBucket site includes an issue tracker, where you can report bugs or make feature requests. I also add bugs to the database from e-mails I get from users. Filling in the bug database helps to make sure that I do not forget things, and also helps other users see what issues are known.

If you want to contribute code to siunitx, you can of course send patches directly to me. Alternatively, the code is hosted using the revision control system Mercurial, which was chosen as it is decentralised and is easy to install on a range of operating systems (I use MacOS X, Windows XP, Windows 7 and Ubuntu!). I’m happy to explain to potential contributors how Mercurial works for developing siunitx.

Roadmap for future releases

The bug database already includes a number of feature requests which are marked to be looked at for version 2.1. The current intention is that the next few months will be devoted to bug fixes in this release (v2.0), with moves to add features for v2.1 beginning in the late summer. I anticipate that v2.1 will be released toward the end of 2010.

It is likely that not all of the features currently marked as to be looked at for v2.1 will be fully working by the time it is released. At the same time, there are some longer term areas which may also need attention. Version 2.2 of siunitx is therefore planned, but with no current list of features marked for inclusion. This version is likely to appear in Spring 2011.

One longer term aim is to include LuaTeX support in siunitx, so that the entire package can work much more rapidly with LuaTeX than when using TeX macros alone. This is not likely to happen until next year (2011), but is in the bug database and is part of the longer term development plan for siunitx.

The internals of siunitx

Currently, the only documented interface to any of the functionality of siunitx is via the key-value control system and functions described in the manual. The internal code of the package is not documented, and there is therefore no guarantee of stability of internal functions. While it is common for users to have to modify the internals of LaTeX2e packages as part of their documents, this is not good programming practice and is not encouraged for siunitx, or indeed in general.

If there is a user function that you require that is not available using the documented tools, please either e-mail or report a bug in the database. One of the general aims of siunitx is to provide a proper documented interface for all of the
functions of the package. I am therefore very happy to add interfaces to internal processes as necessary.

Programmers should note that siunitx is coded using the LaTeX3 ‘expl3′ programming system. This looks somewhat different to traditional TeX or LaTeX programming. Details of the programming environment are documented as part of the expl3 bundle. Currently, none of the internal functions or interfaces are documented, and so are not meant for use outside of siunitx. Other programmers wanting to make use of internal siunitx functions are encouraged to get in contact with me. This will enable me to ensure that the parts of siunitx which are needed by others are documented and are not changed without consultation.