Some TeX Developments

Over the years of working with LaTeX, I’ve picked up a bit about version control systems for code: this post is more about general programming than TeX.

I started out with Subversion, then moved to Mercurial when I got involved in beamer maintenance. The idea is the same whatever system you are using: by keeping a track of changes in the code you help yourself in the long term, and make it easier for other people to help too. Mercurial is one of several ‘distributed’ version control systems (DCVS) that have been developed over the last few years. The idea is that each ‘repository’ (copy of the code) has the history with it, and so is independent of any server. You can still send your changes to a server, and that is very popular, but you don’t have to. Sending code to a public server makes it easy to let other people get involved, report issues and so on, and there are lots of sites that will help you do this.

I picked Mercurial over the other leader, Git, mainly because the other guy involved in looking after beamer went this way and put the code on BitBucket. At the time, BitBucket did Mercurial while GitHub did Git. BitBucket changed hands a little while ago now, and they brought in Git support. They’ve now moved to make Git the ‘standard’ repository type. That tells me that Git is likely to ‘win’ as the most popular DCVS (it’s looked that way for a while), and so it’s time to reconsider my use of Mercurial.

It turns out that moving from Mercurial to Git is pretty easy: there is a script called fast-export that does the job. Converting the code itself is therefore easy: run the script (you need a Unix system, so on Windows I’m using a virtual Ubuntu machine with VirtualBox). Life gets a bit more interesting, though, if you want to keep your issues database. BitBucket does offer issue import and export, but no easy way to convert from Mercurial to Git. At the same time, the way that the two systems refer to individual commits means that if you don’t edit your issues, any links to the commits will be messed up. That means that its as easy to move to GitHub as it is to stay on BitBucket. So that’s what I’ve decided to do (GitHub is pretty popular with other LaTeX developers). I’m working through my repositories, converting to Git and uploading to GitHub, then copying the issue information by hand and doing minor edits. That includes making sure that I keep the links which show how I fixed things. Apart from siunitx, my packages don’t have a lot of issues (no more than a dozen each), so I can do that by hand without too much work. I’d a bit surprised no-one has written a script to do this, but at least it will not take too long. I’d expect everything except siunitx to be moved by the weekend, and even this ‘big job’ to be done within a couple of weeks.

Khaled Hosny has announced on the XeTeX mailing list that XeTeX 0.9999 has just been released. The list of changes is pretty long, as XeTeX has had quite a backlog of issues. Probably the biggest single change is

Port OpenType layout from ICU LayoutEngine to HarfBuzz. HarfBuzz is actively maintained and generally have much wider support for
OpenType spec, the switch fixes a number of OpenType bugs:

• Support version 2 OpenType Indic specs.
• Many other Indic OpenType bugs, and support for the latest additions to OpenType spec.
• Incorrect application of contextual features.
• Incorrect kerning in fonts that has both old “kern” table and new GPOS “kern” feature.
• Allow suppressing Latin ligatures with ZWNJ.
• Support for variation selectors.
• Support for user-specified features with complex scripts.

If you are familiar with layout engines, you’ll know that while ICU has worked very well for XeTeX from day one, it’s no longer being developed while HarfBuzz is being developed. More importantly, HarfBuzz is supported by the open source community well beyond the TeX world, so by moving in this direction XeTeX gets the benefits of the efforts of many other people: part of the point of open source software. I’m sure it’s been a big effort making this change: I’m looking forward to testing it out.

The other headline change, at least for Mac users, is moving to Core Text rather than ATS/ATSUI. Apple have dropped support for the latter, so there was a worry about building XeTeX on the Mac in the future. That’s now sorted, and means XeTeX should work as a 64-bit application on the Mac in future.

If you read the full announcement you’ll see there are lots of other changes and bug fixes. Congratulations to Khaled on this: it’s great to see that XeTeX continues to develop, and that several features have been added to make working with XeTeX and LuaTeX seamlessly are now there.

Clipping boxes is something that TeX does not do: it simply places them on the page. That means that clipping graphics (a pretty common requirement) is actually done by the driver rather than by TeX. The LaTeX graphics package and the driver support that come with it cover quite a lot of cases, and over the years support for a number of other drivers have been written based on the same ideas. However, things are still not 100% identical over all back-ends. A particular gap at the moment is that that XeTeX support code does not offer clipping, because the XeTeX engine does not do this (pdfTeX and LuaTeX both do). Users of pgf might have noticed that it manages to do clipping perfectly happily with XeTeX (or rather they might have wondered why graphics doesn’t when pgf does). Martin Scharrer and I looked at this a while ago for his adjustbox package, and worked out what is actually needed: some PostScript specials in a xdvipdfmx wrapper. The same basic idea is now being integrated into xetex.def, the driver support code used by graphics. This will go to CTAN soon, but some testing would be good. The updated file is available now, so I’d encourage intrepid readers to download and test it!

Many readers will have been watching the updates to CTAN with interest. So far, we’ve seen a new layout to the site, with a promise of new ‘toys’. Some of these have now appeared at least in testing form, and can be accessed at http://www-new.ctan.org.

The biggest change is that you can now register an account, and use this to vote on packages. I’ve of course registered already, and taken a very quick look at the voting system. I notice that at the moment the system doesn’t spot that I’m the author of some packages, so can vote on them! Of course, I’m not going to do that, and if there are lots of people voting it won’t matter.

Anyone who subscribes to the XeTeX or LuaTeX mailing lists will have seen a lot of posts from me over the past couple of weeks, pursuing various aspects of multi-directional typesetting. That might prompt some questions, not least what has raised this interest!

I got a bug report for siunitx a little while ago about interaction with the bidi package. That pushed me to look at something I’ve been wondering about for a while: what we need to do in the LaTeX3 codebase to allow for multiple directions (particularly right-to-left, which as bidi shows should be usable directly in LaTeX). It’s turned out that good documentation on the various primitives available is not so easy to come by, and that there is a lot to get your head around. Luckily, I’ve had some very useful feedback from a range of people, including John Plaice (one of the authors of Omega). It also turns out that not everything is quite settled in engine support for multiple directions, so asking questions is pretty important.

As a package developer, I regularly send material to CTAN. Once my code gets uploaded, there is not much I can do about it spreading around the world. There’s always a slight delay in the various CTAN mirrors picking up new material, while the code only changes TeX Live and MiKTeX once the maintainers of those systems spot the CTAN changes.

For a single package upload, that process simply means a little bit of a delay between me saying that something is fixed and everyone being able to use it. However, where there are linked changes in more than one package then life gets a bit more complicated. A classic situation is siunitx, which uses the LaTeX3 programming environment. I do the releases for LaTeX3 so if there is an issue which needs a change in both siunitx and the LaTeX3 code both get updated on CTAN at the same time. However, it’s not unknown for TeX Live or MiKTeX to pick up only one change initially, leading to issues for end users. That’s a pain, but there is little I can do about it as I don’t control that process.

A while ago I posted about Arara, a new tool to allow automated compilation of (La)TeX documents. As I said then, the big advantage of Arara is that each document can be set up to follow distinct rules which are part of the document itself, rather than being part of the tool. Development of Arara has continued at some pace, and it’s now available to download with a cross-platform installer. The bundle includes pre-built rules for common cases and an option to add to the system path (to make using it easier). So I’d expect a lot of people to try Arara out: it really is a very clear approach to automation.

The TeX Users Group today released TeX Live 2012, the TeX distribution used by many Unix and Windows users and the basis for MacTeX. This year’s new features are very much evolutionary, with perhaps the highlight being support for multiple repositories (thus making using TLContrib easier to use). As always, TeX Live 2012 has been in pretest for some time, and works very nicely. (If you’ve been using the pretest, there is no need to reinstall.)

Pretesting of the next TeX Live is well under way. This is very much evolutionary: no major changes from TL2011. So far I’ve had no surprises, but that does not mean a bit of caution is not needed (I will keep TL2011 installed too!).  Of course, pretesting works best if lots of people get involved, so if you are not doing something critical with documents, or you are comfortable switching between TeX Live installations, then grab a copy and help out.

I recently discovered that you can read issues most of Die TeXnische Komödie online. My German is good enough to get the gist of the articles, and there is some interesting stuff there. I’ve always meant to join DANTE, if only to support their core CTAN server, and so this seemed like an ideal opportunity. So in the post this week I got the various goodies that they send out: I particularly like the sticker!

What’s particular interesting for me is the section in Die TeXnische Komödie detailing the various TeX-related meet-ups that take place in Germany: there are a lot! Here in the UK, I’m pleased that UK-TUG manages a couple of training courses and a single speaker meeting/AGM every year. It’s quite a contrast!