Biblatex back-end update

I wrote recently abut the need to manage biblatex back-ends, and thoughts the maintenance team had on which way to proceed.

After a bit of thought, we’ve gone for a solution we hope works for users and for us. Reversing what seemed like a ‘good idea at the time’, we’ve re-integrated (almost) all of the TeX code into one pathway. This is focussed on Biber, but we have a small stub that converts the relevant parts to work with BibTeX. So users how don’t need Biber or can’t use it can still use BibTeX and get mainly the same results.

There are a few changes, mainly related to places where BibTeX and Biber had ended up with different interfaces and where BibTeX can’t (reasonably) support the Biber one. In those places we’ve dropped the ability to use a ‘BibTeX alternative pathway’ as it makes the code complex and makes switching between back-ends tricky.

Hopefully the balance we’ve gone for will work for everyone: you can still use BibTeX, it still does almost everything it always has with biblatex, but we’ve got a more sustainable code base for the future.

biblatex feedback

Getting an idea of what users actually use can sometimes be tricky. To help understand how people are using biblatex, there’s a short survey set up. In particular, we are looking for some idea of how people use back-ends: as I’ve mentioned before, having two of them means more ‘interesting’ development so it’s important to get some insight into real life use.

Managing biblatex backends

For the past few years, biblatex has been looked after by a small team led by Philip Kime. When we first took this up we were mindful of the need to think carefully about back-ends: how reference data is extracted from a .bib file or similar sources.

There are currently two back-ends for biblatex: BibTeX and Biber. Biber is where the development is taking place and offers Unicode support, whilst BibTeX itself is frozen but also does a lot less ‘stuff’. So there are several features in biblatex that are Biber-only. When the current team took over the maintained, there was consideration of dropping BibTeX entirely. Philip and I have discussed this quite a bit, as the original biblatex developer (Philippe Lehman) picked BibTeX as the original back-end from necessity. (Biber was developed after biblatex, and for extracting and sorting data there was only originally BibTeX.) We decided against it some time ago: what the BibTeX back-end offers is stability but also speed, precisely as it’s more limited that Biber. At least for people like me, in the physical sciences and writing in western European scripts, the BibTeX back-end is perfectly usable.

The way we originally decided to allow continued Biber improvement but keep BibTeX use stable was to split the LaTeX code into two paths. That made sense with the proviso that new Biber features were essentially extensions to the code rather than any changes to existing ideas. However, over time that’s not quite worked out, particularly recently with the new data model driven approach that Philip has developed for Biber. As I’ve detailed elsewhere, that’s led to a new (breaking) syntax for \DeclareNameFormat, as well as various other changes that could be covered in BibTeX but to-date haven’t been. We’ve therefore decided we need to look again at this.

The current plan is for me to work on re-integrating the two back-end code paths, which I’m doing in a fork of biblatex as it’s non-trivial and I don’t want to mess the main development line up. I’ll also look to extend the BibTeX back-end code as appropriate such that we get back to the differences being about the differing capabilities of the back-ends rather than anything in the LaTeX code. I need a little while to do that, probably a couple of months. However, if I get it right we should be in a much stronger position for the future.

biblatex: A new syntax for \DeclareNameFormat

The ‘traditional’ BibTeX model for dividing up names is based around four parts:

  • First name(s)
  • Last name(s)
  • Prefix(es) (the ‘von part’)
  • Suffix(es) (the ‘junior part’)

This works well for many western European names, but falls down for many cases.

As part of Biber/biblatex developments, Philippe Kime has been working on moving beyond this rigid model for names to allow true flexibility. However, this comes with a caveat: a breaking change to \DeclareNameFormat in biblatex. The older syntax takes hard-wired arguments for each name part, but that obviously can’t be extended. The new format only deals with one argument (the name as a whole), but this requires changes to (non-standard) styles.

At the moment, the change is only true for Biber, which means some conditional code is needed. The best way to do that is to test for the older (BibTeX) back-end. For example, in the latest release of biblatex-chem I have in chem-acs.bbx:

% Modify the name format
\@ifpackageloaded{biblatex_legacy}
  {
    % Original syntax for BibTeX model
    \DeclareNameFormat{default}{%
      \renewcommand*{\multinamedelim}{\addsemicolon\addspace}%
      \usebibmacro{name:last-first}{#1}{#4}{#5}{#7}%
      \usebibmacro{name:andothers}%
    }

    \DeclareNameFormat{editor}{%
      \renewcommand*{\multinamedelim}{\addcomma\addspace}%
      \usebibmacro{name:last-first}{#1}{#4}{#5}{#7}%
      \usebibmacro{name:andothers}%
    }
  }
  {
   % New syntax for flexible back end
    \DeclareNameFormat{default}{%
      \renewcommand*{\multinamedelim}{\addsemicolon\addspace}%
      \nameparts{#1}%
      \usebibmacro{name:family-given}
        {\namepartfamily}
        {\namepartgiveni}
        {\namepartprefix}
        {\namepartsuffix}%
      \usebibmacro{name:andothers}%
    }

    \DeclareNameFormat{editor}{%
      \renewcommand*{\multinamedelim}{\addcomma\addspace}%
      \nameparts{#1}%
      \usebibmacro{name:family-given}
        {\namepartfamily}
        {\namepartgiveni}
        {\namepartprefix}
        {\namepartsuffix}%
      \usebibmacro{name:andothers}%
    }
  }

I’ll deal with the differences in back-ends in another post, but for the present this formulation will keep styles working for everyone.

Biblatex: more versatile shorthand lists

One of the useful features of biblatex is shorthands, which can be defined in a BibTeX database and listed in the bibliography. A long-standing request has been to make this even more powerful by allows several different kinds of shorthand, for example for journal abbreviations, general abbreviations, etc. This ability has now been added to the development version of the package, by generalising shorthands to ‘biblists’. Of course, new features always need testing, so it would be great if interested users would grab the code and try it out!

Extending biblatex to support multiple scripts

As regular readers will know, I’ve taken an interest in biblatex since it was first developed. Since the original author disappeared, I’ve been at least formally involved in maintain the code. So far, that’s been limited to tackling a few tricky low-level TeX issues, but there are some bigger issues to think about.

Philip Kime, lead Biber and biblatex developer, is keen to extend the LaTeX end to supporting multiple scripts. The Biber end is already done (in the ‘burning edge’ version), and writes to the .bbl file in the format:

 \field{form=original,lang=default}{labeltitle}{Title}
 \list{form=original,lang=default}{location}{1}{%
   {Москва}%
 }
 \list{form=romanised,lang=default}{location}{1}{%
   {Moskva}%
 }

However, that presents a big issue: how to do that without breaking every existing style. Supporting scripts means we need an additional argument for a very large number of commands: some of them need to have two optional arguments, and some of them need to be expandable:

\iffieldundef[form=original,lang=default]{....}

Reading (two) optional arguments and working through keyval options expandably is tricky, which is where I come in. The natural way for me to solve the first problem is to use LaTeX3, and the xparse package. However, that’s a big change for biblatex, so before I (and the rest of the biblatex team) go for this I though it would be worth raising the issue and looking for opinions. The alternative is to write the code into biblatex directly, but it’s complicated and as I’ve already done the job once I’m reluctant to do this!

So, what I want to know is ‘What do users think?’ Is it reasonable to require xparse as part of `biblate

Traditional BibTeX styles with biblatex

The biblatex package offers a lot of power in creating complex bibliographies. It also uses its own style files, separate from the ‘traditional’ ones used by BibTeX (plain, unsrt, etc.). In many ways that is a good thing, as the traditional styles are rather limited. However, lots of people rely on the traditional styles, so emulating them would be useful. There’s an open question on the TeX-sx site asking how to do this: it’s very much a ‘marker’, as the person who asked is one of the biblatex experts! Marco Daniel has now taken up the challenge, with a GitHub repo set up for development of a set of styles. They are still a work in progress, but it looks like the question will soon have an answer!

A first pass at a physics style for biblatex

I’ve just uploaded a first attempt at biblatex-phys to CTAN: give it a day or so to be mirrored around the world or pick it up from the BitBucket development site. I’ve based the style on REVTeX, which seems to show that the AIP and APS use more-or-less the same style with a few tiny variations. So I’ve provided one style and used some options to control the output. I’m sure there will be a few rough edges, so at the moment I’ve set the version as 0.9, meaning that there may well be adjustments before a truly usable release. However, I’d hope people pick this up and test it: that’s the only way to improve it!

Working on biblatex styles for physics

A while ago, a question was asked on TeX-sx about biblatex support for the IEEE and AIP bibliography styles. I sorted out an IEEE one pretty quickly: it’s a single style and there is a good demonstration document, so this was not too hard. However, to date I’ve not got the physics side of things sorted. I’ve now been reminded about this, and hopefully have the time available to get on with this. So now all I have to decide is which styles to got for: both the AIP and APS seem pretty obvious, but beyond that I’m not so sure.

Co-ordinating biblatex style development

Bibliographies formatting is complicated, and over the years this has led to the development of a lot of BibTeX styles and tools like custom-bib. For biblatex users the output style can be altered from within LaTeX, and while there is not yet an equivalent of custom-bib for biblatex there is some excellent advice available for developing new styles.

One of the key ideas of  is to make it possible to address the citation styles in fields where BibTeX could never provide the correct tools. That shows up most obviously in humanities, where the position of citations in the output needs to feed back into how they appear. Life gets even more complicated when you consider areas that need specialised fields included in their bibliographies: a classic example is law. There are already developments under-way to help support these situation, for example Biber-based flexible data model. Addressing the needs of style authors properly requires co-ordination between the core biblatex team and the style creators, and between different style authors. The question of how to achieve this communication came up recently on TeX-sx.

There are at least a couple of obvious ways for people to stay in touch with each other. First, where there is a need for a specific function to be added to biblatex, the package GitHub site is the place to go. Adding issues makes sure that they don’t get lost in direct e-mails, and also ensures that anyone with a point of view can comment. Secondly, for more open discuss TUG have a bibliography mailing list. To date, this has not been very heavy in terms of traffic, but it would be an ideal forum to co-ordinate effort. It’s publicly archived and anyone can join, so there is no danger of loosing important comments.

Of course, there are other ways to communicate too! All of the team are active on TeX-sx, and when we can we pick up biblatex issues and add them to the to do list. Direct e-mail works too, and I’m sure some style authors will go for direct communication. The key thing is to use the power of biblatex to make life easier for the end-user.