Automating releases to CTAN

I’ve talked in recent posts about various aspects of creating LaTeX packages, focussing on the dtx format. One thing I’ve been promising to cover is automating the release of material to CTAN. Even for a basic package, there are a few files to sort out (the source, a readme file, the documentation and an ins file). For the documentation, you need to typeset the correct version, include the changes and code index. So even in a simple case, a bit of help from the computer is a good thing: I manage to miss stuff quite happily even with some stuff set up.

What do I mean by automation? Well, there is typesetting to do, files to copy and zip files to create. For Windows users, there are also line endings to worry about: CTAN prefer Unix ones for plain text files. All of that can be rolled up into some kind of script (a shell script on Unix or a batch file on Windows). Unix users also have easy access to the ‘make’ utility. The basic tasks are the same whatever method you go for, but I’m going to assume batch files for Windows and make files for Unix (including Mac OS X).

The aim here is not to have to most sophisticated system possible, but to make life easier. So I’ve not necessarily made every refinement I’ve thought of as some of them make what is going on much less clear. I’d also point out that a lot of the ideas here are ones I’ve adapted from elsewhere, or that have been suggested to me. Not much originality, but again that is not the main point. One thing to point out is that I’ve provided settings for copying dtx, ins, sty and pdf files. Other file types would need to be added, but hopefully there is enough here for the pattern to be clear without over-complicating things. You can always add things to a script so that the do nothing if they are not needed. So the same ideas can be used for packages with different requirements, with only a few basic settings to change.

The two files I’m going to provide both aim to give the same functionality: I work with both Windows and Unix, so I need that. As well as being able to clean out the working directory and make documentation, there are also methods to make a CTAN archive and a TDS one (to send to users for direct installation). Finally, I’ve included a local installation option: useful if you don’t update your TeX system regularly and need your own code to be up to date!

Windows batch files

A batch file on Windows (or indeed a shell script on Unix) is simply a list of commands you could type yourself at the command line, but with some flow control added. Recent versions of Windows include a number of extensions beyond the old DOS capabilities: I’m going to use some of these, but that only rules out very old systems so it should be reasonably safe. If you want to grab the entire file in one go, it’s available here.

One problem is that there is no command line tool for creating zip files installed by default in Windows. I’ve tried a few out, and the best seems to be Info-ZIP. It does a good job of marking up binary and text files, and also includes some abilities to sort out line endings. If it doesn’t work for you, other tools such as the Swiss File Knife do the same thing on a file-by-file basis. Whatever you decide, it’s best to put the support tools on the Windows path somewhere.

@echo off

  if not "%1" == "" goto :init

:help

  echo.
  echo  make clean        - delete all generated files
  echo  make ctan         - create an archive ready for CTAN
  echo  make doc          - typesets documentation
  echo  make localinstall - extract packages
  echo  make tds          - create a TDS-ready archive
  echo  make unpack       - extract packages

  goto :EOF

:init

  setlocal

  rem The name of the package to create should be set here: here, the
  rem example package "demopkg" is in use

  set PACKAGE=demopkg

  rem It is possible to unpack dtx files without needing any extra files, but
  rem some people prefer a separate ins file (or there may be no unpacking
  rem to do). This should be set up here: for a self-extracting dtx the
  rem standard setting is fine.

  set UNPACK=%PACKAGE%.dtx

  rem A list of pdf files to be typeset and included in the archive files
  rem created. The files named here will be typeset (looking for source files
  rem in the order .dtx, .tex, .ltx).

  set INCLUDEPDF=%PACKAGE%

  rem Plain text files to be included in the archives: the .txt extension is
  rem automatically stripped when creating the archive.

  set INLCUDETXT=README

  rem Files to typeset

  rem The settings for cleaning up after compilation are divided into two
  rem parts. AUXFILES are deleted after each (La)TeX run, CLEAN only
  rem when the user calls "make clean"

  set AUXFILES=aux dvi glo gls hd idx ilg ind ist log out toc
  set CLEAN=gz ins pdf sty tex txt zip

  rem Sets the order for searching for source files for pdfs

  set PDFSOURCES=dtx tex

  rem The file types for inclusion in the archive files: note that a CTAN
  rem archive should not contain "unpacked" files. Typeset files and their
  rem sources are not inlcuded here: they are dealt with separately

  set CTANFILES=dtx ins pdf
  set TDSFILES=%CTANFILES% sty 

  rem Locations for building archives

  set CTANROOT=ctan
  set CTANDIR=%CTANROOT%\%PACKAGE%
  set TDSROOT=tds

  cd /d "%~dp0"

:main

  if /i "%1" == "clean"        goto :clean
  if /i "%1" == "ctan"         goto :ctan
  if /i "%1" == "doc"          goto :doc
  if /i "%1" == "help"         goto :help
  if /i "%1" == "localinstall" goto :localinstall
  if /i "%1" == "tds"          goto :tds
  if /i "%1" == "unpack"       goto :unpack

  goto :help

:clean

  echo.
  echo Deleting files

  for %%I in (%CLEAN%) do (
    if exist *.%%I del /q *.%%I
  )

  for %%I in (%TXT%) do (
    if exist %%I del /q %%I
  )

:clean-aux

  for %%I in (%AUXFILES%) do (
    if exist *.%%I del /q *.%%I
  )

  goto :end

:ctan

  call :zip
  if errorlevel 1 goto :EOF

  call :doc
  if errorlevel 1 goto :EOF

  echo.
  echo Creating archive

  for %%I in (%SOURCES%) do (
    xcopy /q /y %%I "%CTANDIR%\" > nul
  )
  for %%I in (%CTANFILES%) do (
    xcopy /q /y *.%%I "%CTANDIR%\" > nul
  )
  for %%I in (%INLCUDETXT%) do (
    xcopy /q /y %%I.txt "%CTANDIR%\" > nul
    ren "%CTANDIR%\%%I.txt" %%I
  )

  pushd "%CTANROOT%"
  %ZIPEXE% %ZIPFLAG% %PACKAGE%.zip .
  popd
  copy /y "%CTANROOT%\%PACKAGE%.zip" > nul

  rmdir /s /q %CTANROOT%

  goto :end

:doc 

  call :unpack

  set SOURCES=

  for %%I in (%INCLUDEPDF%) do (
    for %%J in (%PDFSOURCES%) do (
      echo.
      if exist %%I.%%J call :typeset-%%J %%I.%%J
    )
  )

  goto :clean-aux

:file2tdsdir

  set TDSDIR=

  if /i "%~x1" == ".dtx" set TDSDIR=source\latex\%PACKAGE%
  if /i "%~x1" == ".ins" set TDSDIR=source\latex\%PACKAGE%
  if /i "%~x1" == ".pdf" set TDSDIR=doc\latex\%PACKAGE%
  if /i "%~x1" == ".sty" set TDSDIR=tex\latex\%PACKAGE%
  if /i "%~x1" == ".txt" set TDSDIR=doc\latex\%PACKAGE%

  goto :EOF

:localinstall

  call :unpack

  echo.
  echo Installing

  if not defined TEXMFHOME set TEXMFHOME=%USERPROFILE%\texmf

  for %%I in (%TDSFILES%) do (
    call :localinstall-int *.%%I
  )

  goto :end

:localinstall-int

  call :file2tdsdir %1

  if defined TDSDIR (
    xcopy /q /y %1 "%TEXMFHOME%\%TDSDIR%\" > nul
  ) else (
    echo Unknown file type "%~x1"
  )

  goto :EOF

:tds

  call :zip
  if errorlevel 1 goto :EOF

  call :doc
  if errorlevel 1 goto :EOF

  echo.
  echo Creating archive

  for %%I in (%SOURCES%) do (
    call :tds-int %%I
  )
  for %%I in (%TDSFILES%) do (
    call :tds-int *.%%I
  )
  for %%I in (%INLCUDETXT%) do (
    call :tds-txt %%I
  )

  pushd "%TDSROOT%"
  %ZIPEXE% %ZIPFLAG% %PACKAGE%.tds.zip .
  popd
  copy /y "%TDSROOT%\%PACKAGE%.tds.zip" > nul

  rmdir /s /q "%TDSROOT%"

  goto :end

:tds-int

  call :file2tdsdir %1

  if defined TDSDIR (
    xcopy /q /y %1 "%TDSROOT%\%TDSDIR%\" > nul
  ) else (
    echo Unknown file type "%~x1"
  )

  goto :EOF

:tds-txt

  call :file2tdsdir %1.txt

  if defined TDSDIR (
    xcopy /q /y %1.txt "%TDSROOT%\%TDSDIR%\" > nul
    ren "%TDSROOT%\%TDSDIR%\%1.txt" %1
  ) else (
    echo Unknown file type "%~x1"
  )

  goto :EOF

:typeset-dtx

  echo Typesetting %1

  pdflatex -interaction=nonstopmode -draftmode "\AtBeginDocument{\OnlyDescription}\input %1" > nul
  if ERRORLEVEL 1 (
    echo ! Compilation failed
  )

  makeindex -q -s gglo.ist -o %~n1.gls %~n1.glo > nul
  makeindex -q -s gind.ist -o %~n1.ind %~n1.idx > nul
  pdflatex -interaction=nonstopmode "\AtBeginDocument{\OnlyDescription} \input %1" > nul
  pdflatex -interaction=nonstopmode "\AtBeginDocument{\OnlyDescription} \input %1" > nul

  goto :EOF

:typeset-tex

  echo Typesetting %1

  set SOURCES=%SOURCES% %1

  pdflatex -interaction=nonstopmode -draftmode %1 > nul
  if ERRORLEVEL 1 (
    echo ! Compilation failed
  )
  pdflatex -interaction=nonstopmode %1 > nul
  pdflatex -interaction=nonstopmode %1 > nul

  goto :EOF

:unpack

  echo.
  echo Unpacking files

  for %%I in (%UNPACK%) do (
    tex %%I > nul
  )

  goto :end

:zip 

  if not defined ZIPFLAG set ZIPFLAG=-r -q -X -ll

  if defined ZIPEXE goto :EOF

  for %%I in (zip.exe "%~dp0zip.exe") do (
    if not defined ZIPEXE if exist %%I set ZIPEXE=%%I
  )

  for %%I in (zip.exe) do (
    if not defined ZIPEXE set ZIPEXE="%%~$PATH:I"
  )

  if not defined ZIPEXE (
    echo.
    echo This procedure requires a zip program,
    echo but one could not be found.
    echo
    echo If you do have a command-line zip program installed,
    echo set ZIPEXE to the full executable path and ZIPFLAG to the
    echo appropriate flag to create an archive.
    echo.
  )

  goto :EOF

:end

  shift
  if not "%1" == "" goto :main

Most of the ideas here should be pretty straight-forward. The clever part is :file2tdsdir, which I have to say was not my idea at all! It allows the batch file to ‘know’ which type of files go where, so that you only need the information once for use in several places.

To use the file, just alter the settings at the beginning. The pattern should be pretty clear, and most of the rest of the code (for example, :file2tdsdir for correctly placing files) is also quite obvious.

Unix make files

Unix make files work somewhat differently to shell scripts. Each entry is a ‘target’, which is a file to create. I’m not going to explain in detail how they work, but in essense there are a series of fake ‘files’ which are the names of the settings you send to make (for example, make ctan needs a target called ctan). As with the batch file, there are a series of blanks to fill in here to customise things. I’m also sticking with the idea that things are pretty basic: a dtx file, a sty file and some documentation, plus perhaps one or more example tex files. Hopefully the idea is pretty clear. By keeping as much as possible in variables, the idea is to avoid needing to change the bulk of the file to move from one package to another. As with the batch file, the entire thing is available here. to download.

################################################################
################################################################
# Makefile for demopkg                                         #
################################################################
################################################################

################################################################
# Default with no target is to give help                       #
################################################################

help:
	@echo ""
	@echo " make clean               - clean out test directory"
	@echo " make ctan                - create a CTAN-ready archive"
	@echo " make doc                 - typeset documentation"
	@echo " make localinstall        - install files in local texmf tree"
	@echo " make tds                 - create a TDS-ready archive"
	@echo " make unpack              - extract packages"
	@echo ""

##############################################################
# Master package name                                        #
##############################################################

PACKAGE = demopkg

##############################################################
# Directory structure for making zip files                   #
##############################################################

CTANROOT := ctan
CTANDIR  := $(CTANROOT)/$(PACKAGE)
TDSDIR   := tds

##############################################################
# Data for local installation and TDS construction           #
##############################################################

INCLUDEPDF  := $(PACKAGE)
INCLUDETEX  :=
INCLUDETXT  := README
PACKAGEROOT := latex/$(PACKAGE)

##############################################################
# Details of source files                                    #
##############################################################

DTX      = $(PACKAGE).dtx
DTXFILES = $(PACKAGE)
UNPACK   = $(PACKAGE).dtx

##############################################################
# Clean-up information                                       #
##############################################################

AUXFILES = \
	aux  \
	glo  \
	gls  \
	hd   \
	idx  \
	ilg  \
	ind  \
	log  \
	out  \
	tmp  \
	toc  

CLEAN = \
	gz  \
	ins \
	pdf \
	sty \
	tex \
	txt \
	zip 

################################################################
# File buiding: default actions                                #
################################################################

%.pdf: %.dtx
	NAME=`basename $< .dtx` ; \
	echo "Typesetting $$NAME" ; \
	pdflatex -draftmode -interaction=nonstopmode "\AtBeginDocument{\OnlyDescription} \input $<" &> /dev/null ; \
	if [ $$? = 0 ] ; then  \
	  makeindex -s gglo.ist -o $$NAME.gls $$NAME.glo &> /dev/null ; \
	  makeindex -s gind.ist -o $$NAME.ind $$NAME.idx &> /dev/null ; \
	  pdflatex -interaction=nonstopmode "\AtBeginDocument{\OnlyDescription} \input $<" &> /dev/null ; \
	  pdflatex -interaction=nonstopmode "\AtBeginDocument{\OnlyDescription} \input $<" &> /dev/null ; \
	else \
	  echo "  Complilation failed" ; \
	fi ; \
	for I in $(AUXFILES) ; do \
	  rm -f $$NAME.$$I ; \
	done

################################################################
# User make options                                            #
################################################################

.PHONY = \
	clean        \
	ctan         \
	doc          \
	localinstall \
	tds          \
	unpack

clean:
	echo "Cleaning up"
	for I in $(AUXFILES) $(CLEAN) ; do \
	  rm -f *.$$I ; \
	done
	rm -rf $(CTANROOT)/
	rm -rf $(TDSDIR)/

ctan: doc
	echo "Creating CTAN archive"
	mkdir -p $(CTANDIR)/
	rm -rf $(CTANDIR)/*
	cp -f *.dtx $(CTANDIR)/ ; \
	cp -f *.ins $(CTANDIR)/ ; \
	for I in $(INCLUDEPDF) ; do \
	  cp -f $$I.pdf $(CTANDIR)/ ; \
	done ; \
	for I in $(INCLUDETEX); do \
	  cp -f $$I.tex $(CTANDIR)/ ; \
	done ; \
	for I in $(INCLUDETXT); do \
	  cp -f $$I.txt $(CTANDIR)/; \
	  mv $(CTANDIR)/$$I.txt $(CTANDIR)/$$I ; \
	done ; \
	cd $(CTANDIR) ; \
	zip -ll -q -r -X $(PACKAGE).zip .
	cp $(CTANDIR)/$(PACKAGE).zip ./
	rm -rf $(CTANROOT)/

doc: $(foreach FILE,$(INCLUDEPDF),$(FILE).pdf)

localinstall: unpack
	echo "Installing files"
	TEXMFHOME=`kpsewhich --var-value=TEXMFHOME` ; \
	rm -rf $$TEXMFHOME/tex/$(PACKAGEROOT)/*.* ; \
	cp *.sty $$TEXMFHOME/tex/$(PACKAGEROOT)/ ; \
	texhash &> /dev/null

tds: doc
	echo "Creating TDS archive"
	mkdir -p $(TDSDIR)/
	rm -rf $(TDSDIR)/*
	mkdir -p $(TDSDIR)/doc/$(PACKAGEROOT)/
	mkdir -p $(TDSDIR)/tex/$(PACKAGEROOT)/
	mkdir -p $(TDSDIR)/source/$(PACKAGEROOT)/
	cp -f *.dtx $(TDSDIR)/source/$(PACKAGEROOT)/ ; \
	cp -f *.ins $(TDSDIR)/source/$(PACKAGEROOT)/ ; \
	for I in $(INCLUDEPDF) ; do \
	  cp -f $$I.pdf $(TDSDIR)/doc/$(PACKAGEROOT)/ ; \
	done ; \
	cp -f *.sty $(TDSDIR)/tex/$(PACKAGEROOT)/ ; \
	for I in $(INCLUDETEX); do \
	  cp -f $$I.tex $(TDSDIR)/doc/$(PACKAGEROOT)/ ; \
	done ; \
	for I in $(INCLUDETXT); do \
	  cp -f $$I.txt $(TDSDIR)/doc/$(PACKAGEROOT)/ ; \
	  mv $(TDSDIR)/doc/$(PACKAGEROOT)/$$I.txt $(TDSDIR)/doc/$(PACKAGEROOT)/$$I ; \
	done
	cd $(TDSDIR) ; \
	zip -ll -q -r -X $(PACKAGE).tds.zip .
	cp $(TDSDIR)/$(PACKAGE).tds.zip ./
	rm -rf $(TDSDIR)

unpack:
	echo "Unpacking files"
	for I in $(UNPACK) ; do \
	  tex $$I &> /dev/null ; \
	done

You’ll see that on Unix (where we have more tools definitely available) some things are easier. That also applies to finding the local tex root: TeX Live will almost certainly be the TeX system installed, so its tools can be called on to collect the data needed. Both of the above should work with the demonstration package code I talked about last week.