Debian Haskell Package Consistency Test Suite
=============================================


This repository contains the file packages.txt, which lists Haskell packages
and the version that Debian ships, or plans to ship. It also carries additional
information such as patches and whether to run the test suite.

File format
-----------

The file `packages.txt` is a plain text file, with one line per package. Each
line has the format

    <packagename> <version> <attributes>* [# <comment>]

where attributes and the comment are optional.

The package name and version must be Hackage names and version (i.e.
potentially upper case, no Debian revision in the version).

The following flags are supported:

 * ignore:
   This package is expected to be present but otherwise (versions etc.) ignored.
 * meta:
   This package is thrown into the resolver, but it is not compared against the
   archive.
 * notest:
   The test suite of this package is not run, i.e. the test suite dependencies
   are ignored.
 * binary=<pkg>:
   In order to find the corresponding Debian binary package, use this name.
   This is required for Haskell programs that do not build a libghc-foo-dev
   package with a Ghc-Package field in the Packages files.
 * binary:
   As above, but with the Cabal package name taken as <pkg>.
 * group=<group>:
   Indicates that this package should be checked together with other packages
   from the same group, but not packages from different groups. This is used
   when Debian's policy of having at most one version of each Hackage package
   is relaxed, e.g. for packages bundled with GHC.
 * obsolete:
   Indicates that this package should ''not'' be used, but that it is still in
   the archive. Used until the FTP team acts upon a RM request.
 * avoid:
   Indicates that this package should ''not'' be used. Used for optional
   dependencies where we want to force the other alternative.
 * ahead:
   Indicates that this package is known and allowed to be newer than specified
   in the tracked LTS release.
 * behind:
   Indicates that this package is known and allowed to be older than specified
   in the tracked LTS release.
 * key:
   Indicates that this package is a key package.

The tools will warn about unknown flags.

Supporting files
----------------

In addition to `packages.txt`, the follwing files are taken into account:

 * `additional-cabals/<pkg>-<version>.cabal`
   Adds the given files to the internal “repository”, possibly overwriting
   exiting packages.
   This can be used for packages not on hackage, or when the cabal-file editing
   feature on hackage has been used. In that case, the unmodified cabal file
   ought to be added here.
 * `patches/<pkg>/<version>/
   A directory of patches, in the same format as `debian/patches` (so that the
   files can be taken unmodified from there). The patches are applied to the
   Cabal files of the “internal“ repository.
 * `lts.config`
   The version constraings of the tracked LTS release. Can be updated by, for
   example:

       wget https://www.stackage.org/snapshot/lts-2.10/cabal.config -O lts.config

Running the package plan
------------------------

The script has a few dependencies, so run

    apt-get install libfile-slurp-perl libipc-run-perl \
      liblist-moreutils-perl libtext-patch-perl libdpkg-perl \
      cabal-install dctrl-tools patchutils

To test the package plan, run

    cabal update
    ./test-packages.pl

this tool will

 * read `packages.txt`.
 * read the hackage index from `~/.cabal/packages/hackage.haskell.org/00-index.tar"
   (hence the need to run `cabal update`).
 * read the LTS version ranges from `lts.config`.
 * read the Debian package index using `grep-aptavail`.
 * report mismatches (e.g. packages missing or too old in `packages.txt`).
 * creates a custom view on hackage, using the supporting files mentioned above.
 * for every group, runs `cabal install` with a very very long command line to
   check if the package selection constitutes a valid build plan.
   The command lines can be reviewed in the files `cabal-cmd-<group>.sh`
 * reports on the output (e.g. version mismatches, additional dependendencies pulled in).
 * writes statistics to `stats.csv`, using to create nice plots on jenkins.debian.net.

The script supports some options

 * `--allow-upgrade`:
   In that case it allows `cabal` not only the precise version specified here,
   but also newer versions. In theory, this should make it easier to figure out
   what can be upgraded, or what has to be upgraded, given a certain change. In
   practice, the runtime of `cabal` with these options is prohibitive.
 * `--assume-lts`:
   Overrides the version specified in `packages.txt` by the version specified
   in `lts.conf` (if there is one, and the package is not marked `ahead`).
   Useful to find out what non-LTS package break if we follow a newer LTS
   release.
 * `--only-key-packages`:
   Passes only packages marked as key packages, plus all LTS packages, to
   cabal-install. This will report packages that are neither key packages nor
   dependencies of such, and give hints about what should be cleaned up.

The following Dockerfile may help test in a reproducible environment

    FROM debian:sid

    ENV DEBIAN_FRONTEND="noninteractive"
    RUN apt-get update
    RUN apt-get install --yes libfile-slurp-perl libipc-run-perl \
     liblist-moreutils-perl libtext-patch-perl libdpkg-perl \
     dctrl-tools patchutils git cabal-install ghc

    # Replace the following with
    # git clone <URI> -b <BRANCH_NAME>
    # to test a specific branch.
    RUN git clone https://salsa.debian.org/haskell-team/package-plan.git
    RUN cabal update
    WORKDIR package-plan
    CMD ./test-packages.pl

Jenkins integration
-------------------

There is a Jenkins job at
https://jenkins.debian.net/view/haskell/job/haskell-package-plan/
for this. It is triggered upon every commit, plus once per day.

Generally, this should always succeed. The only unavoidable reasons for this to
temporary fail are
 * Obsolete packages are removed by the FTP-Team, until the package is removed
   from `packages.txt`
 * Cabal files are edited on hackage.

It also creates a nice plot at
https://jenkins.debian.net/view/haskell/job/haskell-package-plan/plot/


Other tools here
----------------

`update-suggestions`. To build and run, use

    $ ghc --make -O update-suggestions
    $ ./update-suggestions

Following packages are required for compilation:

 * ghc (obliviously)
 * libghc-attoparsec-dev
 * libghc-process-extras-dev

This program finds packages with newer minor versions. These are likely
non-disruptive to upgrade, so if you have a few moments to spare, you can work
on these and push the “upgradeable” plot down.


How do I... upgrade a package, or add a new one
-----------------------------------------------

Change the entry in `packages.txt`. Run `./test-packages.pl` and see what
breaks. If stuff breaks, add new packages or update existing packages as
required. Add patches as required. Repeat.

Only when the plan is sound, start uploading.


How do I... conveniently edit the patches
-----------------------------------------

One quick way is

    $ cd /tmp
    $ cabal unpack foo
    $ cd foo-1.2.3/
    $ ln -s .../package-plan/patches/foo/1.2.3 patches
    $ quilt push / quilt edit / quilt refresh etc.

This also works great to add new patches.

Special cases
-------------

The following packages are rendered empty when built against our
`packages.txt`.  Our CDBS code can't deal with that, so dependencies
on the following should be patched out of any cabal files in which
they appear.

- bytestring-builder
- nats
- mtl-compat