Abstract Debian makes a "stable release" from time to time. This article shows how the background work looks like, what the major issues are and which improvements are planned; the intended audience are people knowing Debian closely. The author is member of the Debian Release Team. More information is available at http://release.debian.org/ Releasing means Sarge, the recent stable release of Debian, was published at beginning of June 2005 after about three years of hard development. The packages included in that release began their life in "unstable" (border cases are discussed later on). If they were fit enough, they were automatically copied to testing; "fit enough" is determined by a script called britney that pushes packages from unstable to testing that survived a minimum time (usually ten days), are not too broken themselves and do not break other packages in testing. Also, for packages not meant for the next stable release, there exists a suite called "experimental"; packages in that suite stay where they are and provide a playground for changes. The task of release management in Debian is to make sure that Debian can actually release. This sounds like a trivial statement. But in fact, that is our standard for all decisions. It is a hard fight to make sure that testing does not become more buggy. Also, a lot of other people need to be kept in the loop. Debian can only release if almost all people working on Debian believe it. That includes speaking with all the major teams (ftp-masters, installer, tool chain, (base) package maintainers, X, and so on). Behind the curtain Our most famous tool is "britney", the release team's working tool that updates testing from unstable. Other tools include the lists of release critical bugs, both at http://bugs.debian.org/release-critical and http://bts.turmzimmer.net/details.php. Also, there are scripts to track the security status on http://merkel.debian.org/~joeyh/testing-security.html. Of course, also the "normal" tools like madison, grep-dctrl and debdiff are used very much (and partly with small helper scripts). Britney starts considering packages after 10, 5 or 2 days, depending on how urgent the uploads since the last testing migration are. A package needs to be in sync on all architectures, and also be less buggy than the package currently in testing. Also, they must not break another package in testing. Britney is a python script, mixed with c-code, and available at http://ftp-master.debian.org/testing/update_out_code/ Another way to get a package in testing is via testing-proposed-updates; that requires a release team member to manually approve the package. For sanity reasons (as the new package is more or less untested), it is preferred that packages go in via unstable. (Technically, also this part is taken care of by britney.) Britney is run shortly after the daily install on ftp-master, and uses the packages lists of testing and unstable as starting point. Britney produces a file suitable to synchronise the database with by heidi, an ftp-master's tool; the updated database can however be seen only after the next day's installer run. Also, britney produces the so called "testing excuses" and "testing output" that indicates why a package did (not) make it to testing. These explanations are deciphered on http://bjorn.haxx.se/debian/. The release team can influence the result of britney by usage of hints. They especially allow to remove (buggy) packages, freeze and thaw packages and ignore some of the usual preconditions for a testing migration. Britney can be re-run during the day if the needs arise. Release standards A package can only be part of the next stable release if it doesn't contain critical nor grave bugs (makes the package or the system unusable or introduces a security hole). It needs also to be releasable from the maintainers point of view, and it must meet the release standards. The canonical list of release standards is available from http://release.debian.org/. Why managing Woody has 11 different kernel source packages. Sarge had at one time 17 different kernel source packages. Together with the kernel team, the release team was able to reduce the number of source packages to three, one per 2.2, 2.4 and 2.6 kernel major releases. This is quite important to be able to make security support. It also made the life of the installer team easier. However, that showed also an quite important problem - there are currently no maintainers for two architectures, which meant round trip times of about two months for security updates via unstable. Problems and learned lessons One problem noticed quite often were packages depending on each other for the testing update. That is no problem as long as the number of packages is relatively small. Together with changes like shlib bumps and buildd problems, this can generate a large cluster of packages all depending on each other. This will be partly solved by a new feature of britney which allows the release team to decouple library name changes from the testing migration. The strategy to deal with such issues is to get wise of problems as soon as possible (and while the problems are still small enough). Some bigger tools include binary-only NMUs, coordination with maintainers, the ftp-team and the buildd maintainers. Some rule that all maintainers should know by heart is to never upload a package to unstable that is not mean for testing, and especially never to upload a package if the release teams asks you to not do it. Also, maintainers should allow their package to go to testing before they upload the next version. It has happened that a maintainer broke a large transition with an bad upload on day 9 of 10 of his package waiting for testing migration. For stabilising the installer, base was frozen in August 2004. Looking from todays point of view, that freeze was way too long for both the maintainers and the release team. The freeze should be restricted to a relative short interval directly prior to release. Also, the number of frozen packages can be probably reduced, as only the packages with a direct effect on the installer need to be frozen. However, probably some usual build tools should be added to the frozen list as well kernel images. That would prevent major breakage like we experienced with a late cdbs changes this time. The release teams goal is to have all issues fixed in testing when testing becomes stable. However, the current bug tracking system doesn't really support this. That together with an insane amount of requests from maintainers for freeze exception brought us into releasing Sarge with more bugs than necessary. The deployment of the version tracking support in the bug tracking system will be a great help for the next stable release. We also learned that we need more QA. That relates to packages as well as to e.g. the produced CD images. Though this is not really release team's work area, the release could profit quite much from it. One issue that costed us quite often time were issues with single architectures. For the release team, it is quite painful if there is e.g. only one sparc buildd. If anything happens to that machine or its Internet connection, the release can easily be taken aback by weeks - and this has happened in real. Also, having an kernel maintainer for each and every arch is required. The release team was forced to work very much on specific issues for single architectures, like organising people to move a machine from one place to another with a better Internet connectivity. However, that can't be the task of the release team. For etch, there have been discussions on removing architectures which create a major pain for the release team. Even as of Sarge, one sub architecture has been dropped, the support for 80386 processors. Heading towards etch The preparations for the next stable release called etch have already been started. For etch, a fair time line was worked out which will allow us to release 15 to 18 months after the release of Sarge. To make this realistic, the release team tried to minimise the number of release blockers. A lot more wishes became "pet release goals" which means they will not be allowed to block etch, though it would be really nice to get them also be done in time for etch.