How to deploy launchpad-buildd

In Canonical’s datacentre environments, launchpad-buildd is deployed as a .deb package installed in a fleet of VMs. To upgrade it, we need to rebuild the VM images.

Each environment uses its own PPA and management environment:

Environment

PPA and management environment

production

ppa:launchpad/ubuntu/buildd prod-launchpad-vbuilders@is-bastion-ps5

qastaging

ppa:launchpad/ubuntu/buildd-staging stg-vbuilder-qastaging@launchpad-bastion-ps5

These instructions use various tools from ubuntu-archive-tools (copy-package and manage-builders).

Testing on qastaging

  1. Ensure everything has been merged to master.

  2. Check that the recipe has built successfully (you can start a build manually if required), and that the resulting package has been published in the Launchpad PPA.

  3. Run copy-package --from=ppa:launchpad/ubuntu/ppa --suite=jammy --to=ppa:launchpad/ubuntu/buildd-staging -b launchpad-buildd (from ubuntu-archive-tools) to copy the current version of launchpad-buildd to the deployment PPA (jammy here refers to the series being used on qastaging builder instances).

  4. Wait for PPA publishing to complete.

  5. Run mojo run -m manifest-rebuild-images in the management environment (stg-vbuilder@launchpad-bastion-ps5) to start rebuilding images. After a minute or so, juju status glance-simplestreams-sync-\* will show “Synchronising images”; once this says “Sync completed”, images have been rebuilt.

  6. Builders will get the new image after they finish their next build (or are disabled) and go through being reset. Since qastaging’s build farm is typically mostly idle, you can use manage-builders -l qastaging --reset to reset all builders and force them to pick up the new image (from ubuntu-archive-tools).

  7. Perform QA on qastaging until satisfied, see How to perform QA.

Releasing to production

  1. Create a new release branch, e.g. release-213, based on master.

  2. Run DEBEMAIL="<email address>" DEBFULLNAME="<name>" dch -rD focal. The later recipe build will prepend the correct preamble for each Ubuntu release.

  3. Create a commit with a title like releasing package launchpad-buildd version 213, push this branch and open a merge proposal with a title like Release version 213 for review.

  4. Once the release branch has merged to master, tag the release commit (e.g. git tag 213 && git push origin 213).

  5. Check that the recipe has built successfully (you can start a build manually if required), and that the resulting package has been published in the Launchpad PPA.

  6. Run copy-package --from=ppa:launchpad/ubuntu/ppa --suite=focal --to=ppa:launchpad/ubuntu/buildd -b launchpad-buildd to copy the current version of launchpad-buildd to the deployment PPA.

  7. Wait for PPA publishing to complete.

  8. Run ssh prod-launchpad-vbuilders@is-bastion-ps5.internal /home/prod-launchpad-vbuilders/scripts/rebuild-images.sh from the staging management environment (stg-vbuilder@launchpad-bastion-ps5) to start rebuilding images.

  9. Once the new image is rebuilt, which normally takes on the order of 15-60 minutes depending on the architecture, builders will get the new image after they finish their next build (or are disabled) and go through being reset. As a result, manage-builders -v should start showing the new version over time.

  10. Wait for the new version to appear for at least one builder in each region and architecture. If this doesn’t happen after 90 minutes, then ask IS for assistance in investigating; they can start by checking juju status in prod-launchpad-vbuilders@is-bastion-ps5.internal.

  11. Once the updated version is visible for at least one builder in each region and architecture, build farm administrators can use manage-builders --virt --idle --builder-version=<old-version> --reset to reset idle builders, thereby causing builders that haven’t taken any builds recently to catch up.

  12. Close any bugs fixed by the new release.