How to deploy launchpad-buildd¶
In Canonical’s datacentre environments, launchpad-buildd is deployed as a
.deb
package installed in a fleet of VMs. To upgrade it, we need to
rebuild the VM images.
Each environment uses its own PPA and management environment:
Environment |
PPA and management environment |
---|---|
ppa:launchpad/ubuntu/buildd
|
|
ppa:launchpad/ubuntu/buildd-staging
|
These instructions use various tools from ubuntu-archive-tools (copy-package
and
manage-builders
).
Testing on qastaging¶
Ensure everything has been merged to master.
Check that the recipe has built successfully (you can start a build manually if required), and that the resulting package has been published in the Launchpad PPA.
Run
copy-package --from=ppa:launchpad/ubuntu/ppa --suite=jammy --to=ppa:launchpad/ubuntu/buildd-staging -b launchpad-buildd
(fromubuntu-archive-tools
) to copy the current version of launchpad-buildd to the deployment PPA (jammy
here refers to the series being used on qastaging builder instances).Run
mojo run -m manifest-rebuild-images
in the management environment (stg-vbuilder@launchpad-bastion-ps5
) to start rebuilding images. After a minute or so,juju status glance-simplestreams-sync-\*
will show “Synchronising images”; once this says “Sync completed”, images have been rebuilt.Builders will get the new image after they finish their next build (or are disabled) and go through being reset. Since qastaging’s build farm is typically mostly idle, you can use
manage-builders -l qastaging --reset
to reset all builders and force them to pick up the new image (fromubuntu-archive-tools
).Perform QA on qastaging until satisfied, see How to perform QA.
Releasing to production¶
Create a new release branch, e.g.
release-213
, based on master.Run
DEBEMAIL="<email address>" DEBFULLNAME="<name>" dch -rD focal
. The later recipe build will prepend the correct preamble for each Ubuntu release.Create a commit with a title like
releasing package launchpad-buildd version 213
, push this branch and open a merge proposal with a title likeRelease version 213
for review.Once the release branch has merged to master, tag the release commit (e.g.
git tag 213 && git push origin 213
).Check that the recipe has built successfully (you can start a build manually if required), and that the resulting package has been published in the Launchpad PPA.
Run
copy-package --from=ppa:launchpad/ubuntu/ppa --suite=focal --to=ppa:launchpad/ubuntu/buildd -b launchpad-buildd
to copy the current version of launchpad-buildd to the deployment PPA.Run
ssh prod-launchpad-vbuilders@is-bastion-ps5.internal /home/prod-launchpad-vbuilders/scripts/rebuild-images.sh
from the staging management environment (stg-vbuilder@launchpad-bastion-ps5
) to start rebuilding images.Once the new image is rebuilt, which normally takes on the order of 15-60 minutes depending on the architecture, builders will get the new image after they finish their next build (or are disabled) and go through being reset. As a result,
manage-builders -v
should start showing the new version over time.Wait for the new version to appear for at least one builder in each region and architecture. If this doesn’t happen after 90 minutes, then ask IS for assistance in investigating; they can start by checking
juju status
inprod-launchpad-vbuilders@is-bastion-ps5.internal
.Once the updated version is visible for at least one builder in each region and architecture, build farm administrators can use
manage-builders --virt --idle --builder-version=<old-version> --reset
to reset idle builders, thereby causing builders that haven’t taken any builds recently to catch up.Close any bugs fixed by the new release.