The builds don't cover all the possible configurations. If you get a bug report
about a compile error in a particular configuration, try setting up a build to
make sure that it doesn't show up again if it has been fixed.
The document here talks about the powers of a
build Czar.
The Build Czar is empowered to set up more builds on his own for his
convenience. This
page has a step by step instructions on how to do that.
The build czar can get the build configuration by looking at the config portion
of the scoreboard.
Pro-active involvement by the build czar is necessary. Being a pro-active
build czar requires monitoring the git repository carefully and responding
quickly to suspected changes to keep the repo stays stable.
After an email from the Coverity scan project check the Coverity project and
check that there are no new errors reported. If so, contact the person who
introduced the error.
Check the daily test statistics email to determine any drop in test failures.
Do note that when there are a lot of compile errors the test results are really
unusable.
Recipe for Cutting a Minor Kit
The build czar is also in charge for the release of the micro. Cutting a micro is
as simple as cutting butter if things go well. Here is the procedure followed
while cutting a micro:
-
The whole process takes somewhere between 2-3 hours.
-
We suggest you take advantage of GNU Screen so that even if your SSH session is
interrupted, the cutting process can continue. This command must be installed on
the machines we use to cut the release.
-
type
screen
to start screen.
-
execute commands as normal. Note that Ctrl-A is special in screen, so you need
to type Ctrl-A-A to send a Ctrl-A to the shell
-
should your session be interrupted, reconnect and type
screen -x
-
when finished, just type exit twice
-
After login check that you can,
ssh bczar@download.dre.vanderbilt.edu
to ensure that this succeeds. If not fix the problem, if ok exit again back to
-
Prior to starting this, gather aggregate release notes from all developers.
This is usually in the form of an email plea asking to update all NEWS files in
the archive. These NEWS files are used as part of the release notes for the release.
-
Make sure your release system has all the needed tools. This can be achieved on Fedora
using:
yum install perl screen automake doxygen bzip2 tar gzip openssh graphviz zip libtool GitPython python3
yum update
or on OpenSuSE
zypper install perl screen automake doxygen bzip2 tar gzip openssh graphviz zip libtool python-gitpython python3
zypper update
If you want to perform a full build with qt support, than run:
yum install deltarpm ntp rubygem-rmagick bison xerces-c-devel psmisc yum-utils gdb unzip glibc-devel libasan bison redhat-lsb perl-Pod-Usage rubygems clang make patch libcgroup-devel ant setuptool system-config-network-tui system-config-firewall-tui lcov gnuplot java-1.8.0-openjdk perl screen automake doxygen bzip2 tar gzip openssh graphviz zip libtool gcc-c++ boost-devel valgrind openssl-devel gcc qt4 fltk-devel bzip2-devel rsync openssl lzo-devel zziplib-devel acpid acpi nfs-utils java xerces-c xerces-c-devel mc qt qt-devel icecream ruby ruby-devel lksctp-tools-devel git telnet GitPython NetworkManager wget mailx python3
For some optional i686 packages run
yum install openssl-devel.i686 xerces-c-devel.i686 glibc-devel.i686 glibc.i686 libstdc++.i686 ncurses-libs.i686 zlib.i686
-
Setup your ssh access to github according to the github ssh documentation
-
Checkout a new workspace on a Fedora system with the last public release and with
all patches installed.
-
The best place to create the workspace is under
/export/anduriltmp/bczar
(if you are on anduril). Don't
use the home directory itself, it is an NFS share and not really fast.
-
Checkout like this:
-
git clone https://github.com/DOCGroup/ACE_TAO.git ACE_TAO
-
git clone https://github.com/DOCGroup/MPC.git MPC
-
Set $DOC_ROOT to point to the new workspace you checked out.
-
Change directories to $DOC_ROOT
-
Checkout the correct branch you want to release on, the default is
master
but you can also check out an different branch ace6tao2
for the ACE 6.x/TAO 2.x
releases.
-
Tag the release by executing
ACE/bin/make_release.py --micro --update --tag --push
This will only take a couple minutes to complete and once done successfully,
you can carry on with BOTH creating the kits and generating the doxygen
documentation in parallel. NOTE that --micro
should be replaced
with --minor
or --major
as appropriate.
After the repository has been tagged check each file listed below to make
sure version numbers are updated as expected.
In the EXTREMELY unlikely event that something goes wrong during the
tagging of the repo, the following files must be returned to the state
they were in before the release process started and then checked back into git:
- ACE/PROBLEM-REPORT-FORM
- ACE/VERSION.txt
- TAO/PROBLEM-REPORT-FORM
- TAO/VERSION.txt
- TAO/tao/Version.h
- ace/Version.h
The tag will also need to be removed (both in Middleware and MPC):
ACE+TAO-X_Y_Z (where X is the ACE Major version number, and Y & Z are the
Minor and Micro release numbers of the release that is to be restarted).
E.g.:
cd ACE_TAO && git tag -d ACE+TAO-X_Y_Z
cd MPC && git tag -d ACE+TAO-X_Y_Z
Note that this only needs to be done if the tagging fails. If
kit creation fails, simply restart that process.
-
Create the kits by executing
ACE/bin/make_release.py --kit
This will take somewhere around 1 hours to complete.
-
These commands only tags and creates the kits for the software itself, not
documentation, this can be started in parallel with this activity.
-
The kits end up in $DOC_ROOT/packages
To summarize, the following is a transcript of the steps up to this point
executing successfully:
$ ssh ..
No default printer
screen
cd $HOME
rm -rf DOC_ROOT
mkdir DOC_ROOT
cd DOC_ROOT
export DOC_ROOT=$PWD
git clone https://github.com/DOCGroup/ACE_TAO.git ACE_TAO
git clone https://github.com/DOCGroup/MPC.git MPC
cd ACE_TAO
git checkout master
ACE/bin/make_release.py --micro --update --verbose
ACE/bin/make_release.py --micro --tag --verbose
ACE/bin/make_release.py --micro --push --verbose
ACE/bin/make_release.py --kit
Feel free to cut and paste with suitable edits.
-
The packages end up by default under $DOC_ROOT/package-<PID>, you can
copy them to the webserver using the following commands. (Note that <PID>
needs to be the numerical pid of the process that created the kit, use
ls -ald
to determine the correct filename.) At the moment you execute these commands
all users can download these packages.
scp $DOC_ROOT/package-<PID>/ACE*
bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution
When releasing from a branch of master the resulting packages shouldn't be uploaded to
ACE+TAO-distribution
but only the versioned packages should be
uploaded to bczar@download.dre.vanderbilt.edu:/export/www/download.dre/previous_versions
using
scp $DOC_ROOT/package-<PID>/ACE*6*
bczar@download.dre.vanderbilt.edu:/export/www/download.dre/previous_versions
-
After the repository is tagged you can also start generating the doxygen
documentation in parallel with the kit generation above.
-
Login to a release system you prepared with the same packages as above:
screen
-
After login check that you can,
ssh bczar@download.dre.vanderbilt.edu
to ensure that this succeeds. If not fix the problem, if ok exit again back to
your release system.
-
cd /tmp
and remove the contents of the doxygen directory and recreate it again with
rm -rf doxygen
mkdir doxygen
cd doxygen
If you create the doxygen documentation on naboo.dre.vanderbilt.edu
than make sure you use /web/users/isisbuilds/tmp/ACE_wrappers
as working directory
-
Update the workspace with the right version tag (replace the X_Y_Z with the ACE
version number being released e.g. 7_0_5)
git clone https://github.com/DOCGroup/ACE_TAO.git --depth 1 --branch ACE+TAO-7_0_5 ACE_TAO
- Change to the ACE_TAO directory using
cd ACE_TAO
-
Set the needed environment variables using
export ACE_ROOT=$PWD/ACE
export TAO_ROOT=$PWD/TAO
export CIAO_ROOT=
export DANCE_ROOT=
export DDS_ROOT=
-
Check the doxygen version at the shell by executing the command:
doxygen --version
This should be at least 1.6.2
Now you are ready to create documentation
-
cd $ACE_ROOT
$ACE_ROOT/bin/generate_rel_manpages
When this is ready copy the resulting files using
scp ACE-html.tar.gz ACE-html.tar.bz2 ACE-html.zip ACE-html.tar.gz.md5 ACE-html.tar.bz2.md5 ACE-html.zip.md5 bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution
screen
cd $HOME
rm -rf doxygen
mkdir doxygen
cd doxygen
git clone https://github.com/DOCGroup/ACE_TAO.git --depth 1 --branch ACE+TAO-7_0_6 ACE_TAO
cd ACE_TAO
export ACE_ROOT=$PWD/ACE
export TAO_ROOT=$PWD/TAO
export CIAO_ROOT=
export DANCE_ROOT=
export DDS_ROOT=
cd ACE
$ACE_ROOT/bin/generate_rel_manpages
scp ACE-html.tar.gz ACE-html.tar.bz2 ACE-html.zip ACE-html.tar.gz.md5 ACE-html.tar.bz2.md5 ACE-html.zip.md5 bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution
-
While doxygen churns, format a release announcement, including the release
notes gathered from developers.
-
On naboo.dre.vanderbilt.edu once the doxygen generation has finished, update the documentation at www.dre.vanderbilt.edu/Doxygen.
-
Create a temporary directory. eg. tmp and change Directory to this-
-
scp file from the download server
-
Unzip and untar the files
- back out of the temporary directory
-
Create directory 'Current Version No' for example 6.3.4 and change directory into this new one
-
Move contents of the temporary directory's html to this directory -
-
Now back our of this directory and remove the already existing softlink to the "Micro" directory -
-
Create softlink "Micro" linking it to new Documentation using -. If this is a minor release also update the
Stable
link.
-
Remove the directory tmp
cd /web/www/Doxygen
mkdir tmp
cd tmp
scp bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution/ACE-html.tar.bz2 .
bunzip2 ACE-html.tar.bz2;tar -xvf ACE-html.tar
cd ..
mkdir 7.0.5
cd 7.0.5
mv ../tmp/html .
cd ..
rm Micro
ln -s 7.0.5/html Micro
rm -rf tmp
-
On download.dre.vanderbilt.edu where the kit was uploaded to and once BOTH the kit
and doxygen generation have finished their work, you should also move the packages to the
previous versions directory with the appropriate decorators.
-
cd /export/www/download.dre/ACE+TAO-distribution
-
Check that the packages in this directory are all from today
-
Modify
/export/anduriltmp/bczar/copy_script.sh
to use the correct ACE version X.Y.Z and run it.
-
Update the copy_script.sh file for the new micro release
-
Update ACE_wrappers/bin/copy-local-script.sh for use the latest version number and use this script to rename all packages
locally to include the version number. After this is done upload the versioned packages to the github release page
-
Validate the packages on the webserver whether they are really containing the new release. Make at least
one build where you run the TAO Hello world test and check if the libraries are having the
correct version number.
-
Update the ACE_wrappers repo:
- docs/Download.html to show the new release.
-
etc/index.hml to show the new doxygen package you installed
- bin/diff-builds-and-group-fixed-tests-only.sh to give the correct default old_date for this release.
-
Update the ACE/NEWS and TAO/NEWS files to have a new section for the next release.
-
Put the release details also on the github release page for this latest release, see https://github.com/DOCGroup/ACE_TAO/releases
-
Update openSUSE Build service using
osc checkout devel:libraries:ACE:micro ACE
cd "devel:libraries:ACE:micro/ACE"
osc rm ACE+TAO-src-*.tar.gz
cp $DOC_ROOT/package-<PID>/ACE+TAO-src-*.tar.gz .
osc commit
-
Make a new pull request for homebrew-core to update ACE to the new release, see https://github.com/Homebrew/homebrew-core/blob/master/Formula/ace.rb
-
Make a new pull request to Microsoft vcpkg to update ACE to the new release, see https://github.com/Microsoft/vcpkg/tree/master/ports/ace
-
Make a new pull request to OpenDDS to update the configure script to use the new release, see https://github.com/objectcomputing/OpenDDS/blob/master/configure
-
Mail the approved release announcement out to, at minimum the following:
tao-users@list.isis.vanderbilt.edu, tao-announce@list.isis.vanderbilt.edu,
ace-users@list.isis.vanderbilt.edu, ace-announce@list.isis.vanderbilt.edu.
Do this as yourself (not as bugzilla). N.B.
You will not be able to post to the users' lists unless you are subscribed to
them. Odds are you will not be able to post to the announce lists at all. Ask
someone else (like Doug or Johnny) to do this step.
Tips to being a Build Czar
- Trust no one.
- Be careful with this guy, he
is notorious in breaking builds (and fixing them as well...rumour has it that
it's actually a super-scalar, super-pipelined processor capable of out-of-order
execution, in human incarnation).
- Don't forgive people who break ACE :-)
- If a build hasn't run in a long time (symptoms are a "red" in the Last Run
column of the build scoreboard), delete the .disable file in
/path/to/build/directory/BUILD_NAME/ by hand.
- Think of the group who wrote the scoreboard update script, every time you
catch an otherwise not so obvious error with the help of the scoreboard. Tell
DEVO group about it.
- Compile once on Win32 and Linux before cutting a micro.
- Trust the release script when making a release. Don't make tar balls by
hand.
- When all hell breaks loose, don't wait for the nightly builds to monitor
improvement. Instead manually start the builds.
- Don't hesitate to ask for help.
- Install your public key to the different machines you have frequent access
to avoid typing password.
- Update this page if you have any more tips for future build czars :-). This
page is in git under
ACE/docs/bczar/bczar.html
The Realm of the Build Czar
Build Czar Arthur
Many years have passes since the days of the legendary Build Czar Arthur. His
duties were given to him by the mystical Lady of the Lake, who outlined the
first responsibilities of the Build Czar.
Then bespake the Lady of the Lake,
And these were the words said shee:
"I knoweth of thy deeds, thou noble Arthur,
but thy task hath not finished for thee"
"Thou shalt feitch thy trusty steed,
And cleanse thy builds againe;
Then shallt thy ryde hath finnished,
When new kits released thee cann."
Then bespake him noble Arthur
And these were the words said he:
"With what weapons shallt I use,
To asure these from the devil free?"
Then appeered before noble Arthur,
Uppon the ground a sacred scroll
Conjurred by the Lady of the Lake
Borne of the earth in a roll.
She saies, "Clasp this to thine selfe
For thee shallt find need for it.
It shall keep others in the cold,
Only to be ressurected when thee sees fit."
"Others shall join thy person,
To ryde with thee in thy quest;
Thee shallt be thankful of theire help,
And to alsoe hold them steadfast."
"But if theire talke too lodly rise,
And causeth much damage to thine cuntry,
He must come forth, and make proclamation,
For the next one he shall be."
So hath Arthur to the Lady spoke:
"For I sweare, and save my othe,
While enimes and evils I seeke,
I shall fight against them bothe.