Bolting Email Notifications of Failure To Tinderbox

One of the biggest items people have been asking for in the Tinderbox setup we have is to get notifications via email of test and build failures. Recently it was decided that this is now a crucial item so it's moved onto my active todo list - woo!

Many people would think that email notifications would be easy to do - heck, just find the point where the failure is noticed and send an email to everyone who has done a revision recently. Right? ....

Wrong and for the usual bizarre reasons

Let me outline the path of information thru the various parts of the system to give everyone a good background into this.

Tinderbox Client side

Each tinderbox is a python script that does a number of different tasks, some of which are:

• Send an email to the Tinderbox server that a build has started
• Update or checkout from SVN the current source code
• If it's a full build, go to the external/ directory and do

  make all binaries install

  and if it's a "quick" build, go to chandler/ and do

  make install

• Once the source is updated and everything built, run unit tests
• If all tests pass:
  • Run the distribution scripts
  • If this is a full build, check to see if any binary tarballs need uploading
• Send an email to the Tinderbox server with the final status: success, build_failed or test_failed

Once all that is done, the script pauses and then starts it all over again.

Tinderbox Server side

For each email that the Tinderbox server receives, the following happens:

• The email is checked for sanity (the Tinderbox mailbox gets a lot of spam)
• The status information is extracted and parsed
• The log contents is parsed and processed for errors, QA test runs and other content
  • errors are flagged and changed into html links and other info
  • QA test results are extracted and stored into *.perf files that are processed later
  • Full and Brief logs are written to the data dirs
  • A status file is created for the build event

Two cronjobs run on the Tinderbox server, one that runs the Tinderbox perl code to update the various status pages and puts the status file info into an internal database and the other that scans all of the *.perf files and puts that information into a performance database and generates the performance status pages.

Putting all the parts together

Now each of the above steps for the most part are ignorant of the other parts - they get the required information from upstream data sources. The fun part for this task is that while technically the Tinderbox Client is aware of the revision information, it's never been stored or passed downstream and the Server side is blissfully unaware of even the fact that the build uses SVN and just deals in status updates. The Tinderbox2 code uses date and time information internally to line up most of the events so they make sense when presented.

My task will be to inject the revision information into the data flow, make sure Tinderbox2 code doesn't break and then write a scanning program to take the information gathered (similiar to how the performance information is gathered) and make decisions based on the status information. I've decided to take this route because it requires the minimal amount of change to Tinderbox2 while giving me the ability to write most of the tricky code in python.

Implementation Notes

Tinderbox2 client script changes

Currently the Tinderbox2 build scripts were only passing back a single string to the tinderbox.py master script - the build status information. I had to modify the Start() function to collect the svn revision number and return it along with the status string. This required me to change tinderbox.py and all of the current build scripts: newchandler (quick builds), fullchandler, cosmo and scooby.

I also added a routine to do a brute force scan of the captured svn update and svn checkout commands to look for the revision number strings. This did require changing the command used for svn co as it was passing -q which removed that line from the output.

Tinderbox2 changes

The only change I'm making to Tinderbox2 is to the processmail_builds perl script - that's the script that responds to the individual emails from each tbox. It scans the mail, sucks out the log info, parsed many different parts of the log and then writes out log files and other status files so the main tbox cron job can do it's thing.

Because there are known points in processmail_builds to parse the status information that comes from the Tinderbox2 client, I just added a small bit of code to open a *.status file and dump the following information into it:

$TINDERBOX{'tree'},"|",$TINDERBOX{'buildname'},"|",$TINDERBOX{'status'},"|",$PDate,"|",$PTime,"|",$TINDERBOX{'revision'}

This follows the same method that I used to add the code to extract the performance test information. Each Tinderbox2 client's data will get stored to a discrete file that will be picked up and processed by a python program.

Notification agent

This is the next step - stay tuned ...