Guidelines for adding new open-source modules to the Chandler "external" tree

The external tree represents open-source projects which we are using in a relatively unchanged manner. The only changes we make are to wrap the source distribution in a common structure which comprises a Makefile, patches, and a distribution tarball. The Makefile imposes the OSAF build structure on the package, and makes it become part of the OSAF distribution. Any patches required are batched into a single file that becomes part of the basic setup. Since the packages are not expected to change much from release-to-release, we do not check the full sources into our repository, but only those few files which we provide to bootstrap the installation.

The tarballs are downloaded from our server if the developer wishes to do a full build, otherwise the libraries only are packaged in another tarball identified as "debug-<version>.tar.gz" and "release-<version>.tar.gz". The external tree repository contains a Makefile at the top level, and a Makefile with a patch file for each module in use. The top Makefile contains a target called "expand" which is the first target to be invoked on a fresh checkout. This expand target pulls down the source tarball from our server containing the source archives of all the modules in external, then does an extract of the source tarball so that a full build can be performed.

Some terminology which is essential and might be confusing:

• SNAP is a reference to whether "release" or "debug" is being built.
• SITE is used to refer to the path where python modules are distributed.
• PREFIX is the term for the installation path where build products are deployed for packaging. It is within the build hierarchy

For example, suppose your work area is at /home/developer/work, then the following would be the values for each item for a DEBUG=1 build:

• BUILD_ROOT would be /home/developer/work/external
• PREFIX becomes /home/developer/work/external/debug
• SITE would be:
  • linux or cygwin would be /home/developer/work/external/debug/lib/python2.4/site-packages
  • OS X it would be /Users/developer/work/external/debug/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages

Steps for adding

The steps for adding a new "external" open-source module to Chandler are not too complicated for a seasoned developer. They consist of

• creating a new directory within external for the module
• developing the Makefile
• making the appropriate changes for the module to build in our infrastructure
• creating the snapshots
• bumping the version number in the external/Makefile and adding references to the new module
• releasing the new sources-<new-version>.tar.gz

Make a new directory inside external for the module. Typically it will be a shortened version of the distribution name: openssl-0.9.7d.tar.gz would be added within a directory openssl.

Create a Makefile for your project, using one of the other projects as a prototype. The required targets for the Makefile are

• expand
• apply (when patches are required)
• snap
• binaries
• build

Things to keep in mind when constructing the Makefile:

There needs to be a separation of debug and release so that they can reside simultaneously in the build environment. Often that can be done by simply creating the build-dir and running "configure" within it. Otherwise, copying all required files into the build-dir is required (like openssl).

To create the a patch file (if necessary) that will be applied to the source tarball used during the sub-project build process do the following:

• Put the tarball for the module into the directory.
• Determine the changes you need to make for the module to compile on all platforms and make your changes to the Makefile so that you start with a clean directory and

can produce the required configuration. Rename the final directory as <module>-patched, and re-extract a fresh copy of the module.

• Run the command "diff -u -3 -r <module> <module-patched> > patches-<VERSION>

The resulting file will be used when the "make" command is invoked on a fresh checkout.

Confirming the changes

When you have completed the build of the module on its own, add it to the external/Makefile, in both the SYSTEMS and SOURCES sections. Of course, you have already bumped up the VERSION number; if the current VERSION is 0.4-3, then you will make the next VERSION 0.4-4. The SYSTEMS simply names the module, while SOURCES provides the path and archive name. You will be doing all these steps without committing code to the source repository, so that the new work does not interfere with existing builds. You will need to repeat these steps on all platforms, so it would be handy to carry your changes from one environment to another. Thankfully, there is: use the sources target and the binaries target.

You will first run a full "make" and "make DEBUG=1" from external/, then you will run "make binaries" and "make DEBUG=1 binaries", and finally "make sources". At this point, you will find in external/ both "debug-0.4-4.tar.gz", and "release-0.4-4.tar.gz" along with "sources-0.4-4.tar". Now you will need to bring these changes to the two other platforms for conclusion of the build process. The target "package" will prepare an archive that can be moved intact to the other platforms, preserving your un-committed revisions.

Originally written by MarkJaffe - 11 May 2004

Modified by MikeT to reflect current practices