r7 - 13 Jul 2007 - 15:27:10 - MimiYinYou are here: OSAF > Journal Web > TWikiUsers > MikeT > MikeTaylorNotes > LinuxDistributionNotes
Note: This page is a draft and is currently under active revision - please do not make major changes to it without first contacting me (bear). Tweaks, typos and comments are very much appreciated.
-- heck, currently it's very hard to even locate a dmg for 2.4.x
We have gotten a lot closer now that we have no code related issues with Python - all of our patches are for configuration and building. Currently, for the libraries we use, only libxml2, openssl and swig are patched in a non-trivial manner. We would need to ramp up the upstream adoption of our patches.
See Chandler External Libraries for a detailed list of each library and it's status.
Prior to 0.5 we were on the bleeding edge of a lot of libraries and even with Python. With 0.6 in the "done" column and planning for 0.7 underway I don't see us having to work with beta libraries at all. This will allow us to more easily target existing system installed libraries as long as we can create a verification tool to find out when we would need to install a custom version. Some libraries, Berkeley DB for instance, we will pretty much always have a custom installation as it is a known source of grief when the underlying library is upgraded.
One of the things that I think makes even the thought of using system libaries a possibility is the work being done for Python Eggs - that ground work allows us to not sweat the details if we find we do need a custom local copy. It also greatly simplifies the building of installation scripts for a Linux platform and even possible Windows or OS X.
(heikki) I think of this differently. The goal is (should be IMO) to:
OSAF packages that are not Chandler specific (create .debs):
(bear) I see the distinction you are making - I was going to general and you want to focus entirely on a Linux platform (Ubuntu) and then work from there - is that accurate? Some of the above I could agree with given two points that would have to be enforced extremely rigidly:
(heikki) Not sure what you mean by detecting system libraries. Mostly I think the library versions should be handled by the packaging system dependency evaluations (assuming that all packages Chandler relies on, including python packages, are distributed as debs instead of the python setup.py install way). I don't agree bdb (db?) would necessarily need to be in the core Chandler deb if we don't have any changes in it. But seems we could take incremental steps:
(bear) What I mean is, lets say for example, I install Chandler and all is fine but then later update (or something causes an update) to a version of library that introduces a possible crash. Chandler would start and then sometime soon it would crash causing data loss (potentially). To the user it would appear that Chandler is at fault but if we had some sort of "is your system healthy" set of checks during startup then we could warn the user that something has changed. I mention bdb because of all the packages I've seen cause grief it's bdb simply because they can radically change behaviour between versions and suddenly your data is no longer accessible. I've seen it happen to people who use SVN with a bdb repository -- something forces an update to bdb and suddenly the source repository is offline for hours if not days. But we are getting down to implementation details and other minor nits I think.
(heikki) Ideally that case would be taken care of by the package dependency system. Of course, most packages just say version x or newer is good. But in some cases the newer version is not good. In my opinion it is practically impossible to detect if a newer package is incompatible automatically. (Once we find out that a newer version is not good, a new version of Chandler can have updated dependencies to exclude the bad package version(s)). All programs on Linux have this same problem, so I don't think we need to solve it for Chandler. This is also true on Windows (shared DLLs) and Mac with apps that are not a single binary.
Linux Platforms
- Ubuntu (breezy) -- Currently Chandler runs and full builds are possible on Ubuntu (minor dev environment tweaks - nothing unusual)
- Fedora Core 4 -- Currently Chandler runs and full builds are possible on FC4 (minor dev environment tweaks - nothing unusual)
- Gentoo -- Currently ebuilds are available for Chandler - more testing needs to be done with current source
- Debian -- Currently Chandler builds on Debian (sarge) - it's been a while since I've heard of anyone who has done this with current source
- Mandriva (formerly Mandrake) -- An attempt was done for this platform but I haven't heard if a full build has been done
Installation Methods
- Current
- RPM -- currently the distribution tarball is "mangled" into a relocatable RPM that works on FC2, FC3 and FC4
- tarball -- since we include everything, this packaging method works on any platform that we have binary compatibility with
- Possible
- Custom Script -- similiar to the distribution scripts we use to generate the RPM and .DMG installs, we could generate a custom bash or perl script to get basic information from the user and relocate the Chandler image. Very ugly but it has some retro-ish-1980's appeal
- Python Eggs -- if we can build or locate Eggs for all of the libraries then nothing prevents us from packaging EasyInstall? along with a simple script to ensure version info and letting Python Eggs via EasyInstall? to do the download and install work
- Autopackage -- in a lot of ways Autopackage is similiar to Python Eggs as it is a bootstrap shell script that then walks thru the included package info and metadata to either install from the .package file or download what is required.
- OpenPKG -- similiar to Autopackage but uses RPM packages on the backend. RPM is not required on the target system as it bootstraps a custom RPM manager. Currently available on a number of platforms and distros already.
Issues
Currently we bundle all libraries and Python with our distribution. This is done to avoid "dependency hell" and I think this issue will be the one major blocking point for integration into the mainstream distributions. They will all want us to use current system libraries but we have a number of patches to move upstream to enable this. On some platforms, namely Windows and OS X, I don't think we will ever get away from building our own Python. Currently Apple ships Python version 2.3 with Tiger and I don't think we can conceive of a use-case where we put up a dialog box asking the user to retrieve and install a Python dmg before continuing -- heck, currently it's very hard to even locate a dmg for 2.4.x
We have gotten a lot closer now that we have no code related issues with Python - all of our patches are for configuration and building. Currently, for the libraries we use, only libxml2, openssl and swig are patched in a non-trivial manner. We would need to ramp up the upstream adoption of our patches.
See Chandler External Libraries for a detailed list of each library and it's status.
Discussion
With the ground work that Ubuntu, Debian and FC4 have done along side the work of Gnome and KDE it is a lot easier to build an application with the expectation that you will be able to get it to build and run on multiple platforms/distributions. Because we are primarily a Python app, the job is a heck of a lot easier - though that gain is sucked back into a deep dark hole by the complexities of wxPython
Prior to 0.5 we were on the bleeding edge of a lot of libraries and even with Python. With 0.6 in the "done" column and planning for 0.7 underway I don't see us having to work with beta libraries at all. This will allow us to more easily target existing system installed libraries as long as we can create a verification tool to find out when we would need to install a custom version. Some libraries, Berkeley DB for instance, we will pretty much always have a custom installation as it is a known source of grief when the underlying library is upgraded.
One of the things that I think makes even the thought of using system libaries a possibility is the work being done for Python Eggs - that ground work allows us to not sweat the details if we find we do need a custom local copy. It also greatly simplifies the building of installation scripts for a Linux platform and even possible Windows or OS X.
(heikki) I think of this differently. The goal is (should be IMO) to:
- Make it possible for distributions to include Chandler as only those components that OSAF created, and rely on stock components otherwise
- OSAF to possibly create the packages for a single distro (Ubuntu and/or Debian)
| Package | OSAF patches required | Non-patch reasons why OSAF needs to build and distribute this | Notes |
|---|---|---|---|
| SOAPpy | yes | Might not be necessary to have this package | |
| dateutil | no | ||
| icu | no | ||
| jabber-py | yes | Might not be necessary to have this package (not used at all and can be replace with twisted's version -- bear) | |
| m2crypto | no | latest .deb 0.13, Chandler uses 0.15 | |
| openssl | no | Patches are Windows and OSX specific | |
| db | yes | ||
| libxml2 | yes | ||
| python | yes? | ||
| bzip2 | no | System python must have been built with bzip2 support | |
| zlib | no | System python must have been built with zlib support | |
| readline | no | ||
| setuptools | no | ||
| twisted | no | Need later version than currently latest release? (need svn version currently -- bear) | |
| zope | no | ||
| wx | yes | ||
- PyICU?
- PyLucene
- vobject
- zanshin
- chandlerdb
- chandler
(bear) I see the distinction you are making - I was going to general and you want to focus entirely on a Linux platform (Ubuntu) and then work from there - is that accurate? Some of the above I could agree with given two points that would have to be enforced extremely rigidly:
- Any system library that Chandler detects (and startup detection would have to be well done) that doesn't meet the needs would cause Chandler to stop with an error
- Certain "core" packages (chandlerdb, bdb (only because it's so integral to our repository) and chandler itself would be part of the application and not available as individual pieces
(heikki) Not sure what you mean by detecting system libraries. Mostly I think the library versions should be handled by the packaging system dependency evaluations (assuming that all packages Chandler relies on, including python packages, are distributed as debs instead of the python setup.py install way). I don't agree bdb (db?) would necessarily need to be in the core Chandler deb if we don't have any changes in it. But seems we could take incremental steps:
- Create debs (or use the existing debs) for all of the non-OSAF and OSAF packages ecxept:
- Create a core Chandler deb that contains:
- db
- wx
- chandlerdb
- chandler
- Create a core Chandler deb that contains:
- Push all other changes into the respective projects and get updated releases and debs for them
(bear) What I mean is, lets say for example, I install Chandler and all is fine but then later update (or something causes an update) to a version of library that introduces a possible crash. Chandler would start and then sometime soon it would crash causing data loss (potentially). To the user it would appear that Chandler is at fault but if we had some sort of "is your system healthy" set of checks during startup then we could warn the user that something has changed. I mention bdb because of all the packages I've seen cause grief it's bdb simply because they can radically change behaviour between versions and suddenly your data is no longer accessible. I've seen it happen to people who use SVN with a bdb repository -- something forces an update to bdb and suddenly the source repository is offline for hours if not days. But we are getting down to implementation details and other minor nits I think.
(heikki) Ideally that case would be taken care of by the package dependency system. Of course, most packages just say version x or newer is good. But in some cases the newer version is not good. In my opinion it is practically impossible to detect if a newer package is incompatible automatically. (Once we find out that a newer version is not good, a new version of Chandler can have updated dependencies to exclude the bad package version(s)). All programs on Linux have this same problem, so I don't think we need to solve it for Chandler. This is also true on Windows (shared DLLs) and Mac with apps that are not a single binary.
Log In
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.
See list of page contributors for attributions.