r59 - 13 Jul 2007 - 15:27:12 - MimiYinYou are here: OSAF >  Projects Web  >  ChandlerHome > DeveloperDocumentation > DebuggingChandler

Most of this page explains how to run Chandler in a debugger.

Logging

Logging system

Note that sometimes you may be able to find enough information from the various log files that are generated when Chandler runs. For example, see chandler/chandler.log.

The current logging system is described in LoggingFramework .

autolog.py - automatic logging

There is also a system that can automatically log when methods and functions are called, and let you see the structure of program or subsystem procedures: AutoLogger. It is hooks into all function calls that you specify, so is especially useful if you don't know what's getting called when, or need to continuously test for state chagnes.

Debugging Chandler

Debugging with WingIDE

WingIDE (2.X) is one of the commonly used debuggers at OSAF. Note that WingIDE provides free Licenses for non-commercial Open Source development projects.

We have included pre-built project files for each of the platforms, debug and release in the Chandler directory:

  • WingLinuxDebug.wpr
  • WingLinuxRelease.wpr
  • WingMacDebug.wpr
  • WingMacRelease.wpr
  • WingWindowsDebug.wpr
  • WingWindowsRelease.wpr

To debug Chandler using Wing, open the appropriate project and press Run.

Their is one minor issue on Windows: You need to start the Wing by double clicking the project file in the Chandler directory, or make a shortcut to the project file and copy it to your desktop, then open Wing with the shortcut -- otherwise the working directory may not be properly set to the Chandler directory.

Another minor problem occurs because Wing can't always distintuish between an exception that is handled by Chandler and one that isn't handles and is therefore a bug. So, the first time you run Chandler with the project file, Wing will stop on a few exceptions that Chandler will correctly handle. When this happens, choose the "ignore this exception location" checkbox and from then on out Wing won't stop on these exceptions. If you're wondering if an exception is going to be correctly handled, just look up the stack a frame or two and you'll be able to tell if it's going to be handled.

There is an alternate method to debug using Wing, which is more complicated to set up, but which I find useful debugging functional tests. Rather than starting Chandler by pressing Run in the debugger, you can start Chandler from a command line in a shell, add --wing to the command line which will cause Chandler attach to the debugger.

For this to work you'll need to first copy a module named wingdbstub.py from the Wing install directory to the Chandler directory. And you'll need to make sure you don't run the python code optimized, e.g. with -O on the python command line. If you're using RunChandler?.bat, you can set an environment variable "OPTIMIZE" to an empty string which will run Chandler non-optimized. I always set the OPTIMIZE environment variable during development since there is little performance penalty and this includes assert checks, which are useful for catching problems early before they snowball into something hard to track down.

Here's what I put in my .bashrc to define aliases for running/debugging functional tests:

# Put in the location of your OSAF working directory here: export OSAF="/cygdrive/d/John/osaf"

export OPTIMIZE=\

export FT_PARCELPATH=`cygpath -aw $OSAF/chandler/tools/QATestScripts/DataFiles` export FT_SUITE_PATH=`cygpath -aw $OSAF/chandler/tools/QATestScripts/Functional`

alias ftdebug='cd $OSAF/chandler; export PARCELPATH=$FT_PARCELPATH;./debug/RunChandler.bat --stderr --nocatch --create --wing -f $FT_SUITE_PATH/FunctionalTestSuite.py'

alias ftrelease='cd $OSAF/chandler; export PARCELPATH=$FT_PARCELPATH;./release/RunChandler.bat --stderr --nocatch --create --wing -f $FT_SUITE_PATH/FunctionalTestSuite.py'

To run a functional test I open the appropriate Wing project, set a breakpoint if appropriate, then type ftdebug or ftrelease in a shell (cygwin on Windows). Wing breaks on unhandler exceptions, asserts, or breakpoints that I've set.

The project files and these instructions may require changes as Chandler changes, so please submit bugs if any of this doesn't work. The most likely thing to change is the set of files included in the project file, so as files come and go from Chandler they will not automatically be added to the project file. If you don't see a file in the project, you might need to add it.

All these instructions work for all the platforms. The only platform specific problem I run into is on Mac PPC: Wing is so slow it makes development using the debugger not very practical.


Debugging with PyCrust

To enter the debugger, select Test -> Show PyCrust Debugger....


Debugging with Python's interpreter

If you are getting an error and want to a really quick-and-dirty fix, you can run Python's built-in interpreter.

If you modify osaf/chandler/debug/RunDebug to give the "-i" option to Python, then you'll be tossed into the Python interpreter when it quits. The python interpreter lets you set breakpoints, but is quite primitive. This is useful if Chandler is quitting with an error. It isn't as useful when everything is running fine, since the python interpreter doesn't fire up until you quit Chandler.

Also, you can run hardhat.py -i (for an interactive session); use -di for debug mode


Debugging with PDB

You should be able to use the Python Post-Mortem Debugger (PDB) to launch Chandler, except that at the moment, you can't. See Bug:577 for details (and for whether it's been fixed or not).

However, pdb can be useful for debugging problems that occur in background threads, where other methods (like Wing) sometimes don't work too well. To trigger a breakpoint in some Python code, insert the following:

    import pdb
    pdb.set_trace()

and launch Chandler from the command line. When the pdb.set_trace() call is executed, you're given the (Pdb) prompt and can enter Pdb Commands. In particular, you can get a list of commands by entering help at the prompt!


Debugging with ActiveState Komodo

It is possible to debug Chandler with ActiveState Komodo 3.1 or later - don't even try with anything older.

  1. Launch Komodo and edit it's preferences:
    • Edit > Preferences... > Debugger: Make Komodo listen on port 9000
  2. Copy dbgp directory from Komodo install into Chandler site-packages directory
  3. Launch Chandler with --komodo command line argument
  4. Let Komodo attach to the Chandler process
  5. Hit Go in Komodo

It can be a little slow, so be prepared to wait...

You can also start Chandler from Komodo itself. Create a Chandler project (or modify the one below):

  • Chandler.kpf: Chandler project for Komodo - you need to edit to suit your paths

Then

  1. Edit > Preferences... > Languages > Python: Change the Python interpreter to the Chandler version, and add chandler and parcels directories to additional imports
  2. Open Chandler.py in Komodo.
  3. Hit Go.
  4. Fill in the script arguments, and environment values and go.

Multi-threaded debugging in Komodo

Setting breakpoints in background python threads works thanks to some clever tricks by the Komodo folks. Unfortunately, these tricks don't work with Chandler's RepositoryThread objects: They install handlers at the point python starts executing in a new thread it has created, but in the case of RepositoryThread (or actually its superclass, PyLuceneThread), thread creation is handled by gcj, not python.

Here's a patch that enables debugging of RepositoryThreads in Komodo. It's been tried on version 3.10.0-alpha1 on Mac OS X, YMMV on other versions:

Index: repository/persistence/Repository.py
===================================================================
--- repository/persistence/Repository.py        (revision 7037)
+++ repository/persistence/Repository.py        (working copy)
@@ -342,4 +342,25 @@
 class RepositoryThread(PyLucene.PythonThread):
-    pass
+    def __init__(self, *args, **kwds):
+        try:
+            import dbgp.client as client
+        except ImportError:
+            client = None
+                
+        if client is not None:
+        
+            oldTarget = kwds['target']
+            
+            def newTarget(*args, **keywds):
+                # This all copied out of Komodo's client.py
+                c = client.backendCmd()
+                ## the values for host and port do not matter here, they are retrieved
+                ## appropriately from the class variables
+                c.connect('', 9000, self.name)
+                c.runThread(oldTarget, args, keywds)
+                
+            kwds['target'] = newTarget
+            
+        super(RepositoryThread, self).__init__(*args, **kwds)
+

Debugging with Eclipse and pydev plugin

Preparation

  1. Set a custom Python interpreter: Windows > Preferences... > Pydev > Interpreter - Python to Chandler's python executable.
  2. In the same panel, you may need to add the following to PYTHONPATH (not sure about this step):
    • [...]site-packages/chandlerdb/item
    • [...]site-packages/chandlerdb/persistence
    • [...]site-packages/chandlerdb/schema
    • [...]site-packages/chandlerdb/util
    • [...]site-packages/M2Crypto
    • [...]site-packages/wx
  3. Optional: confirm that you have all the expected compiled libs found in PYTHONPATH, for example:
    • bz2.pyd
    • icuin36.dll
    • libdb44.dll
  4. Optional (to enable pylint support) Put in the Chandler's version of site-packages/pylint/lint.py file in the pylint location field here: Python interpreter: Windows > Preferences... > Pylint
  5. Right click the Chandler project root in the Navigator, and select Properties > Pydev - project type. Set it to Python 2.5.
  6. Right click the Chandler project root in the Navigator, and select Properties > Pydev - PYTHONPATH. Add the following directories:
    • /chandler
    • /chandler/parcels
  7. Right click on Chandler.py in the Navigator, and select Debug As > Debug... Make sure you select the chandler Chandler.py configuration. Make sure that project is the Chandler project, main module is Chandler.py. In the arguments tab, make sure base directory is the chandler root directory, and the interpreter is the Chandler python. In the Environment tab, add PATH environment variable whose value is chandler/release/bin (or chandler/debug/bin). (On Linux set LD_LIBRARY_PATH instead: absolute paths to release/lib:release/db/lib:release/icu/lib) Select the "Append to native environment option". In the Common tab, check the box to display in Debug favorites menu. Hit close, and let it save.

Now, when you want to debug Chandler, find the debug toolbar dropdown button, and select chandler Chandler.py to run. Most likely it will be the first, so you could just click the bug symbol to run it.

Eclipse with Pydev seems to be able to debug multiple Chandler threads without any gimmicks.


Debugging with other Python Debuggers

PythonWin (win32), Idle (Unix), and Boa (Unix) have also been recommended as debuggers but the person writing this sentence hasn't yet had time to go look into them. (If you know something about these, please add info! @@@)


Debugging with GDB and frontends

Launching Chandler under GDB

For those who are interested here are the steps to running Chandler under gdb to debug Python language, PyLucene, and wxWidgets bugs.

1. Download and compile the entire Chandler internal and external source (make DEBUG=1 world):

BuildingChandlerDesktop (The long and more difficult way - build binaries yourself)

2. The most straightforward way is to just run Chandler, then attach to the running process. This should work even if you're trying to debug an end-user download, or "make install" build of Chandler source (with or without DEBUG=1). See "Attaching to a running Chandler using GDB" below.

3. The RunChandler (and RunPython) scripts support launching Chandler by setting the environment variable GDB. For example:

bash$ GDB=1 ./release/RunChandler --profileDir /tmp/profile --verify

Chandler directory is .
Using ./release/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python -O ./Chandler.py --profileDir /tmp/profile --verify
-O ./Chandler.py --profileDir /tmp/profile --verify
GNU gdb 6.3.50-20050815 (Apple version gdb-477) (Sun Apr 30 20:01:44 GMT 2006)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries ..... done

(gdb) run -O ./Chandler.py --profileDir /tmp/profile --verify

The command-line arguments ("./Chandler.py --profileDir /tmp/profile --verify") are printed beneath the line that starts with "Using ..."; the clever user has copy-and-pasted them to gdb's "run" command to launch the Python interpreter, which executes Chandler the same way it would normally (although the interpreter is running under gdb).

You can set breakpoints, watchpoints, examine the stack, or interrupt the process with control-C just like any other C/C++ application.

GDB macros for debugging python code

The python project source contains gdb macros for viewing the Python call stack, local variables, etc:

http://svn.python.org/projects/python/trunk/Misc/gdbinit

Typically, you'd want to install this in your ~/.gdbinit, which is read by gdb when it starts. For example, with this in place you can use the pystack macro to get the python stack for all threads by:

(gdb) thread apply all pystack
Thread 7 (process 4830 thread 0x5803):
/Users/moondawg/chandler/release/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/threading.py (195): wait
/Users/moondawg/chandler/release/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/threading.py (101): wait
/Users/moondawg/chandler/release/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/Queue.py (139): get
/Users/moondawg/chandler/release/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/threading.py (420): run
....
Thread 6 (....)
... etc

(For those who like to minimize typing, t a a pystack is an equivalent command to the above). In some cases, gdb may appear to hang after printing out a thread's stack; usually hitting Control-C allows it to move on to the next thread.

On PowerPC Mac, I've had issues with the pystackv crashing gdb; I think this has to do with gdb being generally clueless about local variables when they're stored in registers.

Attaching to a running Chandler using GDB

Sometimes, you'll want to attach to a running copy of Chandler. For instance, it might have gotten "stuck", and you might want to find out why. Also, this can be more reliable then trying to run the python binary.

To do this, first find out Chandler's pid (actually, the pid of the python interpreter running Chandler) by firing up a shell, and running

ps ax | grep -i python

You can then execute

gdb attach pid

or, if you have it,

ddd attach pid

for the DDD graphical frontend.

If you're successful, you'll see something like:

Attaching to program: `/Users/petunia/src/chandler/debug/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python', process 22436.
Reading symbols for shared libraries ....... done
0x90016f48 in semaphore_wait_signal_trap ()
(gdb)

and you can use all the tricks in the previous sub-section to get at python state.

GDB Frontends

DDD

ddd is an old Unix graphical frontend to GDB. It's clunky, but way way better than the commandline! You can point-and-click to see local variables and introspect their data structures and the like.

You invoke and use it in the exact same way as GDB.

Emacs

Emacs has a semi-gui interface to GDB, which is nicer because it shows where in the source code you are, but that's about it. Inside emacs, run Alt-X gdb [enter], then do "attach PID" and the rest. But when you do "frame 5", it will show the location in the code.

This has issues getting pathnames, so you have to start emacs in e.g. trunk/internal/wx/src to get the paths right for debugging wx. The gdb "cd" command may be able to do this too.

I recommend DDD instead. (-brendan)

others to try

eclipse ... kdbg ... whatever kdevelop uses (kdbg?) ... more?...

Debugging Python Extensions

If you are developing a Python extension which is written in some other language besides Python, there are some debugging tips as well:

  • DebuggingPythonExtensions

Late-breaking Issue: GDB on Mac OS X 10.4.7.

A recent software update on Mac OS X seems to have had a negative effect on running gdb on Chandler. The symptom will be an inability to locate dynamic shared libraries. In the debug version you may actually get an error message to this effect. In the release version you just get a message about "no suitable image found".

This is a workaround which may work for you, assuming you have admin access and can write to the directory /usr/local/lib

This assumes you have checked out chandler to CHANDLERDIR and done a build such that these libraries are in place. (This may be a "make compilers" or "make world", I'm not sure -- Reid 2006-08-28)

cd CHANDLERDIR/external/icu/icu-3.4/build_debug/lib
sudo ln -s $cwd/libicuuc.dylib.34.0 /usr/local/lib/libicuuc.dylib.34
sudo ln -s $cwd/libicui18n.dylib.34.0 /usr/local/lib/libicui18n.dylib.34
sudo ln -s $cwd/libicudata.dylib.34.0 /usr/local/lib/libicudata.dylib.34
cd CHANDLERDIR/chandler/debug/db/lib
sudo ln -s $cwd/libdb_java-4.4.dylib /usr/local/lib/libdb_java-4.4.dylib

These libraries to not change very often over time, and this workaround should be okay for a long while.

External Resources

  • Debugging in Python
  • Comparison of Python Editors
PageInfo
PageType DevDocPage
MaintainedBy MorgenSagen
PageStatus Work in progress -- this page is still being drafted?
Trash.CommentsWelcome2 Feel free to contribute comments?
Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r59 < r58 < r57 < r56 < r55 | More topic actions
 
Open Source Applications Foundation
Except where otherwise noted, this site and its content are licensed by OSAF under an Creative Commons License, Attribution Only 3.0.
See list of page contributors for attributions.