-- MikealRogers - 07 May 2006

Writing Cosmo Automated Tests Guide

This document describes how to write test cases for cosmo using the test tools and framework developed at OSAF. In addition it discusses some internals of the test framework and tools that may help in writing more advanced test cases.

This document assumes knowledge of the Python programming language and the Cosmo Sharing Server.

Dive right in

Below is a simple test class that will test user creation in cosmo via CMP.

from DAVTest import DAVTest 1

class CosmoTest(DAVTest): 2

    def startRun(self): 3
        
        self.testStart('Setup Accounts') 4
            
        # ------- Test Create Account ------- #
           
        cmpheaders = self.headerAdd({'Content-Type' : "text/xml; charset=UTF-8"}) 5
        cmpheaders = self.headerAddAuth("root", "cosmo", headers=cmpheaders) 6
           
        #CMP path
        cmppath = self.pathBuilder('/cmp/user/cosmo-multigetTestAccount') 7
        
        #Create testing account      

  
        bodycreateaccount = '<?xml version="1.0" encoding="utf-8" ?> \
                                 <user xmlns="http://osafoundation.org/cosmo/CMP"> \
                                 <username>cosmo-TestAccount</username> \
                                 <password>cosmo-test</password> \
                                 <firstName>cosmo-test</firstName> \
                                 <lastName>TestAccount</lastName> \
                                 <email>cosmo-test@osafoundation.org</email> \
                                 </user>'

 
                                 
        #Create account and check status
        self.request('PUT', cmppath, body=bodycreateaccount, headers=cmpheaders) 8
        self.checkStatus(201) # 201 ACCOUNT CREATED 9

• 1
  • This test assumes that you have DAVTest.py somewhere that python can directly import it. All the current cosmo test scripts are in the repository under cosmo/test/functional/HTTPTest which is where DAVTest.py is located.
• 2
  • All collections of test cases are contained within a single class, the class should import from DAVTest. DAVTest is a class written to include certain useful functions when testing WebDAV and CalDAV, it inherits from HTTPTest which inherits from TestObject^?, more on this later.
• 3
  • startRun() is the first test function run by the testing framework for each test class. startRun() is only run once per suite and should assume a fresh repository. All of your test classes should include at the very least an account setup test within startRun().
• 4
  • self.testStart() is a method in TestObject^?. It's purpose is to instruct the framework that a new test is starting and is given a string that describes the test. This is very useful during debugging.
• 5
  • self.headerAdd() is a method in HTTPTest. HTTPTest creates a bound dictionary called self.headers during instantiation with some same header defualts. self.headerAdd returns a dictionary with self.headers added to the dictionary passed to it.
• 6
  • self.headerAddAuth() is a method in HTTPTest. It takes three arguments, username, password, headersDictionary. It returns a dictionary headersDict with auth headers added to it.
• 7
  • self.pathBuilder() is a method in HTTPTest. It takes one argument, a string for your intended path. This is important because your test never knows exactly where cosmo is going to be installed. A root path string can be passed to your test class during instatiation and will be prepended to all your paths so long as you create them with pathBuilder.
• 8
  • self.request() is a method in HTTPTest. It's the most important and most often used method in testing cosmo. It takes 4 arguments; method, path, body, headers. This test uses PUT, so the string 'PUT' is sent as the first argument followed by the path and body strings we created earlier and our headers dictionary. This will send that request to cosmo.
  • Notice that we didn't need to pass a host or port to the function. This is because those variables will be defined during instantiation of your test class.
  • self.test_response contains the last response from a self.request() call. All the other verification test methods look for the response in self.test_response.
• 9
  • self.checkStatus() is a method in HTTPTest. Its very simple, it takes one integer argument and checks self.test_response to verify that it's status code matches. You don't have to write any pass/fail handling or failure reporting, it's all handled for you.

Well then let's do something with it.

Running tests

Most of the cosmo tests currently written can be run by invoking the script in the command line and passing it a few arguments. This has been very useful so simple testing and debugging. You should add the code sample below to the end of your test scripts.

if __name__ == "__main__":
    
    import sys
    
    host = 'localhost'
    port = '8080'
    path = '/cosmo'
    debug = 0
    
    for arg in sys.argv:
        args = arg.split("=")
        if args[0] == "host":
            host = args[1]
        elif args[0] == "port":
            port = int(args[1])
        elif args[0] == "path":
            path = args[1]
        elif args[0] == "debug":
            debug = int(args[1])
        
    print "host %s port %s path %s debug %s" % (host, port, path, debug)
    cosmotest = CosmoTest(host=host, port=port, path=path, debug=debug) 1
    cosmotest.fullRun() 2

• 1
  • All of the arguments passed during this class instantiation are required. There are other optional arguments not shown that are used for running recurring tests, masking for suite runs, and arguments used for stress test compatibility which will all be discussed later.
• 2
  • fullRun() is a method in TestObject^?. It will run the startRun() method of your test class and other test methods if applicable.

What the hell is a recurring test?

There are two seperate types of tests in cosmo automation. Tests that can only be run once with the same response, and tests that can be run many times with the same response. For example, setting up a user and doing a few PUT requests of ICS files will work the first time they are ran, but if run again against the same repository they will all fail. But REPORT tests can be run many times and should always get the same output.

Lets see how to create a REPORT test.

from DAVTest import DAVTest

class CosmoTest(DAVTest): 

    def startRun(self):
        
        self.testStart('Setup Accounts') 
            
        # ------- Test Create Account ------- #
           
        cmpheaders = self.headerAdd({'Content-Type' : "text/xml; charset=UTF-8"}) 
        cmpheaders = self.headerAddAuth("root", "cosmo", headers=cmpheaders) 
           
        #CMP path
        cmppath = self.pathBuilder('/cmp/user/cosmo-multigetTestAccount') 
        
        #Create testing account      

  
        bodycreateaccount = '<?xml version="1.0" encoding="utf-8" ?> \
                                 <user xmlns="http://osafoundation.org/cosmo/CMP"> \
                                 <username>cosmo-TestAccount</username> \
                                 <password>cosmo-test</password> \
                                 <firstName>cosmo-test</firstName> \
                                 <lastName>TestAccount</lastName> \
                                 <email>cosmo-test@osafoundation.org</email> \
                                 </user>'

 
                                 
        #Create account and check status
        self.request('PUT', cmppath, body=bodycreateaccount, headers=cmpheaders) 
        self.checkStatus(201) # 201 ACCOUNT CREATED 

        self.headers = self.headerAddAuth("cosmo-TestAccount", "cosmo-test") 1

        #Create Calendar on CalDAV server   
        self.calpath = self.pathBuilder('/home/cosmo-TestAccount/calendar/' % self.appendUser)
        self.request('MKCALENDAR', self.calpath, body=None, headers=self.headers)
        self.checkStatus(201)

        # ------- Test Creation of events view ICS ------- #
        
        #Construct headers & body
        self.testStart('Put 1-4.ics')
        puticsheaders = self.headerAdd({'Content-Type' : 'text/calendar'})      
        put1icspath = self.pathBuilder('/home/cosmo-multigetTestAccount%s/calendar/1.ics' % self.appendUser)
        put2icspath = self.pathBuilder('/home/cosmo-multigetTestAccount%s/calendar/2.ics' % self.appendUser)
        put3icspath = self.pathBuilder('/home/cosmo-multigetTestAccount%s/calendar/3.ics' % self.appendUser)
        put4icspath = self.pathBuilder('/home/cosmo-multigetTestAccount%s/calendar/4.ics' % self.appendUser)        
        f = open("files/reports/put/1.ics")
        put1icsbody = f.read()
        f = open("files/reports/put/2.ics")
        put2icsbody = f.read()
        f = open("files/reports/put/3.ics")
        put3icsbody = f.read()
        f = open("files/reports/put/4.ics")
        put4icsbody = f.read()
        
        #Send request and check status
        self.request('PUT', put1icspath, body=put1icsbody, headers=puticsheaders)
        self.checkStatus(201)
        self.request('PUT', put2icspath, body=put2icsbody, headers=puticsheaders)
        self.checkStatus(201)
        self.request('PUT', put3icspath, body=put3icsbody, headers=puticsheaders)
        self.checkStatus(201)
        self.request('PUT', put4icspath, body=put4icsbody, headers=puticsheaders)
        self.checkStatus(201)

    def recurringRun(self): 2
        
        # ------- Test 1.xml : basic VEVENT, summary "event 1" (tzid=US/Eastern) ------- #
        
        self.testStart('Test 1.xml : basic VEVENT, summary "event 1" (tzid=US/Eastern)')
        
        #Setup request 
        f = open('files/reports/multiget/1.xml') 3
        report1body = f.read()
        self.request('REPORT', self.calpath, body=report1body, headers=self.headers) 
        self.checkStatus(207)
        
        #Verify correct items in response
        self.verifyDAVResponseItems(['1.ics', '2.ics', '3.ics', '4.ics'], inelement='{DAV:}getetag', positive=['']) 4
        self.verifyDAVResponseItems(['1.ics', '2.ics', '3.ics', '4.ics'], inelement='{urn:ietf:params:xml:ns:caldav}calendar-data', positive=['UID'])
        self.verifyDAVResponseInElement('1.ics', '{urn:ietf:params:xml:ns:caldav}calendar-data', positive=['UID:54E181BC7CCC373042B28842@ninevah.local']) 5
        self.verifyDAVResponseInElement('2.ics', '{urn:ietf:params:xml:ns:caldav}calendar-data', positive=['UID:9A6519F71822CD45840C3440@ninevah.local'])
        self.verifyDAVResponseInElement('3.ics', '{urn:ietf:params:xml:ns:caldav}calendar-data', positive=['UID:DB3F97EF10A051730E2F752E@ninevah.local'])
        self.verifyDAVResponseInElement('4.ics', '{urn:ietf:params:xml:ns:caldav}calendar-data', positive=['UID:A3217B429B4D2FF2DC2EEE66@ninevah.local'])

• 1
  • Don't forget to add auth headers to self.headers for the test account you just created.
• 2
  • runRecurring() should contain all your tests that can be run multiple times with the same response from cosmo.
• 3
  • We like to keep cosmo automation code pretty lean. With the exeption of the user creation string we try to keep big body strings in seperate files that we read in during the test execution. All of those files should kept the cosmo repository under test/functional/HTTPTest/files/ .
• 4
  • self.verifyverifyDAVResponseItems is a method in DAVTest. The method is used for verifying a variety of proper data in a DAV MultiStatus^? response. The method is fairly complex and borders on "magic".
  • The first argument is a list containing strings which will be searched for in each {DAV:}response element. If one string is not found it will report failure. If more elements exist than searched for the method will report failure.
  • The second argument is inelement='element to seach in'. This element should be inside each {DAV:}response. If one {DAV:}response doesn't contain this element the method will report failure.
  • Additional arguments are positive=List and negative=List. Positive should be a list to verify each string is in inelement. Negative should be a list to verify each string does not exist in inelement.
  • As you can see the example above verifies that each {DAV:}response has an etag, the following test method verified each calendar-data element has a UID.
• 5
  • self.verifyDAVResponseInElement is a method in DAVTest. This method is used for testing DAV multistatus response elements.
  • The first argument is a string which will be searched for in each {DAV:}response element. The first one found will be used for testing.
  • The second argument is the element you want to search in.
  • Additional arguments are positive=List and negative=List. Positive should be a list to verify each string is in the element. Negative should be a list to verify each string does not exist in the element.

Backing up

That was a lot to take in but now we having a functioning test case. Before we go in to more detail about how the framework works we should retrofit the case we just wrote to work inside the cosmo stress test framework.

The issue is that startRun() is meant to run on a clean repository only once and all the tests within startRun() will get different responses if we run them again. The stress test framework uses a pool of threads and creates many instances of these test classes and runs the test methods against the same cosmo server, so our test class will have to be have to be changed in some way to work inside the stress test framework.

You may have noticed by now that none of the test tools and framework are cosmo specific, they contain HTTP and DAV specific functionality but nothing specific to cosmo. To accomplish cosmo specific tasks we simply use some simple conventions and leverage product independent features in the framework.

One feature in the framework we haven't discussed is the append variables. During test class instantiation a dictionary and list called self.appendDict and self.appendList are created in each class instance. The cosmo stress framework uses the appendDict to define a username append string that should be appended to each username in cosmo test cases.

Below is an example of user creation that will work inside our cosmo stress framwork.

class CosmoTest(DAVTest): 

    def startRun(self):
        
        self.testStart('Setup Accounts') 
            
        # ------- Test Create Account ------- #

        try:
            self.appendUser = self.appendDict['username'] 1
        except KeyError:
            self.appendUser = '' 2
           
        cmpheaders = self.headerAdd({'Content-Type' : "text/xml; charset=UTF-8"}) 
        cmpheaders = self.headerAddAuth("root", "cosmo", headers=cmpheaders) 
           
        #CMP path
        cmppath = self.pathBuilder('/cmp/user/cosmo-multigetTestAccount%s' % self.appendUser) 3
        
        #Create testing account      

  
        bodycreateaccount = '<?xml version="1.0" encoding="utf-8" ?> \
                                 <user xmlns="http://osafoundation.org/cosmo/CMP"> \
                                 <username>cosmo-%sTestAccount</username> \
                                 <password>cosmo-test</password> \
                                 <firstName>cosmo-test</firstName> \
                                 <lastName>TestAccount</lastName> \
                                 <email>cosmo-test%s@osafoundation.org</email> \
                                 </user>'(self.appendUser, self.appendUser)

4
                                 
        #Create account and check status
        self.request('PUT', cmppath, body=bodycreateaccount, headers=cmpheaders) 
        self.checkStatus(201) # 201 ACCOUNT CREATED 

        self.headers = self.headerAddAuth("cosmo-%sTestAccount" % self.appendUser, "cosmo-test") 5
        self.calpath = self.pathBuilder('/home/cosmo-%sTestAccount/calendar/' % self.appendUser) 6
        self.request('MKCALENDAR', self.calpath, body=None, headers=self.headers)
        self.checkStatus(201)

• 1
  • self.appendDict['username'] is the convention we use for username appending inside the stress test framework. The problem is that if this isn't defined, like when we run this test outside the stress framework, we want it to work the same as before
• 2
  • Now we make sure that self.appendUser is set to an empty string if this test isn't running inside the stress framework.
• 3
  • The CMP path must be appended as well, as of cosmo 0.3
• 4
  • Cosmo only needs two unique strings in each user creation, username and email address.
• 5
  • During our headerAddAuth we need to make sure we are appending the user as well
• 6
  • self.calpath will also need to be changed to reflect the new username

Notice that we create variables that are bound to the class. The reason is that durring our recurringRun() we will most likely need to know the calpath and userAppend strings during various tests.

Great, so how does all that work

Most test cases can be written using what we've already discussed. But some more advanced test cases require more knowledge about how DAVTest and the frameworks we written to run these cases. This knowledge can also help tremendously when debugging tests.

Behind the scenes a lot of stuff is going on. First off you need to understand where all of this functionality comes from. When expaining the various test methods earlier we made it a point to note which class they are derived from.

You test class should inherit from DAVTest, DAVTest inherits from HTTPTest and doesn't have it's own __init__() method.

HTTPTest inherits from TestObject^?. HTTPTest has it's own __init__() method that looks like this:

class HTTPTest(TestObject):
    
    def __init__(self, host, port, path, debug=0, headers=None, tls=False, mask=0, recurrence=1, appendDict={}, appendList=[], appendVar='', printAppend='', threadNum=None): 1
        
        TestObject.__init__(self, debug=debug, mask=mask, recurrence=recurrence, appendVar=appendVar, printAppend=printAppend, threadNum=threadNum, appendDict=appendDict, appendList=appendList) 2
        
        if headers is None:
            self.headers = {'Host' : "localhost:8080",
                             'Accept' : "*/*"}
        else:
            self.headers = headers 3
        
        self.connection = {"host" : host, "port" : port, "path" : path} 4
        self.request('OPTIONS', path, body=None, headers=self.headers) 5

• 1
  • This seems a bit large but it's really not, most of it is offloaded on to TestObject^?.__init__().
  • The important variables are host, port, and path. Since TestObject^? is generic and can be used for testing just about anything these connection specific variables are needed for testing HTTP.
• 2
  • Here is where we intialize all the necessary functionality in TestObject^?. More on all this later.
• 3
  • This is where we set some sane defaults for self.headers.
• 4
  • self.connection is a dictionary containing all the necessary information for the server we will be connecting to.
• 5
  • This is a quick and dirty sanity test. Since self.request doesn't have any exception handling this will fail in python if the server is unavailable. Although this solves many headaces you might run in to if you've misconfigured the test run, it doesn't solve all of them. There is no status code check so any response will cause the instantiantion to succeed.

Now on to test object.

Although there is not cosmo specific test methods in any of the tools we've developed, they were initially developed for cosmo testing. TestObject^? is where solved the majority of practical problems of debugging and running the same set of test classes in a variety of ways. For instance all the current cosmo test classes are run in the following ways:

• Full Suite. This runs every cosmo test class, supressing all the test output until the end, then tallies up all of the results and prints a summary of the test suite run.
• Single Tests When debugging an issue we will run a single test, watching each pass and fail individually and investigating.
• Recurring Tests Running a single test class or many test classes with large amount of recurring runs, this is used for a quick and dirty stress test of cosmo.
• Full Stress Testing We spawn multiple threads, each runs every cosmo test class over and over again on a loop each class instance running recurring tests.

This has required us to build a lot of features in to TestObject^?; output masking, debugging flags, recurring test runs, append variables, and results handling to name a few. Below is the TestObject^? initialization method.

class TestObject:
    
    def __init__(self, debug=0, mask=0, recurrence=1, appendVar='', printAppend='', appendDict={}, appendList=[], threadNum=None):
        
        self.debug = debug
        self.mask = mask
        self.results = []
        self.resultNames = []
        self.resultComments = []
        self.recurrence = recurrence
        self.appendVar = str(appendVar)
        self.printAppend = printAppend
        self.threadNum = threadNum
        self.appendDict = appendDict
        self.appendList = appendList

Most of this you will never care about. But some of it may be relevant to an in depth test case you need to write.

Methods

Since DAVTest is an abstraction the xml parsing and pass/fail reporting is all handled for you. But it would be stupid to think that this abstraction will cover all the test cases you may want to write. Below is a list of methods in HTTPTest and TestObject^? along with simple descriptions.

TestObject^? Methods

printOut(self, string)

This method is used for all printing by other test methods. If masking is set the output is supressed.

testStart(self, testname)

This method is used to signal the begining of a new test. At the time of this writing the only thing relevant the method does is print the name of the test starting if debugging is set. Eventually this will be used to calculate the number of specific tests in a test class instead of assuming each self.report() call is an indiviual test.

report(self, result, test=None, comment=None)

This method is used to report pass or fail and will print failures if masking == 0. It will output all results if debugging > 0.

runRecurring(self)

This method runs the recurringRun method of you test class. It uses the self.recurrence value to decide how many time to run the recurring tests.

end(self)

This method prints out all the passes and failures in your test class and gives you a summary.

fullRun(self)

This method will your test functions in the following order; startRun(), runRecurring(), end().

HTTPTest Methods

headerAdd(self, headers)

This method takes a dictionary and returns a dictionary containing headers+self.headers.

headerAddAuth(self, username, password, headers=None)

This method takes a dictionary of headers and adds auth headers for username and password. If headers is not specified it will use self.headers.

pathBuilder(self, path)

This method takes a string and returns that string the self.path prepended to it.

checkStatus(self, status)

This method checks the status code in self.test_response to see if it matches status.

xmlParse(self)

This method will parse self.test_response.read() as xml using elementtree. The elementtree object created after successfull xml parsing is assigned to self.xml_doc

request(self, method, url, body=None, headers={})

The method sends an HTTP request to the server defined in self.connection. If no body is defined it will use None, if no headers are sent it will use an empty dictionary. Since this uses httplib some headers that are not defined may be added. For example Content-Length may be added to many requests by httplib.HTTPConnection