Open Automation Framework (OAF)

This document describes a proposal for a distributed test framework which eventually will be used by all automated functional test tools at OSAF.

Background

This document was started back in February 2006 and signed off on by development (primarily Chandler development). Since other issues have dominated QA's time since then not much has been done on this framework, but many tools have been written for the various products with this framework in mind and some of the issues this framework was intended to solve have been solved at a smaller scale (CATS 0.2). With the transition in focus at OSAF to the 'ecosystem' and the upcoming hosted service some of the priorities for a new framework have shifted, along with a necessity to replace the current "micro-frameworks" of our new test tools.

Proposed Architecture for OAF (Open Automation Framework)

The proposal for this architecture is below. To get to these features we will go through three dot version of the framework outlined at the end of this document.

Requirements for architecture:

• Tests must be easily writeable. The test framework is intended mainly to provide a common framework for writing _test tools_ but in doing so lays down some requirements for how the tests will need to be written using those tools. It is important that those requirements be simple and easy to learn.
• All current test tools (broadsword tool suite (HTTPTest, JSONTest, etc), CATS 0.2) must be ported to use this framework easily This is less of an issue than it seems since in the time since February we have replace the old CATS framework and developed new tools for testing Cosmo and Scooby with this framework in mind.
• Extending/Enhancing framework and/or libraries should be easy and not impact other aspects of the test system. Currently adding features to most test tools and their respective micro-frameworks is a difficult process and not much code exists to help facilitate this development.
• Framework should support distributed computing. This is an obvious although highly ambitious goal. The framework would need to be able to creat a cosmo user and add events (HTTPTest), bring up chandler and view/edit/add events to that calendar (CATS) and then verify that scooby can see and edit all those events (JSONTest).
• Framework should include deliverables for all pieces of the framework This includes a server for managing multiple resources and running tests, example scripts and tools, and full documentation.

Single framework client instance flow chart:

• OAF Flow Chart:
  

Color Chart

• Light Blue - Optional input
• Blue - Required input
• Red - Built in to Framework
• Purple - Plugin for Framework
• Yellow - Processes run by framework as defined by user input

Test Script layout

Scripts will have to conform to a very simple style.

from TestTool import TestCaseClass

class TestingExample(TestCaseClass):

    def startTest(self):

        test_method_creations()

    def recurringTest(self):

        test_methods()

TestClass.startTest() is run at the beginning of the test and only run once on a single test run. TestCalss.recurringTest() is run as many times are 'recurrence' is set in the framework. And example would be if a test needs to first enable and add events only once in the beginning, then you could use GET or REPORT on those methods as many times as the user wished with the same results.

OAF state encapsulations

All test frameworks should have a notion of state. Am I inside a test, is that in a suite, which suite, etc. OAF has three tiers of encapsulation Suite, Test, and Action.

You can have multiple suites, containing multiple tests which contain multiple actions. Individual PASS/FAIL "report" calls MUST be made within actions. In fact if the framework receives a report call when it is not within an action is creates one for the encapsulation. Test writers and tools developers define the action state and make report calls, the framework handles the suite and test state management.

Note: The framework doesn't allow for a tests writer or tool developer to break the framework state. If actionStart() is called twice without actionEnd() the framework will close down the previous action. If for any reason a test fails in python the framework will end the test, log the failure and the traceback, cleanup state, and move on to the next test.

Modular Standardized Output

Eventually OAF will be able to have many custom configurations for it's output which can be different for stdout/file/SQL etc. This feature is not a priority for first three dot versions and will prioritized after the initial development work is done although the design and development of the framework will keep this in mind.

What is a requirement is that the output be incredibly standardized. ALL framework output must be in the form of OAF compliant tuples which will be described in another document. The reason for this standardization is so that they can be easily pushed over the network to a controlling server during distributed testing.

Distributed Testing

Distributed computing is not easy, and therefor neither is writing a distributed testing framework. There are many method and a few successfull frameworks. One such framework is STAF and many of the concepts for this distributed test framework come from STAF.

One of the major requirements for STAF was that it be able to work with a variety of programming languages. Because of this requirement they engineering a protocol for sending certain datatypes over the network (this was before SOAP) and wrote libraries in each language to support this.

All the test tools at OSAF are written in Python and in some cases even encapsulate data from other sources written in languages other than python. We intend to continue using Python and want to leverage the full power of Python to the best of our ability. This has allowed us to architect a much simpler and dynamic distributed testing framework than STAF could achieve.

One technology we intend to use is Pyro which is a Distributed Object Technology system for python. With Pyro we'll be able to run remote methods in test objects without adding any extra code in the tests themselves. We also won't have to do any of our own serialization or write any XML requests for executing remote methods, we get to call remote python within python.

Release Plan

• 0.1 (Drop in replacement)
  • Release MUST be able to replace the micro frameworks for broadsword-0.2 and CATS-0.2.
• 0.2 (Distributed testing Beta)
  • Release MUST support distributed testing and will be used to begin creating our own in house test farm.
• 0.3 (Distributed testing Final with additional support for end to end performance analysis)
  • Release MUST be a full end to end distributed testing framework. In addition there should be some framework level support for easy end to end performance analysis of output from multiple resources.