[NEW ISSUE] TestCase client - need to add capability for Test Bridge tocheck the actual error returned by the runtime.

[Mike Edwards <mike_edwards@uk.ibm.com>]

Raiser:        
       Mike Edwards

Target:        
       sca-assembly-1.1-testcases-cd01.pdf

Description:

Currently, the test suite Testcase client
runner code only checks that some error has occurred and does not check
any details of the
error.  

Folk who have runtimes are requesting
that their runtime bridge code should be able to check the actual error
code returned for any
particular testcases so that they can
ensure that their runtime is generating the correct error for those testcases
that expect error responses.

Proposal:

The design proposed is to add an extra
method to the RuntimeBridge interface that allows for checking of the actual
error returned.

This method is called by the Base testcase
runner whenever it receives an error running a testcase and the error was
an expected one.
The runtime Bridge code implements the
method.  The implementation must contain all the information required
to evaluate whether
the error is the correct one for the
testcase being run.  The expectation is that if the method simply
returns, then everything is OK (and
so a default empty implementation would
always indicate "ok"), but if the error is incorrect, an exception
is thrown - ideally this will be
a JUnit exception such as is generated
by the JUnit Assert methods, since the base test runner uses JUnit in its
implementation.

1) Extended RuntimeBridge class, with
new checkError method....

public
interface
RuntimeBridge {

        /**
         *
Sets
the
Test
configuration
for
a
particular
test
run
         *
@param
testConfiguration
         */
        public
void
setTestConfiguration(TestConfiguration testConfiguration);

        /**
         *
Start
the
contribution(s)
which
represent
the
application
under
test
in
the
SCA
runtime
         *
@param
contributionLocation
a
String
containing
the
location
pattern
for
the
contribution(s),
         *
which
is
a
URI
containing
one
or
more
"%1"
substrings
where
the
name
of
the
contribution
is
         *
substituted
to
get
the
URI
of
the
contribution
         *
@param
contributionNames
-
an
array
of
contribution
names
as
strings
         *
@return
true
if
the
contributions
were
loaded
and
started
in
the
SCA
runtime,
false
otherwise
         */
        boolean
startContribution(String contributionLocation, 
         
               
                String[] contributionNames)
throws
Exception;

        /**
         *
Stop
the
contribution(s)
         */
        void
stopContribution();
        
        /**
         *
Have
the
runtime
bridge
check
runtime
specific
exceptions
to
see
         *
if
the
exception
that
was
thrown
was
expected
for
the
specified
test.

         *
How
this
check
is
performed
is
runtime
specific
         *
 
         *
@param
testName
the
name
of
the
test
being
run,
e.g.
ASM_4001
         *
@param
exception
the
exception
object
that
was
caught
         *
@throws
Throwable
if
the
exception
that
was
received
was
not
expected
or
was
         *
the
wrong
exception
         */
        public
void
checkError(String testName, Throwable exception) throws
Throwable;        
        
} //
end interface RuntimeBridge


2) Modified BaseJAXWSTestCase class,
which calls the new checkError method...

...
/**
 *
A
generic
test
client
based
on
JAX-WS
APIs
 */
public
class
BaseJAXWSTestCase {

...
    
    /**
     *
Set
up
the
runtime
-
pass
the
configuration
to
the
runtime
bridge
and
then
start
the
contribution(s)
which
represent
     *
the
SCA
application
     *
@throws
Exception
     */
    @Before
    public
void
setUp() throws
Throwable {
         
  runtimeBridge.setTestConfiguration(getTestConfiguration());
         
  //
Don't proceed to run the test unless the contribution loaded successfully...

         
  proceed
= false;
         
  try
{
         
          if
( startContribution() ) proceed
= true;

         
  } catch
(Exception e) {
         
          //
If the SCA runtime refuses to start an invalid contribution, then this
is also
         
          //
regarded as a successful outcome, if the expected outcome is marked as
"exception"
         
               
 System.out.println(
"Exception received
during startup - detail: "
+ e.getMessage() );
         
  checkException(e);
         
  } //
end try
    }

...

    /**
     *
Run
the
test,
if
the
initialization
of
the
SCA
application
was
successful
     *
@throws
Exception
-
if
the
output
from
the
application
does
not
match
the
expected
output
     */
    @Test
    public
void
testDummy() throws
Throwable {
         
  //
If an exception were thrown during initialization, let's go no further
         
  if(
proceed
== false
) return;
//TODO
- check, should we be failing here?
         
  
         
  //
System.out.println("Test " + testName + " starting");
         
  String output = null;
         
  try
{
         
          output = invokeTest( testConfiguration.getInput()
);

         
  } catch
( TestException_Exception e ) {
         
          TestException exceptionContent
= e.getFaultInfo();
         
          System.out.println("Service
fault received - detail: "
+ exceptionContent.getMessage() );
         
  checkException(e);
         
          return;
         
  } catch
(Throwable e) {
         
          //e.printStackTrace();
         
               
 System.out.println(
"Exception received
- detail: " + e.getMessage()
);
         
  checkException(e);
         
          return;
         
  }
         
  //
If there are multiple outputs possible, check each one until there is a
match
         
  String[] expectedOutput = testConfiguration.getExpectedOutput();
         
  AssertionError e = null;
         
  
         
      for(
String expected : expectedOutput ) {
         
              e = null;
         
          try
{
         
               
  assertEquals( expected, output );
         
          } catch
( AssertionError eReturned ) {
         
               
  e = eReturned;
         
          } //end
try
         
          if
( e == null
) break;
         
      } //
end for 
         
      if(
e != null
) throw
e;
         
  System.out.println("Test
" + testConfiguration.getTestName()
+ " completed
successfully");
    }
    
...
    
    public
void
checkException(Throwable e) throws
Throwable{
         
      assertEquals( testConfiguration.getExpectedOutput()[0],
"exception"
);
         
      
         
      runtimeBridge.checkError(testConfiguration.getTestName(),
e);
         
      
         
      System.out.println("Test
" + testConfiguration.getTestName()
+ " completed
successfully");
        
    } //
end method checkException

    /**
     *
Start
the
supplied
contribution(s)
via
the
Runtime
Bridge
     *
@return
true
if
the
contributions
were
started
successfully
     *
@throws
Exception
     */
...

} //
end class BaseTest

3) Example implementation of the checkError
method in the Tuscany runtime bridge:

        public
void
checkError(String testName, Throwable ex) throws
Throwable {       
        String
expectedMessage = expectedErrorMessages.getProperty(testName);
        String
receivedMessage = ex.getMessage();
        
        if
(expectedMessage == null){
         
  fail("Null
expected error message for test "
+ testName + 
         
           "Please
add message to oasis-sca-tests-errors.properties");
        }
        
        if
(receivedMessage == null){
         
  fail("Null
received error message for test "
+ testName);
        }
        
        int
messageStart = receivedMessage.indexOf("]
- ");
        
        if
(messageStart < 0){
         
  fail("Message
separator not found for test "
+ testName);
        }
        
        receivedMessage
= receivedMessage.substring(messageStart + 4);
        
        assertEquals(
expectedMessage, receivedMessage );
        
        return;
    } //
end method checkError


Yours,  Mike.

Strategist - Emerging Technologies, SCA & SDO.
Co Chair OASIS SCA Assembly TC.
IBM Hursley Park, Mail Point 146, Winchester, SO21 2JN, Great Britain.
Phone & FAX: +44-1962-818014    Mobile: +44-7802-467431  
Email:  mike_edwards@uk.ibm.com

---

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU