S60 Open to new features

As part of a now-yearly review, planning and scheduling of our
“autotest” project, I undertook a Google-review of href="http://www.google.com/search?hl=en&client=safari&rls=en-us&q=automated+test+development&btnG=Search">automated
test development.  Like href="http://www.google.com/search?hl=en&client=safari&rls=en-us&q=software+development+method&btnG=Search">software
development methods, the models for developing automated tests are
continuously changing.  The consensus seems that software
development companies should pursue somewhere between 100% automated
regression and integration tests to just enough automation that brings
value to test and QA, with “value” being defined in numerous fuzzy,
long or complex ways.  Everyone who writes on the subject at least
agrees that the right type of automated testing can provide high
positive net value to a software development organization, but what
value can be provided by what tests is an answer that can only be
provided in the context of what is being tested.

Wikipedia’s href="http://en.wikipedia.org/wiki/Software_testing_controversies">Software
Testing Controversies [as of publication of this blog entry]
summarizes the questions facing testers, developers and managers
today.  Should testing be href="http://en.wikipedia.org/wiki/Agile_software_development">Agile
or traditional, exploratory or scripted, manual or automated? Our
organization has had to answer all the issues, and oh, the difficulty!

Traditional testing refers to structured href="http://en.wikipedia.org/wiki/Waterfall_model">waterfall and
heavy documentation development models such as href="http://en.wikipedia.org/wiki/Capability_Maturity_Model">CMM
or test document standard href="http://en.wikipedia.org/wiki/IEEE_829">IEEE 829.  These
models grew out of the regulatory environment for military, medical and
avionic software.  Organizations operating in a regulatory
environment have more legal incentive to create very detailed manual
test cases.  The more vangarde “anti-waterfall” methods are
collectively referred to as “Agile” and include methods such as href="http://en.wikipedia.org/wiki/Extreme_programming">Extreme
Programming (XP), href="http://en.wikipedia.org/wiki/Scrum_%28development%29">Scrum,
Test
First or Test-driven Development, href="http://en.wikipedia.org/wiki/Lean_software_development">Lean
and Open Source.

Exploratory testing refers to free-style testing by domain
experts.  Test specifications can be kept loose and vague, with
the assumption that product test exports will use the test outlines to
run new and novel tests each time they test.₁   Other
organizations require very detailed test specifications that are
executed exactly as written.  Kaner argues across href="http://www.kaner.com/articles.html">his many papers that
exploratory testing by experts finds more bugs, but that less
information on test coverage can be recorded and that good testing gets
tied to key employees who are the domain experts.

Proponents of Agile development methodologies such as XP (eXtreme
Programming) and “Test First” advocate “full coverage” of the unit
tests or put another way, “Test everything that could break.”₂
  However, any significant prose on what full coverage of
automated tests really means will reveal that “full coverage” can’t
even be fully defined because it presupposes knowledge of every branch,
state and error condition.  University and corporate research into
software test methodologies has shown that full regression automated
test suites work best as “change detectors” but are less effective at
finding new bugs.₃ 

Informal corporate research revealed that “Over 88% of the bugs were
found during the manual testing that was done as part of the
development of the [automated regression tests.]“₄  
Another
limitation of an automated regression suite is it’s lack of ful test
coverage.   Even achieving 100% code coverage in automated test
suites does not find all bugs.  “Dick Bender’s Soft Test program
provides estimates of [10⁵⁰] possible test cases of a
program. Some programs have yielded estimates as high as 10¹⁰⁰…
How can we say that rerunning 10,000 [automated regression test cases,]
that the program has passed before should give us confidence that it
won’t fail any of these many others? (See also Marick, 1997.) ” ₄

One issue of particular importance to the Carbide.c++ team is
configuration testing.  Carbide.c++ supports so many platforms,
debug interfaces, and operating systems both host and target. 
Validation of Carbide.c++ on so many tasks is an expensive and
time-consuming process.  Automation enables test suite execution
across many platforms.  The execution of existing automated tests
against a broad variety of configurations leverages the power of a
single test suite many fold.  “You can save time, reduce human
error, and obtain good tracking of what was done by automating
configuration and compatibility testing… you might recover the cost
of automating [testing of 30 or more configurations] in less than a
week.” ₄

Nokia is committed to modern Agile development methodologies and test
automation.  Research into modern software and test development
methodologies, and implementation of what works best, has been and
continues to be a goal of this oganization.  The Carbide.c++
development and QA department are committed to utilizing the software
testing libraries available for automated testing of the Eclipse
platform with unit tests, regressions tests and expanding our test
coverage.