ࡱ> M Kbjbj== !~WWCl8J,v1N^t:D$MMMMMMM$P =RM9M&M&&&@M&M&&!2CDYK  Pvs6(,rK.KN01NK.RRK&Describe methods of ensuring that software is reliable - testing, testing, agreements between software houses and purchaser for testing. Understand the need for maintenance release(s). Understand the reasons why fully tested software may fail to operate successfully when implemented as part of an information technology system Methods of ensuring that software is reliable - chapter 62 Remember the following stages used in ICT 3 and ICT6: 1. Unit Testing - where each individual component is tested. 2. Module Testing - a module is a collection of dependant components or sub-routines. 3. Subsystem Testing - collections of modules are integrated into modules. 4. System Testing - Subsystems are integrated to make up the entire system. This stage of testing will examine whether or not the system meets its requirements specification. 5. Acceptance Testing - This is the final stage in the testing process before the system is accepted for operational use. These stages will have to be repeated when modifications are made. Make or buy? P.335 At the start of the systems life cycle, decisions have to be made about how to acquire the software that is needed. The options are: Software can be written by the end user A specialist department could design, write, test and evaluate the software External consultants could be called in to write and test the software An off-the-shelf package could be bought Software could be leased, with an annual fee payable for use. End-user written software p.335 Computer-literate end-users can produce their own applications, using software such as Excel or Access. Their applications will be designed to do a specific job. The advantages of this approach are that the end-user will know his requirements precisely and he should get exactly what he wants without waiting. Disadvantages are that the end-user could leave and then wouldn't be available to provide technical support. His application may be incompatible with other software used by the organisation. This approach is only suitable for minor projects with a limited life-span. Writing software in-house Advantages are that information can be kept within the organisation. Disadvantages are that this approach requires people with certain skills who might need to be recruited. An external consultancy may have more experience of specialist skills External consultants p.336 The job could be put out to tender. Cost is a consideration but a software house that has a proven track record of successful projects might be worth the extra money. Buying a package p.336 The advantages of buying an off-the-shelf package are: Cheaper Immediately available Documentation is usually available Training courses may be available Technical support is usually available The organisation can look at reviews of the product and/or talk to other users Upgrades are usually available on a regular basis The disadvantages are: The package may not do exactly the job that is required. It may be incompatible with hardware or software that is already being used. Leasing Software p.336 Instead of buying software, it can be leased for an annual fee. The cost is less in the short-term but in the long run this is a more expensive option. Modifying existing packages p.336 Software can be bought and then modified. This is a dangerous option because it means that the manufacturer will no longer provide support. The modifications can cause errors in other parts of the software, which would then require more modifications.  HYPERLINK "http://search.internet.com/www.webopedia.com?IC_QueryText=testing&IC_QueryDatabase=www.webopedia.com" \t "_blank" Internet.com PC Webopedia definitions and links for testing - white box, black box, alpha and beta.  HYPERLINK "http://www.thekjs.essex.sch.uk/yates/it05-7.htm" \t "_blank" The KJSSome highly recommended class notes entitled "Software Acquisition and Testing."  HYPERLINK "http://www.ondaweb.com/sti/" \t "_blank" Software Testing Institute  HYPERLINK "http://www.softwaretestinginstitute.com/" http://www.softwaretestinginstitute.com/ Homepage of the STI. Loads of articles and links to material concerned with software testing. http://www.aptest.com/resources.html Free evaluation copies of software testing tools. These provide some useful insights.  HYPERLINK "http://www.softwareqatest.com/" http://www.softwareqatest.com/ Truly hideous looking site! Don't be put off as this site contains some excellent material. Problems in using software - BUGS Most choices in a program route include a series of steps that contain further choices and steps creating many complex paths. A program of several hundred lines can have tens of such choices and paths. Important programs within an organisation will have thousands or even millions of lines as well as hundreds or thousands of decision points. With so many lines of code and decision points a large number of errors will creep in. The consequences of these errors in code (or BUGS - remember Grace Hopper) can be expensive and in certain cases fatal to life or to the business. Why do errors occur? Typing errors - almost certain to happen many times in dealing with tens of thousands of lines of code. Lack of clarity in program specification Misunderstanding by the end-users concerning what actions are needed under every possible combination of circumstances. Featuritis - glue piles of features without thinking about integration and how separate parts interact. Programmers take undocumented shortcuts that sometimes are not supported in future releases. Similarly programmers dont follow standard interface guidelines. Improper use of a complex programming technique to code a tricky passage Inadequate testing - leaving testing until coding is almost finished makes it difficult to turn back original design considerations. Poor Documentation leaves programmers who come back to edit programs in distress. Pressure of time usually commercial Cost - each line of the space shuttles flight control software costs NASA about $1000 or 10 times more than for typical commercial software. Would you buy a word-processor or a spreadsheet for $5000 no matter how bug free it was? Or would you pay 90% less and live with the bugs? Famous examples of bugs. In 1945 Grace Hopper who spearheaded the standardisation of COBOL was asked to investigate why the computer she was working on had stopped. After looking inside the computer, she found a bug. It was a moth that had been beaten to death by an electrical relay signal. The moth was removed by tweezers (debugged). Summer of 1991 - telephone breakdowns in California and along Eastern seaboard - errors in signalling software. - immediately before breakdown a bug was introduced by changing 3 lines of code in the several million line signalling program. After this tiny change no one thought it necessary to retest the program. Commercial pressure - give new feature NOW. In 1986 two cancer patients in East Texas received fatal doses of radiation from the Therac-25 computer controlled radiation therapy machine. There were several errors including the failure of programmer to detect mis-coordination between concurrent tasks. FAMOUS CASE Solutions The only way to make certain no errors exist is to test every line of code and every option. (See the diagram on P.337) However studies indicate that in a large application (e.g. payroll, hotel reservation system) would require thousands of years even on the most powerful computers available. This is NOT a realistic option, therefore: - Sweat over the design specification - managers and customers often find it difficult to specify how a proposed program is supposed to perform. Cleanly divides up tasks - makes it easier to co-ordinate and integrate. Structured programming - modularised. When bugs appear they are easily isolated and corrected. If necessary whole sections can be changed without affecting the basic logic. Avoid shortcuts Use comments liberally to describe code Test extensively - BOTH the individual components and the interworkings of the entire system. But note - 400 people spent five years working on the Boeing 777 flight-control software. Independently validate the product Include backup systems Technology-change and process-change within the organisation need to be managed Syntax errors. A compiler will report badly formed statements after it has tried to compile the source code; an interpreter will report illegal statements as it attempts to execute them Logic errors - test data is created to try to trap errors. Normal data - the most general data for which the program was designed to handle. Extreme values - test behaviour of the system at the upper and lower limits of acceptability. Zero or null values important here. Exceptional data - illegal data - particularly where non-programmers use systems. Methods used to test Software p. 337 a Testing (alpha testing) Commercial software is traditionally developed to an incomplete state with some questions of design left unresolved. It is then issued to a restricted audience of testers usually within the developers own company. Testing in-house i.e. with test data provided by software house, this is needed to test the implementation against design specification b. Testing (beta testing) Beta testing is done when software is being prepared for release. When changes on alpha test have been made the software is then issued to a number of privileged customers in exchange for their constructive comments. e.g. authors of how-to manuals, computer magazine reviewers. Testing is needed to detect errors not identified at the alpha stage, this is top-down testing i.e. real users become testers Another method might be to use off-site software house with `live data Several beta versions are often released to iron out problems Agreements May exist between software houses and the purchaser for testing, in return purchaser is involved in development of the product, is given discounted prices and priority for upgrades or bug-fixes. Maintenance Releases p. 338/9 The maintenance phase of a system begins once a system goes into production and lasts for as long as the system continues to be used. The maintenance process is triggered by requests from users who may report bugs or request new features. The cost and impact of each change will be assessed before a decision to go ahead is authorised. Minor changes in software packages are usually released with version numbers such as 3.1, 2.3, 3.3 etc. Major releases have new version numbers e.g. 4.0, 5.0. A number of reasons may trigger maintenance Previously undetected errors may be discovered in the software The original requirements may change to meet the changing needs of the user. Hardware developments may give scope for advances in software. New legislation may be introduced, which impacts on software (e.g. a new tax will affect accounting software) Lehman and Belady 1985 p.339 The law of continuing change A program that is used in a real-world environment necessarily must change or become progressively less useful in that environment. The law of increasing complexity As an evolving program changes, its structure tends to become more complex. Extra resources must be devoted to preserving and simplifying the structure. The law of large program evolution Program evolution is a self-regulating process. System attributes such as size, time between releases and the number of reported errors are approximately invariant for each system release. The law of organisational stability Over a programs lifetime, its rate of development is approximately constant and independent of the resources devoted to system development. The law of conservation of familiarity Over the lifetime of a system, the incremental change in each release is approximately constant. Types of Maintenance Release P.338 Perfective The system can be made better without changing its functionality e.g. improving performance speed, memory usage, etc. AdaptiveThe company may change the way it works e.g. moving from stand-alones to networks.CorrectiveFixing bugs which only come to light after release e.g. the millenium bug.HOW?mail-shot to all licensed usersdespatch of update disk/floppycomputer bulletin board with detail of patches, fixed, known errors, etc. EXAMINATION QUESTIONS Before releasing a new package the Software Company carries out alpha and beta testing (a) What are these two types of testing and why are they both needed? (6) (b) Explain why, once the package has been released there may be a need for maintenance releases and how might these be dealt with. (6) (a) Alpha testing: Testing in-house, with test data provided by software house, needed to test implementation against design specification; 3x1=3 Beta testing off-site software house, with `live data, by real users, needed to detect errors not identified at alpha stage; 3x1=3 (b)Need for maintenance: Perfective - improving performance speed, memory usage, etc. Corrective: fixing bugs, which only come to light after release. Dealt with by mail-shot to all licensed users, despatch of update disk/floppy, and computer bulletin board with detail of patches, fixed, known errors, etc. 1997.5 (6 marks) A software company is preparing to release a new application program. Describe the two types of testing carried out before the final release of the software. Explain why both are needed. Each type must include justification to warrant 3 marks. alpha testing (1) - testing in-house (1) with data provided by software house (1) needed to test implementation against design spec.(1) beta testing (1), off-site/real user testing (1), using live data (1), needed to detect errors not detected at alpha stage (1) Beta testing involves a wider audience (1) & different environments. (1) Candidates may take an alpha/beta/reasons for both approach - use a 2+2+2 model 1999.2 (5 marks) A software company has notified customers of a maintenance release for its accounting package. The notification states that a programme of alpha and beta testing will be carried out to ensure that the maintenance release is reliable. (a) State three reasons why a maintenance release might be necessary. (3) (b) What is meant by the terms: (i) alpha testing? (ii) beta testing? (2) (a) ADAPTIVE MAINTENANCE: EG To deal with y2k or EMU changes (1) To deal with external issues such as tax law, budgetary, tax rate, etc (1). To deal with hardware or software developments, new processors, new operating systems, etc (1) PERFECTIVE MAINTENANCE: EG To enhance functionality/ introduce new features of the package (1) To decrease processing time (1) To improve HCI (1) CORRECTIVE MAINTENANCE EG: To fix bugs/logic errors, coding errors, etc - NOT problems must be more specific (1) Etc Any 3x1=3(b) alpha testing carried out by software house (1) beta testing carried out by selection of software users (1) June 2002.1 Software houses go through a long testing programme before releasing a product. Despite this, problems can still occur with that product. Give three reasons why testing may not be completely successful. (3 marks) requirement to keep development cost to defined limits (1) requirement to keep development time to deadlines (1) in order to gain/maintain edge over competition - get product to market first (1) user has used product in a way that no-one has previously done (1) new hardware/ software is released the company was not aware of (1 ) inadequate test plan / data (1 ) etc. Spring 2003.7 A software company is producing a software package to perform an initial assessment of students entering colleges. a. There is a fixed deadline for the release of this package. Describe two effects that this might have on the final product. (4 marks) b. The production of this package is a complex task. For this reason the company has decided to allocate sub-tasks to separate development teams. Describe two benefits of this approach. (4 marks) c. Two weeks after the release of the package, several colleges report identical problems with the software. Describe what the company should do in this situation. (2 marks) a. Testing may not be fully carried out (1) so e.g. only the major functions of the software are checked/ only checked against a small set of hardware (1) Functionality may be left out (1) as it is deemed superfluous to requirements (1) Programming is not fully documented (1) so it is difficult to improve/ correct the software (1) 2 x (2,1,0) marks b. This should reduced the development time (1) as parts can be worked on simultaneously (1) Personnel with particular expertise can be given parts of the system (1) so that the parts of the system are as efficient as possible (1) By using a modular approach (1) the system should be easier to test/ modify/ maintain (1) 2 x (2,1,0) marks c. Produce a maintenance release/ software patch/ update (1) so that the software can operate with the operating system/ hardware that the colleges have (1) Provide a website/ document (1) so that technical staff have instructions on how to address the issue (1) 1 x (2,1,0) marks June 2003.1 Differentiate between alpha-testing and beta-testing. (4 marks) A2 Module 5 (ICT5) TOPIC 14.9 Software reliability  HYPERLINK "http://www.nchadderton.zen.co.uk/front.htm" http://www.nchadderton.zen.co.uk/front.htm Page  PAGE 1 of  NUMPAGES 9 LVjUg# 1 M A C ^ OP\]YZ ƻƮƻƻƮƻƻƮƻƻƒƻ0JCJOJQJ$jB*CJOJQJUph0JB*CJOJQJphB*CJOJQJphjB*CJOJQJUph5B*CJmH phsH  B*CJph5B*CJph6B*CJmH phsH 9KLTQ##$d%d&d'dNOPQ#$d%d&d'dNOPQ' & F$d%d&d'dNOPQJK d & 2 3 M B C ^ #$d%d&d'dNOPQ' & F $d%d&d'dNOPQU]s.`w^$If' & F!$d%d&d'dNOPQ#$d%d&d'dNOPQYm78b$$IfT  0634ab$If 56&!+r IZmrGHV a b !!F"a"?#N#O#e#w##<$E$Z$h$$$$$$$v%%%%&&&&&5B*CJph B*CJph0JCJOJQJjB*CJOJQJUph$jB*CJOJQJUphB*CJOJQJphGm!rJn & F"$If' & F $d%d&d'dNOPQHV X b !F"?#zzzuoo$If & F ' & F$d%d&d'dNOPQ[$$IflH J! t04 la ?#O#w#/$R$i$$$$$v%w%%w#$d%d&d'dNOPQ[$$Ifla    t04 la$If %&&&&&'4()**.+,--u----#$d%d&d'dNOPQ' & F $d%d&d'dNOPQ&&4(*,+,,,,--..1129222w33W4z45.555555<6E666667788;%;5;;.<;<<<~====>>^?廱B*CJOJQJph5B*CJOJQJph56B*CJph6B*CJph5B*CJmH phsH B*CJmH phsH >*B*CJph B*CJph5B*CJph5>*B*CJph6-....01P111122}- & F%$d%d&d'd1$7$8$H$NOPQ1$7$8$H$' & F$$d%d&d'dNOPQ#$d%d&d'dNOPQ 2w3W4555555H[$$Ifl4 J! t04 la$If' & F$d%d&d'dNOPQ- & F$d%d&d'd1$7$8$H$NOPQ55;6<6E666666666777877t`0l$$Ifl09  t04 la$If77777777@888f#$d%d&d'dNOPQl$$Ifl09  t04 la$If 8]999|:;%;6;;.<<===vX$$Ifl,""04 la & F&$If#$d%d&d'dNOPQ ===>?8?K?b?c????O@j@@@@@XAbA$If$If^^?b?c?g???O@j@@@AAAAqBvBB'D)D7DFtJvJJJJJJJJJJJJJ1Kؽyng CJmH sH jCJUmH sH  CJmH sH 6CJCJ5CJ5B*CJph B*CJphB*CJmH phsH 56B*CJmH phsH 6B*CJph6B*CJmH phsH 56B*CJmH phsH 6B*CJmH phsH B*CJmH phsH B*CJOJQJmH phsH #bAcAAAAAAAGEECX$$Ifl"9#09#4 la$IfX$$Ifl "9#09#4 laA2BlBBB(CzCCD#D(D)D7DDD2EEEcFFF' & F'$d%d&d'dNOPQFFBGGGH HeHHII[I_I' & F)$d%d&d'dNOPQ' & F($d%d&d'dNOPQ#$d%d&d'dNOPQ _IIcJuJvJJJJJKKKK$a$$a$ ##$d%d&d'dNOPQ' & F*$d%d&d'dNOPQ 1K2K3K]K^KaKfKgKmKnKoKpKtKuKKKKKKK55CJmHnHsH uj5CJUmH sH 5CJmH sH  CJmH sH 0JCJmH sH jCJUmH sH j CJUmH sH 0 / =!"#$%DyK )http://www.softwaretestinginstitute.com/yK Rhttp://www.softwaretestinginstitute.com/DyK http://www.softwareqatest.com/yK >http://www.softwareqatest.com/!DyK +http://www.nchadderton.zen.co.uk/front.htmyK Vhttp://www.nchadderton.zen.co.uk/front.htm i@@@ Normal CJOJQJ_HkH'mH sH tH V@V Heading 1($<$d%d&d'd@& 5;KH88 Heading 2$<@&5<@< Heading 3$@&5B*CJph<A@< Default Paragraph Font,, Header  9r , @, Footer  9r &)@& Page NumberdO"d Body1 ,p@ P !<#OJQJmH sH tH u(U@1( Hyperlink>*B*F^@BF Normal (Web)dd[$\$ OJPJQJD~KLTQ#d&23MB C ^    U ] s . ` w  ^Ym78m!rJnHVXbF?Ow/ R i v!w!!"""""###z${$$%-&u&&&&'''')*P****++w,W-.......;/>>c????B@@@A AeAAIB[B_BBcCuCvCCCCCDD       00! ! ! ! ! ! ! ! ! !   " " " " " " " " "     000 0 000 0 0 0 00000000$ 0$ 0$ 0$ 0%     & & & & (' ' ' ' ' ' ' (( ( ( ) ) ) * * 0@0@0 055&^?1KK'-29>?#%-2578=bAAF_IK(*+,./01345678:;<=K) O\Y 5DXTXTXTXX5oX! _Hlt44340838 _Hlt44340826 _Hlt44340827 _Hlt44340965 _Hlt44340966 _Hlt44340835 _Hlt44340975 _Hlt44340976SVVD@@@@@@@@TWWDCCCCfDpDtDDDD'0R]q%r%-%-%/2/////0 080@022.535550868<8A8P8T8d8e8g:l:::;;;;(<*<z<~<<<= =#=&=CCCDD3333333333333333333333333333TXQU{$$$CCCCCDDJ Regan.Shared IT Folder:A-Level:ICT5:14.9 RELIABILITYR Moorby<Macintosh HD:Temporary Items:AutoRecovery save of 14.9 RELIAR Moorby.Shared IT Folder:A-Level:ICT5:14.9 RELIABILITYR Moorby.Shared IT Folder:A-Level:ICT5:14.9 RELIABILITYR Moorby.Shared IT Folder:A-Level:ICT5:14.9 RELIABILITYR Moorby.Shared IT Folder:A-Level:ICT5:14.9 RELIABILITYNorth Chadderton2Macintosh HD:Current:A-Level:ICT5:14.9 RELIABILITYICT Departmentuntitled:ICT5:14.9 RELIABILITYICT Department8Server HD:Shared IT Folder:A-Level:ICT5:14.9 RELIABILITY D8H:\learnict05\as_a2\topics\software\14.9 RELIABILITY.dot  ;/~L ^+NJVGLeRH58`Do50]gA2%@UD"C$ ?ABCDEFGIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqstuvwxy{|}~Root Entry Fxs6Data @1TableHRWordDocument!~SummaryInformation(rDocumentSummaryInformation8zCompObjjObjectPoolxs6xs6  FMicrosoft Word Document MSWordDocWord.Document.89q