Analysis of Changes Depend on Use Cases

A PREDICTION MODEL FORREQUIREMENTS EVOLUTION IN THENEXT SOFTWARE RELEASELin ShiInstitute <strong>of</strong> S<strong>of</strong>tware Chinese Academy <strong>of</strong> Sciences

Motivation2 Requirements creep or change may cause calamity Cost, Schedule, Quality If given the requirements that are instable at thebeginning <strong>of</strong> s<strong>of</strong>tware development, engineers cando: Figure out the root-causes <strong>of</strong> instable requirements Improve the requirements processes Raise the flexibility <strong>of</strong> s<strong>of</strong>tware designs Can we predict requirements to be changed basedon historical requirements evolution data?

Intuitions3 Requirements which are frequently changed inhistorical versions are also likely to change in thesubsequent versions— Frequency The probability <strong>of</strong> a requirement change may relateto the gaps between versions where changeshappened and the predicting version— Center <strong>of</strong> Change Gravity The probability <strong>of</strong> a requirement change may relateto the gap between the version which a requirement isinitialized and the predicting version — Lifecycle

Variable Definitions4 Frequency Times that a requirement has been changed Center <strong>of</strong> Change Gravity Sum <strong>of</strong> gaps between the version that a changehappened and the predicting version / Frequency Lifecycle Gaps between the version which a requirement isinitialized and the predicting version

Example5Requirements12345Change History√ √ XX √ √X X X√ X √NA X XV0 V1 V2 V3Predicting Version = V4, Frequency ( 1 ) = 1+1+0 = 2 Gravity( 1 ) = ((V4-V1) + (V4-V2)) / 2 Lifecycle ( ) = V4-V25

Research Question6FrequencyGravityLifecycleAnyrelationship?Requirementsevolution in thenext release

Prediction Approach7 Logistic Regression (LR) is a predictionapproach that can be used when the targetvariable is a categorical variable with twocategories. LR aims to determine whether there is someform <strong>of</strong> functional dependency between theexplanatory variables and the dependentvariable.

8Logistic Regression Equation

Overview <strong>of</strong> Approach9V1√XX√V4V2V3√√XXXX√X√XXLearnerPredictorXX√√XX

Case Study10 To investigate, we used 322 changed requirementsfrom 8 versions <strong>of</strong> QONE. Unit <strong>of</strong> requirements Features, use cases, function points, user stories…Version V0 V1 V2 V3 V4 V5 V6 V7# <strong>of</strong> use case 186 189 195 221 276 310 303 339# changed use cases(add+del+mod)N/A 14 43 109 121 143 38 63

Data Collection11 added = 2, modified = 1, deleted = -1,unchanged = 0Use Case ID V1 V2 V3 V4 V5 V6 V71 0 0 2 1 0 -1 0… … … … … … … …322 0 2 0 0 1 1 0

Data12 Training data Sum <strong>of</strong> four subsets where PredictV = V3,V4,V5,V6 Testing data One subset where PredictV= V7 Input Use Cases’ change records from V1 to V7 Output Use Case IDs predicted to change in V8

Data13 To investigate the performance evolution with theincreasing <strong>of</strong> sample size, we trained another threeprediction models#Model Training Data Testing DataM1 subsets where PredictV = V3,V4,V5,V6 PredictV = V7M2 subsets where PredictV = V3,V4,V5 PredictV = V6M3 subsets where PredictV = V3,V4 PredictV = V5M4 subsets where PredictV = V3 PredictV = V4

Prediction Model14# Model M4 M3 M2 M1Sample size (use cases) 221 497 807 1110w 1w 2w 0Likelihoodvalue 4.065 3.683 3.811 4.769p-value 0.011 0 0 0value -0.489 -0.247 -0.964 -1.138p-value 0.904 0.734 0.138 0.013value -1.867 -1.715 -1.965 -2.144p-value 0 0 0 0p-value 0.017 0 0 0Ratio 8.131 21.161 23.784 43.994

Prediction Model15# Model M4 M3 M2 M1Sample size (use cases) 221 497 807 1110w 1w 2w 0Likelihoodvalue 4.065 3.683 3.811 4.769p-value 0.011 0 0 0value -0.489 -0.247 -0.964 -1.138p-value 0.904 0.734 0.138 0.013value -1.867 -1.715 -1.965 -2.144p-value 0 0 0 0p-value 0.017 0 0 0Ratio 8.131 21.161 23.784 43.994

Significance level1610.90.80.70.60.50.4w1w20.30.20.10M4 M3 M2 M1

Evaluation – Classification tables17ObservedPVM1 (size = 303) M2 (size = 310)PredictedPredictedPVPercentage ObservedPVNo Yes CorrectNo YesPercentageCorrectNo 253 22 92 No 253 31 89.08PVYes 15 13 46.43 Yes 23 3 11.54Overall 87.79 Overall 82.58ObservedPVM3 (size = 276) M4 (size = 221)PredictedPredictedPVPercentage ObservedPVNo Yes CorrectNo YesPercentageCorrectNo 108 111 49.32 No 108 70 60.67PVYes 20 37 64.91 Yes 20 23 53.49Overall 52.54 Overall 59.28

Evaluation – Classification tables18ObservedPVM1 (size = 303) M2 (size = 310)PredictedPredictedPVP PVPercentage Observed 0 :percentage correct <strong>of</strong> usecases predicted to be unchangedNo Yes CorrectNo YesPercentageCorrectNo 253 22 92 No 253 31 89.08PVYes 15 13 46.43 Yes 23 3 11.54Overall 87.79 Overall 82.58PM3 (size = 276) 1 :percentage correct <strong>of</strong> useM4 (size = 221)cases predicted to be changedPPredictedPredictedavg : the overall percentage correctPVPVObservedPercentage ObservedPercentageNo Yes CorrectNo Yes CorrectPVNo 108 111 49.32 No 108 70 60.67PVYes 20 37 64.91 Yes 20 23 53.49Overall 52.54 Overall 59.28Select 0.2 as the cut value

Percentage Correct19100.00%90.00%80.00%70.00%60.00%50.00%40.00%30.00%20.00%10.00%0.00%P1P0Pavg• Less #modified usecases than others (26)• Nearly half <strong>of</strong>modified use casesM4 M3 M2areM1firstly changed.

Feedback from the Company20 Action In the design phases, the project managers made thearchitecture <strong>of</strong> those predicted use cases more flexible,e.g., using low-coupling criteria to design those usecases.‣ Consequence‣ The design <strong>of</strong> QONE is capable <strong>of</strong> future changes.

Feedback from the Company21 Action The product managers changed the requirementanalysis process. They first filtered those changerequests that can be mapped with the prediction results,and ranked them with high priority.‣ Consequence‣ The efforts <strong>of</strong> expert analysis on the prediction resultsdecreased.

Conclusion22 Predict requirements to be change in the earlyphase is possible. 46.43% correct <strong>of</strong> predicted modified use cases,and 92% correct <strong>of</strong> predicted unchanged use cases. Predictions can be further be used as inputs forimproving the requirements and design processes.

23Thank you!

24Questions ?