On Monotone Regression - School of Maths and Stats Local Site

Why do we want the monotonicity constraint?Underlying phenomenon can be monotone. For exampleGrowth curve data, for example tumour growthMonotonic data transformation of data, e.g. in a regression orANOVA situationForensic science — e.g. predicting age from maturity score (createdfrom teeth sizes)nonparametric calibrationdose-response curvesBerwin A Turlach (UWA) Monotone Regression 29 August 2011 3 / 32

Selective reviewMuch work done on constrained nonparametric smoothingsome using kernel smoothing methodsFriedman and Tibshirani (1984), Mammen (1991), Marron et al. (1997), Fisher et al. (1997), Mammenet al. (2001), Hall and Huang (2001), Dette et al. (2006), Dette and Pilz (2006)but mainly in the splines literature◮ using linear or quadratic programming approachesDierckx (1980), Wright and Wegman (1980), Micchelli et al. (1985), Irvine et al. (1986), Villalobosand Wahba (1987), Elfving and Andersson (1988), Ramsay (1988), Fritsch (1990), Schmidt andScholz (1990), Schwetlick and Kunert (1993), Tantiyaswasdikul and Woodroofe (1994), He and Shi(1998), Turlach (2005)◮◮using semi-indefinite programming approachesWang (2008), Wang and Li (2008)via the penalty termRamsay (1998), Heckman and Ramsay (2000)Berwin A Turlach (UWA) Monotone Regression 29 August 2011 4 / 32

Selective review (ctd)◮◮◮via boosting techniquesLeitenstorfer and Tutz (2007), Tutz and Leitenstorfer (2007)via Bayesian approachesRamgopal et al. (1993), Lavine and Mockus (1995), Perron and Mengersen (2001), Holmes andHeard (2003), Neelon and Dunson (2004), Brezger and Steiner (2008), Shively et al. (2009),Hazelton and Turlach (2011), Meyer et al. (2011)for some theory seeUtreras (1985), Mammen and Thomas-Agnan (1999), Meyer (2008)Berwin A Turlach (UWA) Monotone Regression 29 August 2011 5 / 32

The big questionTo smooth or not to smooth,that is the question:Whether ’tis nobler in the mind tolet the data speak for themselves,or to fit parametric models to them,and by fitting miss some features?Berwin A Turlach (UWA) Monotone Regression 29 August 2011 7 / 32

PolynomialsPolynomials are usually parameterised as follows:p(x) = β 0 + β 1 x + β 2 x 2 + . . . + β p x pThis is linear in its parametersParameters can be estimated using standard linear regressiontechniquesNothing difficult here!Berwin A Turlach (UWA) Monotone Regression 29 August 2011 8 / 32

Example: monotone polynomial fit to simulated datay−20 −10 0 10 20 30●●● ●● ● ●●●● ●●●● ● ● ● ●●●●●● ●● ● ● ● ●●● ● ● ● ●● ● ●●● ● ●●●● ●●●●●−1.0 −0.5 0.0 0.5 1.0xFigure 3: Monotone polynomial (degree 5) fitBerwin A Turlach (UWA) Monotone Regression 29 August 2011 10 / 32

Comparison: unconstrained polynomial and monotone polynomialy−20 −10 0 10 20 30●●● ●● ● ●●●● ●●●● ● ● ● ●●●●●● ●● ● ● ● ●●● ● ● ● ●● ● ●●● ● ●●●● ●●●●●−1.0 −0.5 0.0 0.5 1.0xFigure 4: Comparison - standard polynomial / monotone polynomialBerwin A Turlach (UWA) Monotone Regression 29 August 2011 11 / 32

The Problem and existing methodsRamsay (1998) wrote:How to fit a monotonepolynomial to data“Attempting to impose monotonicity on polynomials quicklybecomes unpleasant”Elphinstone (1983) provided formulations of monotone polynomialsHawkins (1994) fitted monotone polynomials to data by introducinghorizontal inflection points (HIPS)Berwin A Turlach (UWA) Monotone Regression 29 August 2011 12 / 32

Formulations of monotone polynomialsElphinstone provided this formulation:∫ xp(x) = d + a0K∏ {(c2i + bi 2 ) + 2b i t + t 2} dti=1where 2K + 1 is the degree of the polynomial, and b i , c i are arbitrary real values and d is the intercept and a is ascaling parameter that also determines whether our polynomial is monotonic increasing or decreasing.This has first derivativep ′ (x) = aK∏i=1{}(ci2 }{{} + b2 i ) + 2b i x + x 2} {{ }Both c 2 iand (b i + x) 2 ≥ 0, hence monotonicityBerwin A Turlach (UWA) Monotone Regression 29 August 2011 13 / 32

Problems with formulationElphinstone’s second formulation was used by Hawkins (1994) andHeinzmann (2008)∫ xp(x) = d + a0This has two minor problems:K∏ {1 + 2bi t + (bi 2 + ci 2 )t 2} dti=1We cannot express p(x) = x p in this mannerThe ci2 term has the potential to cause problems in optimizationroutinesBerwin A Turlach (UWA) Monotone Regression 29 August 2011 14 / 32

Alternative formulationPenttila (2006; personal communication):∫ xp(x) = d + a0K∏ {b2i + 2b i t + (1 + ci 2 )t 2} dti=1p(x) = x p can be expressed using this formulationUsing Penttila’s formulation our objective function becomesRSS =[ (n∑y j − d + aj=1∫ xj0i=1)] 2K∏ {b2i + 2b i t + (1 + ci 2 )t 2} dtBerwin A Turlach (UWA) Monotone Regression 29 August 2011 15 / 32

Objective function - cubic exampleFor the cubic we haveRSS =n∑ [yj − ( β 0 + β 1 x j + β 2 xj 2 + β 3 xj3 )] 2j=1When we impose the monotonicity constraint this becomes:RSS =[ (n∑y j − d + ab1x 2 j + ab 1 xj 2 + a(1 + c1 2 ) x )]j3 23j=1Not too difficult but is non-linear in parameters — needs non-linearoptimisation routinesBerwin A Turlach (UWA) Monotone Regression 29 August 2011 16 / 32

Objective function K > 1?For K = 2 this polynomial becomes more complexp(x) = d + a{b 2 1b 2 2x +b 1 b 2 (b 1 + b 2 )x 2 +13 [4b 1b 2 + b 2 2(1 + c 2 1 ) + b 2 1(1 + c 2 2 )]x 3 +12 (b 1 + b 2 + b 2 c 2 1 + b 1 c 2 2 )x 4 +15 (1 + c2 1 )(1 + c 2 2 )x 5 }and hence the non-linear optimisation becomes more complex andcomputationally intensive!Berwin A Turlach (UWA) Monotone Regression 29 August 2011 17 / 32

Minimization of the objective functionConsidered several different optimisation routines includingNon-derivative based algorithmsCoordinate descentBlock coordinate descentLevenberg–MarquardtAfter simulations concluded Levenberg–Marquardt was most suitableLooked also at a modification using constrained c i instead of c 2 iBerwin A Turlach (UWA) Monotone Regression 29 August 2011 18 / 32

yyyySome simulated and real dataConsidered 4 data sets−20 0 10 20 30●●●●●● ● ● ●●●●● ● ● ● ●● ●●●●● ● ●● ●●● ● ●●●●● ● ●●● ●●●●●●● ●● ●●80 100 140 180● ●●●●●●●●●●● ●●●●●●●●●●●●●●● ●●●●−1.0 −0.5 0.0 0.5 1.05 10 15xx0 2 4 6 8 10 14●●●●●●●●●●●●●●●●●●●●●●●●●−0.05 0.00 0.05 0.10●●● ● ● ●● ●●● ● ●●●●● ●● ●●●0 2 4 6 8 10 12−1.0 −0.5 0.0 0.5 1.0xxFigure 5: Example data setsBerwin A Turlach (UWA) Monotone Regression 29 August 2011 19 / 32

Simulation results — summaryTable 1: RSS for EHH formulationEHH Elphinstone Penttilac 2 c ≥ 0 c 2 c ≥ 0 c 2 c ≥ 0Data 1 (Hawkins) K=1 886.32757 323.69448K=2 63.60159 63.60159K=3 44.94751 41.99854K=4 41.90708 41.90566Data 2 (Ramsay) K=1 348.78796 348.78796K=2 351.11169 138.79971K=3 351.85594 40.68825K=4 352.22267 40.50511Data 3 (SimD0) K=1 0.002000 0.002000K=2 0.001469 0.001469K=3 0.001399 0.001421K=4 0.001414 0.001402Data 4 (SimD2) K=1 25.67706 25.67706K=2 25.40864 25.40864K=3 11.44768 11.24709K=4 13.91453 11.08235Many times the c 2 parameterisation converges to a local minimaBerwin A Turlach (UWA) Monotone Regression 29 August 2011 20 / 32

Simulation results — summary (ctd)Table 2: RSS focussing on Penttila formulationEHH Elphinstone Penttilac 2 c ≥ 0 c 2 c ≥ 0 c 2 c ≥ 0Data 1 (Hawkins) K=1 323.69448 323.69448 323.69448K=2 63.60159 63.60159 63.60159K=3 41.99854 41.99854 41.99854K=4 41.90566 41.90566 41.90566Data 2 (Ramsay) K=1 348.78796 348.78796 348.78796K=2 138.79971 138.79971 138.79971K=3 40.68825 40.68825 40.68825K=4 40.50511 40.5051 40.68825Data 3 (SimD0) K=1 0.002000 0.002000 0.002000K=2 0.001469 0.001469 0.001469K=3 0.001421 0.001399 0.001422K=4 0.001402 0.001401 0.001422Data 4 (SimD2) K=1 25.67706 25.67706 25.67706K=2 25.40864 25.40864 25.40864K=3 11.24709 11.24709 11.24709K=4 11.08235 11.08235 11.87866The Penttila formulation fails to converge in some instancesBerwin A Turlach (UWA) Monotone Regression 29 August 2011 21 / 32

Simulation results — summary (ctd)Table 3: RSS comparing Elphinstone’s 2 formulationsEHH Elphinstone Penttilac 2 c ≥ 0 c 2 c ≥ 0 c 2 c ≥ 0Data 1 (Hawkins) K=1 323.69448 323.69448K=2 63.60159 63.60159K=3 41.99854 41.99854K=4 41.90566 41.90566Data 2 (Ramsay) K=1 348.78796 348.78796K=2 138.79971 138.79971K=3 40.68825 40.68825K=4 40.50511 40.5051Data 3 (SimD0) K=1 0.002000 0.002000K=2 0.001469 0.001469K=3 0.001421 0.001399K=4 0.001402 0.001401Data 4 (SimD2) K=1 25.67706 25.67706K=2 25.40864 25.40864K=3 11.24709 11.24709K=4 11.08235 11.08235Elphinstone’s original formulation performs marginally betterBerwin A Turlach (UWA) Monotone Regression 29 August 2011 22 / 32

Back to the forensic science problem — cubicAge vs Maturity scoreAge (years)6 8 10 12 14RSS=277.5●● ● ●●● ●●●●●●●● ● ●●●●●●●●●● ●●●●● ●● ● ● ● ●● ● ● ● ●●● ● ●● ●●● ●● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ●●●●● ●● ● ● ●● ●●●● ●● ●● ●● ● ●●● ●● ● ● ● ●●● ●●●● ● ● ●● ●● ● ●● ● ●● ● ●● ●● ●● ●●● ● ●●● ●●● ● ●●● ● ●● ● ● ●● ● ●●● ● ●●● ●● ●●●−1.0 −0.5 0.0 0.5 1.0Maturity score (standardised)Figure 6: Forensic science data, K=1Berwin A Turlach (UWA) Monotone Regression 29 August 2011 23 / 32

Back to the forensic science problem — quinticAge vs Maturity scoreAge (years)6 8 10 12 14RSS=273.6●● ● ●●● ●●●●●●●● ● ●●●●●●●●●● ●●●●● ●● ● ● ● ●● ● ● ● ●●● ● ●● ●●● ●● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ●●●●● ●● ● ● ●● ●●●● ●● ●● ●● ● ●●● ●● ● ● ● ●●● ●●●● ● ● ●● ●● ● ●● ● ●● ● ●● ●● ●● ●●● ● ●●● ●●● ● ●●● ● ●● ● ● ●● ● ●●● ● ●●● ●● ●●●−1.0 −0.5 0.0 0.5 1.0Maturity score (standardised)Figure 7: Forensic science data, K=2Berwin A Turlach (UWA) Monotone Regression 29 August 2011 24 / 32

Back to the forensic science problem — septicAge vs Maturity scoreAge (years)6 8 10 12 14RSS=271.8●● ● ●●● ●●●●●●●● ● ●●●●●●●●●● ●●●●● ●● ● ● ● ●● ● ● ● ●●● ● ●● ●●● ●● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ●●●●● ●● ● ● ●● ●●●● ●● ●● ●● ● ●●● ●● ● ● ● ●●● ●●●● ● ● ●● ●● ● ●● ● ●● ● ●● ●● ●● ●●● ● ●●● ●●● ● ●●● ● ●● ● ● ●● ● ●●● ● ●●● ●● ●●●−1.0 −0.5 0.0 0.5 1.0Maturity score (standardised)Figure 8: Forensic science data, K=3Berwin A Turlach (UWA) Monotone Regression 29 August 2011 25 / 32

Back to the forensic science problemAge vs Maturity scoreAge (years)6 8 10 12 14● ● ●●●●●●●●●●●● ● ●●●●●●●●●● ●●●●● ●● ● ● ● ●● ● ● ● ●●● ● ●● ●●● ●● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ●●●●● ●● ● ● ●● ●●●● ●● ●● ●● ● ●●● ●● ● ● ● ●●● ●●●● ● ● ●● ●● ● ●● ● ●● ● ●● ●● ●● ●●● ● ●●● ●●● ● ●●● ● ●● ● ● ●● ● ●●● ● ●●● ●● ●●●−1.0 −0.5 0.0 0.5 1.0Maturity score (standardised)Figure 9: Forensic science data, K=1:3Berwin A Turlach (UWA) Monotone Regression 29 August 2011 26 / 32

ReferencesBrezger, A. and Steiner, W.J. (2008). Monotonic regression based on BayesianP-splines: An application to estimating price response functions from store-level scannerdata, Journal of Business & Economic Statistics, 26(1):90–104Dette, H., Neumeyer, N., and Pilz, K.F. (2006). A simple nonparametric estimator of astrictly monotone regression function, Bernoulli, 12(3):469–490Dette, H. and Pilz, K.F. (2006). A comparative study of monotone nonparametric kernelestimates, Journal of Statistical Computation and Simulation, 76(1):41–56Dierckx, P. (1980). An algorithm for cubic spline fitting with convexity constraints,Computing, 24:349–371Elfving, T. and Andersson, L.E. (1988). An algorithm for computing constrainedsmoothing spline functions, Numerische Mathematik, 52:583– 595Elphinstone, C. (1983). A target distribution model for non-parametric densityestimation, Communications in Statistics - Theory and Methods, 12(2):161–198Fisher, N.I., Hall, P., Turlach, B.A., and Watson, G.S. (1997). On the estimation of aconvex set from noisy data on its support function, Journal of the American StatisticalAssociation, 92(437):84–91, doi:10.2307/2291452Berwin A Turlach (UWA) Monotone Regression 29 August 2011 27 / 32

References (ctd)Friedman, J.H. and Tibshirani, R. (1984). The monotone smoothing of scatterplots,Technometrics, 26(3):243–250Fritsch, F.N. (1990). Monotone piecewise cubic data fitting, in J.C. Mason and M.G.Cox, editors, Algorithms for Approximation II, Chapman and Hall, London, pages 99–106Hall, P. and Huang, L.S. (2001). Nonparametric kernel regression subject tomonotonicity constraints, The Annals of Statistics, 29(3):624–647Hawkins, D.M. (1994). Fitting monotonic polynomials to data, ComputationalStatistics, 9(3):233–247Hazelton, M.L. and Turlach, B.A. (2011). Semiparametric regression with shapeconstrained penalized splines, Computational Statistics & Data Analysis,55(10):2871–2879, doi:10.1016/j.csda.2011.04.018He, X. and Shi, P. (1998). Monotone B-spline smoothing, Journal of the AmericanStatistical Association, 93(442):643–650Heckman, N. and Ramsay, J.O. (2000). Penalized regression with model-based penalties,Canadian Journal of Statistics, 28:241–258Berwin A Turlach (UWA) Monotone Regression 29 August 2011 28 / 32

References (ctd)Heinzmann, D. (2008). A filtered polynomial approach to density estimation,Computational Statistics, 23(3):343–360, doi:10.1007/s00180-007-0070-zHolmes, C.C. and Heard, N.A. (2003). Generalized monotonic regression using randomchange points, Statistics in Medicine, 22:623–638Irvine, L.D., Marin, S.P., and Smith, P.W. (1986). Constrained interpolation andsmoothing, Constructive Approximation, 2(2):129–151Lavine, M. and Mockus, M. (1995). A nonparametric Bayes method for isotonicregression, Journal of Statistical Planning and Inference, 46:235–248Leitenstorfer, F. and Tutz, G. (2007). Generalized monotonic regression based onB-splines with an application to air pollution data, Biostatistics, 8(3):654–673Mammen, E. (1991). Estimating a smooth monotone regression function, The Annals ofStatistics, 19(2):724–740Mammen, E., Marron, J.S., Turlach, B.A., and Wand, M.P. (2001). A general projectionframework for constrained smoothing, Statistical Science, 16(3):232–248,doi:10.1214/ss/1009213727Mammen, E. and Thomas-Agnan, C. (1999). Smoothing splines and shape restrictions,Scandinavian Journal of Statististics, 26:239–252Berwin A Turlach (UWA) Monotone Regression 29 August 2011 29 / 32

References (ctd)Marron, J.S., Turlach, B.A., and Wand, M.P. (1997). Local polynomial smoothing underqualitative constraints, in L. Billard and N.I. Fisher, editors, Graph-Image-Vision,volume 28 of Computing Science and Statistics, Interface Foundation of North America,Inc., Fairfax Station, VA 22039–7460, pages 647–652Meyer, M.C. (2008). Inference using shape-restricted regression splines, The Annals ofApplied Statistics, 2(3):1013–1033Meyer, M.C., Hackstadt, A.J., and Hoeting, J.A. (2011). Bayesian estimation andinference for generalised partial linear models using shape-restricted splines, Journal ofNonparametric Statistics, to appearMicchelli, C.A., Smith, P.W., Swetitis, J., and Ward, J.D. (1985). Constrained L papproximation, Constructive Approximation, 1(1):93–102Neelon, B. and Dunson, D.B. (2004). Bayesian isotonic regression and trend analysis,Biometrics, 60:398–406Perron, F. and Mengersen, K. (2001). Bayesian nonparametric modeling using mixturesof triangular distributions, Biometrics, 57:518–528Berwin A Turlach (UWA) Monotone Regression 29 August 2011 30 / 32

References (ctd)Ramgopal, P., Laud, P., and Smith, A. (1993). Nonparametric Bayesian bioassay withprior constraints on the shape of the potency curve, Biometrika, 80:489–498Ramsay, J.O. (1988). Monotone regression splines in action (with discussion), StatisticalScience, 3(4):425–461Ramsay, J.O. (1998). Estimating smooth monotone functions, Journal of the RoyalStatistical Society, Series B, 60(2):365–375Schmidt, J.W. and Scholz, I. (1990). A dual algorithm for convex-concave datasmoothing by cubic C 2 -splines, Numerische Mathematik, 57:333–350Schwetlick, H. and Kunert, V. (1993). Spline smoothing under constraints onderivatives, BIT, 33:512–528Shively, T.S., Sager, T.W., and Walker, S.G. (2009). A Bayesian approach tonon-parametric monotone function estimation, Journal of the Royal Statistical Society,Series B, 71(1):159–175Tantiyaswasdikul, C. and Woodroofe, M.B. (1994). Isotonic smoothing splines undersequential designs, Journal of Statistical Planning and Inference, 38:75–88Turlach, B.A. (2005). Shape constrained smoothing using smoothing splines,Computational Statistics, 20(1):81–104, doi:10.1007/BF02736124Berwin A Turlach (UWA) Monotone Regression 29 August 2011 31 / 32

References (ctd)Tutz, G. and Leitenstorfer, F. (2007). Generalized smooth monotonic regression inadditive modeling, Journal of Computational and Graphical Statistics, 16(1):165–188Utreras, F.I. (1985). Smoothing noisy data under monotonicity constraints: Existence,characterization and convergence rates, Numerische Mathematik, 47:611–625Villalobos, M. and Wahba, G. (1987). Inequality-constrained multivariate smoothingsplines with application to the estimation of posterior probabilities, Journal of theAmerican Statistical Association, 82(397):239–248Wang, X. (2008). Bayesian free–knot monotone cubic spline regression, Journal ofComputational and Graphical Statistics, 17(2):373–387Wang, X. and Li, F. (2008). Isotonic smoothing spline regression, Journal ofComputational and Graphical Statistics, 17(1):21–37Wright, I. and Wegman, E. (1980). Isotonic, convex and related splines, The Annals ofStatistics, 8:1023–1035Berwin A Turlach (UWA) Monotone Regression 29 August 2011 32 / 32