Pay for Quality
Pay for Quality
Pay for Quality
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
KCE Reports 118 <strong>Pay</strong> <strong>for</strong> <strong>Quality</strong> 9<br />
2.1.1.3 <strong>Quality</strong> appraisal<br />
<strong>Quality</strong> appraisal of systematic reviews was based on the criteria of the Dutch<br />
Cochrane Center (Form Va). 18 Validity rating by use of this Cochrane tool was based on<br />
7 items (namely research question, search strategy, relevance selection, quality appraisal<br />
included studies, data extraction, study description, heterogeneity and pooling). If more<br />
than three items were assessed as unclear or insufficient, a publication was excluded.<br />
However, <strong>for</strong> the majority of excluded reviews the reason was the narrative nature of<br />
the review or the lack of specification of methods used (Appendix 5).<br />
<strong>Quality</strong> appraisal of primary studies was per<strong>for</strong>med using a specifically constructed tool,<br />
based on a combination of existing tools 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 . The tool<br />
consists of ten generic items (namely research question, patient population and setting,<br />
intervention, comparison, outcome, design, sample size, statistics, generalisability,<br />
confounders addressed) and four design specific items (randomization, blinding,<br />
clustering effect, number of data points) (Appendix 6). For each primary study, a score<br />
has been given to each of the 14 items, ranging from -1 to +1. A score of +1 on a<br />
certain item corresponds with ‘a good result’, a score of -1 corresponds with ‘a bad<br />
result’, and a score of 0 corresponds with ‘this item has not been discussed in the<br />
article’. To pass the quality appraisal, an article has to gain a score of minimum 8 out of<br />
14. An article can be excluded on the basis of an ‘overrule’ argument when a certain<br />
criteria was not met, hence the overall validity has to be questioned (<strong>for</strong> example no<br />
significance testing, too small sample size etc…).<br />
<strong>Quality</strong> appraisal of the studies reporting cost-effectiveness and modelling effects of<br />
P4Q programmes, was also per<strong>for</strong>med by using two specifically constructed tools (See<br />
appendix 7). The quality appraisal tool <strong>for</strong> the modelling studies is based on the<br />
modelling guidelines from the KCE and on the ISPOR guidelines by Weinstein et al.<br />
It consists of the following items: model design (transparent and as simple as possible);<br />
assumptions and data input tested in extensive sensitivity analysis; original data set<br />
provided; sources used presented and described in detail; scenarios (<strong>for</strong> models that<br />
extrapolate to longer time periods); calibration (results should be logically consistent<br />
with real-life observations and data); face validity (the results of the model should be<br />
intuitively correct) and cross-validation (transparent enough to allow an explanation of<br />
the differences with other models <strong>for</strong> the same intervention). The tool used <strong>for</strong> the<br />
cost-effectiveness studies is based on the pharmaco-economic guidelines from the KCE<br />
31<br />
. It consists of the following items: literature review; perspective of the evaluation;<br />
target population; comparator; analytic technique; study design; calculation of costs;<br />
valuation of outcomes; data sources; incremental cost-effectiveness; time horizon;<br />
modelling; handling uncertainty; discount rate.<br />
For each item one of the following scores have been assigned: +, +/-, -. A ‘+’ score on<br />
given item corresponds with a good result, a ‘-‘ score corresponds with a bad result, a<br />
‘+/-‘ score corresponds with an in between result. Given the limited amount of<br />
modelling and cost-effectiveness studies, the quality appraisal is only per<strong>for</strong>med to<br />
in<strong>for</strong>m the reader about the quality of the studies, but was not used to exclude any<br />
study. Overall the quality of the modelling and cost-effectiveness studies was acceptable.<br />
Comparison between reviewers (DDS and PVH) identified seven non corresponding<br />
review publications out of 5480 potentially relevant reviews (Cohen’s Kappa 99.9%<br />
interrater reliability) and 18 non corresponding primary publications out of 5517<br />
potentially relevant primary publications (Cohen’s Kappa 99.7% interrater reliability).<br />
Appendix 8 and 9 list the citations that were included in full text analysis.<br />
As a result 103 primary P4Q evaluation articles, 5 modelling articles and 3 costeffectiveness<br />
articles, resulting from the primary publication search (phase 2) and from<br />
the review search (phase 1), are included in this review.<br />
31, 32