Funding Medical Research Projects: Taking into Account Referees’ Severity and Consistency through Many-Faceted Rasch Modeling of Projects’ Scores.Tesio L, Simone A, Grzeda MT, Ponzio M, Dati G, Zaratin P, Perucca L, Battaglia MA.J Appl Meas. 2015;16(2):129-152.
The funding policy of research projects often relies on scores assigned by a panel of experts (referees). The non-linear nature of raw scores and the severity and inconsistency of individual raters may generate unfair numeric project rankings. Rasch measurement (many-facets version, MFRM) provides a valid alternative to scoring. MFRM was applied to the scores achieved by 75 research projects on multiple sclerosis sent in response to a previous annual call by FISM-Italian Foundation for Multiple Sclerosis. This allowed to simulate, a posteriori, the impact of MFRM on the funding scenario. The applications were each scored by 2 to 4 independent referees (total = 131) on a 10-item, 0-3 rating scale called FISM-ProQual-P. The rotation plan assured “connection” of all pairs of projects through at least 1 shared referee. The questionnaire fulfilled satisfactorily the stringent criteria of Rasch measurement for psychometric quality (unidimensionality, reliability and data-model fit). Arbitrarily, 2 acceptability thresholds were set at a raw score of 21/30 and at the equivalent Rasch measure of 61.5/100, respectively. When the cut-off was switched from score to measure 8 out of 18 acceptable projects had to be rejected, while 15 rejected projects became eligible for funding. Some referees, of various severity, were grossly inconsistent. The FISM-ProQual-P questionnaire seems a valid and reliable scale. MFRM may help the decision-making process for allocating funds to MS research projects but also in other fields. In repeated assessment exercises it can help the selection of reliable referees. Their severity can be steadily calibrated, thus obviating the need to connect them with other referees assessing the same projects.
We have posted this video before. If you speak German or are German (apologies) you know it is all a pile of pants, but you get the feeling.
Peer review is one of the more infuriating things in science and when you get a rejection that you feel it is based on a poxy review..Science life I am afraid
There are always alternative views to any scenario and it is true when you get peer review (click here-excuse the language. Nothing to do with science, but to all Yorkists LOL, Lancs &Mancs avoid:-)
Some referees are constructive even in the dismantling of your grant and this useful, some talk out of their bottom:-). This study suggest analysis to assess if the the referees are consistent.