Training and Interobserver Reliability in Qualitative Scoring of Skeletal Samples

Abstract

This article reports on inter- and intra-observer reliability testing of the Coimbra method for fibrocartilaginous entheseal changes and discusses the results in the context of reliability for other qualitative methods that score skeletal variation. Four tests of interobserver and one test of intraobserver reliability were conducted. Three of the interobserver tests were conducted in person (Geneva, Coimbra A, Coimbra B) and one was conducted on-line. Before each test, the observers trained together and refinements were made to the method. Inter- and intra-observer reliability was assessed using percentage agreement, kappa scores, and Krippendorff’s alpha. On-line testing had the lowest reliability (60% agreement, κ = 0.292, α = 0.369). Final reliability results in the Coimbra B test increased from the initial Geneva test (68.6 to 80.0% agreement, κ = 0.522 to 0.589, α = 0.604 to 0.666). Factors such as training, observation conditions, visual acuity, and differences in interpretation of scoring criteria impact reliability. Suggestions are made to address the general problem of reliability in qualitative methods.