Developing and validating multiple choice test items 3rd ed
Some known and common problems are discussed herein and solutions are addressed.
Placements for all feedlots in Iowa totaled 100,000 head, down 7 percent from last month but up 8 percent from last year.
Recently, a colleague of mine asked for some advice on how to compute interrater reliability for a coding task, and I discovered that there aren’t many resources online written in an easy-to-understand format – most either 1) go in depth about formulas and computation or 2) go in depth about SPSS without giving many specific reasons for why you’d make several important decisions.
The primary resource available is a 1979 paper by Shrout and Fleiss, which is quite dense.
So I am taking a stab at providing a comprehensive but easier-to-understand resource.
Reliability, generally, is the proportion of “real” information about a construct of interest captured by your measurement of it.