Assessing the Reliability and Validity of Expert Interviews

Abstract

Testing the reliability of experts should be a key element of expert interviews. Using the Condorcet Jury Theorem, it is shown that expert reliability can provide an indication of the validity of expert-opinion data. The theoretical framework is applied to expert-interview data collected in the Domestic Structures and European Integration (DOSEI) project. Special attention is paid to the role of ‘leading’ experts and salient issues. Evaluating the DOSEI data, the main findings are that (i) with some exceptions, there are acceptable levels of inter-expert agreement, (ii) whether the leading expert is included or not does not make a large difference to expert agreement, and (iii) experts are more in agreement on salient issues.