The reliability of content analysis of computer conference communication

The focus of this article is the reliability of content analysis of students' computer conference communication. Content analysis is often used when researching the relationship between learning and the use of information and communications technology in educational settings. A number of studies where content analysis is used and classification systems are developed are presented and discussed along with the author's own development and use of a classification system. However, the question of the reliability of content analysis is not often addressed or discussed in the literature. On examining the reliability of classifications in an empirical study of study groups' academic discussions in computer conferences in a distance education course, the present author found the reliability to be extraordinarily low. For some classifications the deviation was as high as 13% when the same person (coder) classified the same computer conference message at two different times. When two different coders classified the same computer conference messages, the deviation was as high as 27%. This low reliability-and the lack of discussion of this crucial matter in the literature-has profound implications. Not just for the author's own research but for all studies and results based upon content analysis of computer conference communication. Therefore, this issue needs to be addressed. A possible solution-where each computer conference message can be classified as having both one and/or other kinds of information-is proposed. This might not be a solution to the problem of low reliability of content analysis and the use of classification systems, but it does shed light on the problem and goes some way towards reducing it. © 2005 Elsevier Ltd. All rights reserved.