Internet Surveys vs Traditional
Paper-and-Pencil Surveys |
'Evidence of the
Afterlife' Supplemental Material Home Page
It is appropriate to consider the reliability of the NDERF
Internet-based survey methodology in comparison to the more traditional
paper-and-pencil surveys. To study this, I performed a search of the National
Library of Medicine searching for studies, preferably randomized studies, which
compared the two survey methods. Below are the links to 18 studies, and all or
part of the accompanying study abstract. The consensus of these studies is that
responses to Internet surveys are as reliable as responses to paper-and-pencil
surveys. This is not a comprehensive review of the literature on this subject,
but to the best of my knowledge, there is general acceptance in the scholarly
literature that responses to Internet surveys, as is used on the NDERF website,
are as reliable as responses to paper-and-pencil surveys.
This helps validate the findings of the survey on the NDERF website
as reliable. Other methods that I used to determine the reliability of the
responses to the NDERF survey are described in Evidence of the Afterlife: The
Science of Near-Death Studies. From the preceding, I am confident of the
reliability of the survey responses to the NDERF survey. There are additional
comments about the NDERF study methodology in other sections in this part of the
website.
Reliability of short form-36 in an Internet- and a pen-and-paper version.
Basnov M,
Kongsved SM, Bech P, Hjollund NH.
Inform Health Soc Care. 2009 Jan;34(1):53-8.
We found little or no evidence of a difference in
test-retest reliability and internal consistency when we compared an Internet-
and a pen-and-paper version of SF-36.
Gordon JS, McNew R.
Nurs Clin North Am. 2008 Dec;43(4):605-19, vii. Review.
Research shows that the type and quality of responses
one receives with online surveys are comparable with what one receives in
paper-based surveys.
Steenhuis MP, Serra M, Minderaa RB, Hartman CA.
Psychol Assess. 2009 Jun;21(2):231-4.
Correspondence between the Internet and interview versions at the
level of symptom scores was excellent, and correspondence with respect to the
presence/absence of ADHD was good. Although highly comparable diagnostic
outcomes between self-administration through the Internet and interviewer
administration were found, further study using other DISC-IV modules is
required.
Richter JG, Becker A, Koch T, Nixdorf M, Willers R, Monser R,
Schacher B, Alten R, Specker C, Schneider M.
Ann Rheum Dis. 2008 Dec;67(12):1739-41. Epub 2008 Jul 22.
RESULTS: Scores obtained by direct data entry on the Tablet PC did not differ
from the scores obtained by the paper-pencil questionnaires in the complete
group and disease subgroups. No major difficulties using the Tablet PC occurred.
62.1% preferred remote data entry in the future. Seven (4.6%) patients felt
uncomfortable with the Tablet PC due to their rheumatic disease. CONCLUSIONS:
Self-administered questionnaires via Tablet PC are a facile and capable option
in patients with rheumatic diseases to monitor disease activity, efficacy and
safety assessments continuously. Tablet PC applications offers directly
available data for clinical decision-making improves quality of care by
effective patient monitoring, and contributes to patients' empowerment.
Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH.
J Med Internet Res. 2007 Sep 30;9(3):e25.
CONCLUSIONS: The Internet version of the questionnaire was superior
with respect to completeness of data, but the response rate in this population
of unselected patients was low. The general population has yet to become more
familiar with the Internet before an online survey can be the first choice of
researchers, although it is worthwhile considering within selected populations
of patients as it saves resources and provides more complete answers. An
Internet version may be combined with the traditional version of a
questionnaire, and in follow-up studies of patients it may be more feasible to
offer Internet versions.
Internet administration of the Edinburgh
Depression Scale.
Spek V, Nykl'cek I, Cuijpers P, Pop V.
J Affect Disord. 2008 Mar;106(3):301-5. Epub 2007 Aug 8.
CONCLUSION: The psychometric properties of the internet-administered EDS are
comparable to those of the paper and pencil EDS.
Vallejo MA, Ma'anes G, Isabel Comeche MA, D'az MI.
J Behav Ther Exp Psychiatry. 2008 Sep;39(3):201-8. Epub 2007 May 17.
The objective of this study is to compare the results of the Symptom Check List
(SCL)-90-R and the General Health Questionnaire (GHQ)-28 administered on paper
with the results of administration via Internet. Forty university students were
asked to respond to the two questionnaires, first in one format (Internet or
paper), and 1 week later in the other format. For the GHQ-28, no statistically
significant differences were found between the two forms of presentation,
Internet or paper, except in the social dysfunction subscale. In the case of the
SCL-90-R, there were differences in the Positive Symptoms Distress Index (PSDI)
and in the somatization and obsession-compulsion subscales. The effect of using
the two formats in combination was also analyzed. Internal consistency of the
instruments is high for both types of presentation (alpha=0.91 in GHQ-28;
alpha=0.97 in SCL-90-R). Correlation between the two formats is also high, with
values ranging between 0.76 and 0.92, p<0.001, except in the social dysfunction
subscale. Finally, correlation between the GHQ-28 and SCL-90-R is also high, for
both the paper and Internet results (r=0.88, p<0.01 on paper and r=0.83, p<0.01
via Internet). The differences found should be verified in future research with
other types of sample.
Vallejo MA, Jord'n CM, D'az MI, Comeche MI, Ortega J.
J Med Internet Res. 2007 Jan 31;9(1):e2.
BACKGROUND: Internet psychology services are rapidly increasing and that implies online assessment. To guarantee the results of these new online evaluation procedures, it is necessary to have reliable and valid assessment tools. OBJECTIVE: In this work we analyzed the online versions of two popular psychopathology screening questionnaires: the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R). METHODS: A total of 185 psychology students were recruited from two universities in Madrid, Spain. All of them had Internet access at home. A test-retest situation and factorial analysis were used to generate reliability and validity data. Both paper-and-pencil questionnaires (test) and their online versions (retest) were completed by 100 participants (median gap = 17 days). RESULTS: Results suggest that both online questionnaires were fairly equivalent to their paper-and-pencil versions, with higher reliability values for the SCL-90-R. Factorial analysis tended to reproduce the structure shown in former investigations of both questionnaires, replicating the four-factor structure of the GHQ-28 but failing to do so with the nine-factor structure of the SCL-90-R. Instead, a large unrotated factor appeared. CONCLUSIONS: Further research should be carried out to confirm these data, but our work supports the online use of both assessment tools. The psychometric properties of the online version of GHQ-28 is similar to the paper-and-pencil and we can recommend its utilization in a Web environment. In contrast, SCL-90-R can only be recommended as a global index for psychological distress, using the Global Severity Index (GSI), not necessarily its subscales; and it should be considered that the online scores were lower than the ones with the paper-and-pencil version.
Coles ME, Cook LM, Blake TR.
Behav Res Ther. 2007 Sep;45(9):2232-40. Epub 2007 Jan 12.
Study results support the equivalence of these measures of obsessive compulsive disorder (OCD) symptoms and beliefs independent of administration method (paper versus secure project website). These findings create new opportunities for conducting OCD-related research online.
AbuAlRub RF.
J Nurs Scholarsh. 2006;38(2):200-4.
RESULTS: The American and Jordanian samples differed significantly on all demographic variables. However, they had similar patterns of responses in regard to the association between job stress and social support from coworkers. CONCLUSIONS: The predicted comparability of findings with different methods and samples lends credence to the reliability and validity of Internet-based research.
Mangunkusumo RT, Moorman PW, Van Den Berg-de Ruiter AE, Van Der Lei
J, De Koning HJ, Raat H.
J Adolesc Health. 2005 Jan;36(1):70.e1-6.
CONCLUSIONS: Health questionnaires via Internet were positively evaluated and
generally resulted in equal scores of health status/health behavior compared
with the P&P mode. We recommend further research with other questionnaires, and
in other settings as well with regard to score equivalence between web-based and
P&P-administered questionnaires.
Internet versus mailed questionnaires: a randomized comparison.
Ritter P, Lorig K, Laurent D, Matthews K.
J Med Internet Res. 2004 Sep 15;6(3):e29.
CONCLUSIONS: Among a convenience sample recruited via the Internet, results from those randomly assigned to Internet participation were at least as good as, if not better than, among those assigned mailed questionnaires, with less recruitment effort required. The instruments administered via the Internet appear to be reliable, and to be answered similarly to the way they are answered when they are administered via traditional mailed paper questionnaires.
Gosling SD, Vazire S, Srivastava S, John OP.
Am Psychol. 2004 Feb-Mar;59(2):93-104.
The rapid growth of the Internet provides a wealth of new research opportunities for psychologists. Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods. Six preconceptions about Internet samples and data quality are evaluated by comparing a new large Internet sample (N = 361,703) with a set of 510 published traditional samples. Internet samples are shown to be relatively diverse with respect to gender, socioeconomic status, geographic region, and age. Moreover, Internet findings generalize across presentation formats, are not adversely affected by nonserious or repeat responders, and are consistent with findings from traditional methods. It is concluded that Internet methods can contribute to many areas of psychology.
Cronk BC, West JL.
Behav Res Methods Instrum Comput. 2002 May;34(2):177-80.
Students, faculty, and researchers have become increasingly comfortable with the Internet, and many of them are interested in using the Web to collect data. Few published studies have investigated the differences between Web-based data and data collected with more traditional methods. In order to investigate these potential differences, two important factors were crossed in this study: whether the data were collected on line or not and whether the data were collected in a group setting at a fixed time or individually at a time of the respondent's choosing. The Visions of Morality scale (Shelton & McAdams, 1990) was used, and the participants were assigned to one of four conditions: in-class Web survey, in-class paper-and-pencil survey; take-home Web survey, and take-home paper-and-pencil survey. No significant differences in scores were found for any condition; however, response rates were affected by the type of survey administered, with the take-home Web-based instrument having the lowest response rate. Therefore, researchers need to be aware that different modes of administration may affect subject attrition and may, therefore, confound investigations of other independent variables.
Fouladi RT, McCarthy CJ, Moller NP.
Assessment. 2002 Jun;9(2):204-15.
The viability of using the World Wide Web to collect data from three widely used instruments by clinicians and researchers was investigated. The instruments were the Inventory of Parental and Peer Attachment, the Negative Mood Regulation Scale, and the Trait Meta-Mood Scale. Data were collected from two comparable groups of college students, and differences in response patterns on paper-and-pencil and World Wide Web versions of the measures, at both the item level and scale score level, were documented. Although mode of administration effects were statistically significant, the magnitude of the effects was in general very small. The basic similarity of the properties of the measures using paper-and-pencil and online Internet modes of administration suggests the viability of the Internet for assessing these and other psychological phenomena.
A comparison of World-Wide Web and paper-and-pencil personality questionnaires.
Pettit FA.
Behav Res Methods Instrum Comput. 2002 Feb;34(1):50-4.
Does the manifestation of response set effects in World-Wide Web (WWW) questionnaire responses differ from that in paper-and-pencil (PP) questionnaire responses? Two thousand six hundred and forty-nine volunteers responded to a WWW questionnaire, and 458 volunteers responded to a PP questionnaire consisting of five personality scales. Five response sets were examined. For four of them--random response, item nonresponse, extreme response, and acquiescent response--no statistically significant differences between PP and WWW data were found. The PP administration elicited a statistically higher number of errors (uncodable responses), although eta 2 was less than .02. An analysis of interitem and interscale correlations did not differentiate WWW and PP data. Data from the Marlowe-Crowne Social Desirability Scale, the Perfectionist Self-Presentation Scale, and the Computer Anxiety Scale did not differentiate WWW and PP data in terms of either means or scale reliabilities. It was concluded that WWW data may be comparable to PP data and that the WWW is a potentially useful and valid data collection tool.
Horswill MS, Coster ME.
Behav Res Methods Instrum Comput. 2001 Feb;33(1):46-58.
The Internet has been exploited successfully in the past as a medium for behavioral research. This paper presents a series of studies designed to assess Internet-based measures of drivers' risk-taking behavior. First, we compared responses from an Internet sample with a traditional pencil-and-paper sample using established questionnaire measures of risk taking. No significant differences were found. Second, we assessed the validity of new Internet-based instruments, involving photographs and photographic animations, that measured speed, gap acceptance, and passing. Responses were found to reflect known demographic patterns of actual behavior to some degree. Also, a roadside survey of speeds was carried out at the locations depicted in the photographic measure of speeding and, with certain exceptions, differences between the two appeared to be constant. Third, a between-subject experimental manipulation involving the photographic animation measure of gap acceptance was used to demonstrate one application of these techniques.
Web-based administration of a personality questionnaire: comparison with traditional methods.
Davis RN.
Behav Res Methods Instrum Comput. 1999 Nov;31(4):572-7.
The World-Wide Web holds great promise as a mechanism for questionnaire-based research. But are data from Web-based questionnaires comparable to data from standard paper-and-pencil questionnaires? This study assessed the equivalence of the Ruminative Responses Scale in a Web-based format and in a paper-and-pencil format among introductory psychology, upper-level psychology, and non-psychology students. Internal consistency coefficients were comparable across the groups. The participants in the Web sample reported higher levels of self-focused rumination than did the other groups. Women in the Web sample reported more self-focused rumination than did women in the other groups. In the Web sample, results did not covary with access location. These results suggest that findings from Web-based questionnaire research are comparable with results obtained using standard procedures. The computerized Web interface may also facilitate self-disclosure among research participants.