Print

Print
Yup. C IS a no brainer. As supported by a voluminous literature and standard survey texts, many authors would be AERA members.
It is important to consider the PSYCHOLOGY behind how people use these scales.
It is assumed that people, respondents, have an underlying continuous sale of satisfaction and they make responses according to criteria they attach to the verbal labels in the options.
Respondents like to see a full range of options even though they only use 1. Scales with a neutral are highly desirable, because respondents like to have that option. As many have noted asymmetry of options is biassing, if respondents divide POTENTIAL responses into equal category sizes their criterion for satisfied will be lower
When their chosen option is not available, they tend to respond at random thus adding ‘noise’ to the results.
For the same reason it is good practice always to give a non-response option, if there is more than 1 item in a survey [may not apply here]

then there is a question of the PURPOSE of the study and any proposed analysis.
is there an intent to compare with previous or future years? or between grad students and tenured profs, or between 1 day and full conference attenders, etc.?
if so what analyses are planned? hopefully NOT rank methods
mostly categories are dichotomised and logistic regression is used. this almost certainly gives a more realistic picture than only offering 2 options in initial survey
more information MAY be obtained using rank methods, e.g. logistic regression on cumulative proportions, but this is rarely done.
2 messages
Use C
Plan use and analysis of results
best
Diana
On 30 Apr 2015, at 06:28, Paul Barrett <[log in to unmask]> wrote:

Hello Burke
 
Scale C is a ‘no brainer’. Scale A is ludicrous, Scale B nearly so (you need that middle option to capture those who have no strong opinion one way or another).
 
In my opinion, using an unbalanced response scale is nothing more than seeking to a priori bias the responses by inhibiting plausible response options.
 
Regards .. Paul
 
Chief Research Scientist
__________________________________________________________________________________
M: +64-(0)21-415625
 
From: Educational Research Methodology List [mailto:[log in to unmask]] On Behalf Of Burke Johnson
Sent: Thursday, April 30, 2015 2:51 PM
To: [log in to unmask]
Subject: Re: PROBLEM with the 4-point scale used on the "AERA Requests Feedback about the 2015 Annual Meeting"
 
Please note that my objective is to determine the degree to which participants are satisfied or dissatisfied with the current object of evaluation (the conference). (If I only cared about satisfied vs. dissatisfied I could use a dichotomous scale, but that is not my interest.) Also, I do not know a priori what the responses will be because that is an empirical question; neither does the AERA know (for this year's conference) despite what might have occurred multiple years ago. (Check out David Hume's problem of induction.) Next, a survey or marketing or evaluation researcher would not recommend that a client use an unbalanced scale, except in exceptional circumstances. In my opinion, this is not one of those circumstances. The vast majority of books on measurement and marketing research exclude discussion of the practice altogether because it is controversial. An unbalanced scale can bring the one advantage mentioned by Paul, but it also brings with it the problem of not being defensible IF one wants to know the distribution of satisfaction and dissatisfaction and make claims about this to the public. That is what I want to know. I thought that was the purpose of the evaluation being discussed here. We cannot (we should not) just assume that most everyone is satisfied when evaluating a program or other object of interest and then systematically eliminate categories for responding (in my n=1 expert opinion). How would you feel if you were just a little dissatisfied (or not quite sure until you carefully considered the issue) or if you were extremely dissatisfied and you noticed that satisfied people were given three options but you were only given one option to place your opinion? Would you feel that the rating scale was fair, or would you question it? That is the person that is hurt by this scale. Is that fair for a conference focused on social justice? How would a third-party evaluator or auditor or even an interested consumer feel when shown the scale currently used, IF the purported purpose is to obtain empirical evidence of the degrees of satisfaction/dissatisfaction with the conference(and not just to differentiate responses in the scale region of satisfaction and censor the rest)? In short, I am arguing that the costs of using the unbalanced scale in the case under consideration are greater than the benefits. 
 
Perhaps some additional experts in survey research (other than us) will now reply to this thread. We have an entire SIG devoted to survey research and we have SIGs devoted to various sorts of evaluation. It would be useful to hear their opinions.  
 
The question again: What following scale do others recommend be used to evaluate the degree of satisfaction/dissatisfaction with the AERA conference, scale A, scale B, or scale C (or some other scale)? And why so?
 

Scale A:  
1=dissatisfied
2=somewhat satisfied
3=satisfied
4=very satisfied.

Scale B:
1=Very dissatisfied
2=Somewhat dissatisfied
3=Somewhat satisfied
4=Very satisfied

or

Scale C:
1=Very dissatisfied
2=Somewhat dissatisfied
3=Neutral
4=Somewhat satisfied
5=Very satisfied
 
Thanks and Cheers. 
BJ
 
On Wed, Apr 29, 2015 at 7:19 PM, Swank, Paul R <[log in to unmask]> wrote:
Sorry, but I disagree. What is the use of a 6 or 7 point scale where half of the values are rarely used. Consider the following distribution of responses on a six point scale.

Strongly disagree.   1
Disagree.                 5
Slightly disagree.   50
Slightly agree.      100
Agree.                   500
Strongly agree.     344

Wouldn't you rather know about the variation in 500 agrees the 56 who disagreed?

Paul Swank
Sent from my iPad

On Apr 29, 2015, at 6:26 PM, Burke Johnson <[log in to unmask]<mailto:[log in to unmask]>> wrote:

That might be true Paul, but using a biased scale is not the solution. The solution is to add to the number of categories on the scale for finer discrimination.
Cheers,
Burke

On Wed, Apr 29, 2015 at 6:04 PM, Swank, Paul R <[log in to unmask]<mailto:[log in to unmask]>> wrote:
The purpose is to spread people out on a continuum. If all the responses are occurring in just a few categories, then the balanced scale fails. This has been a known problem in the scaling literature for years. In bff balanced scales make good sense when a substantial number of the responses are on the positive side of the scale.

Paul Swank

Sent from my iPad

On Apr 29, 2015, at 5:50 PM, William Pendleton <[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>>> wrote:

Exactly.  If the purpose is to equalize the numbers in each grouping, the question is a bad one  That end can be done in coding. Using the unbalanced form that would seem to decrease the number of unfavorable responses to force that end strikes me as poor design.

On Wed, Apr 29, 2015 at 6:17 PM, Swank, Paul R <[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>>> wrote:
The problem with many balanced scales is that when evaluating events or people or objects raters tend to stick to the positive end of the scale. This is a particular bias in raters, especially untrained ones. When 70% or more of the responses are on a positive end then differentiation suffers. There are 2 ways to deal with this, either expand the scale to more levels, which may just give you a large number of levels not endorsed (the negative ones), or make the scale unbalanced. The latter method is more efficient. And by spreading out so many positive responses across more levels, better differentiation is achieved.

Paul Swank, Professor
Health Promotion and Behavioral Sciences
School of Public Health
University of Texas Health Science Center Houston

Sent from my iPad

On Apr 29, 2015, at 4:55 PM, Burke Johnson <[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>><mailto:[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>>>> wrote:

Question for research and measurement experts,

Are any of you concerned about the unbalanced 4-point rating scale being used by the AERA to evaluate the recent annual conference? Here is the unbalanced scale they are using:

1=dissatisfied
2=somewhat satisfied
3=satisfied
4=very satisfied.

It seems pretty convenient and unfair to have 3 of 4 responses indicating satisfaction and only one indicating dissatisfaction in my professional opinion.

I informed the Executive Director of the AERA about this problem two years ago and she did not make a change to a recommended scale.

I had recommended that they use one of the two following scales:

1=Very dissatisfied
2=Somewhat dissatisfied
3=Somewhat satisfied
4=Very satisfied

or

1=Very dissatisfied
2=Somewhat dissatisfied
3=Neutral
4=Somewhat satisfied
5=Very satisfied

If you agree with me about this scaling problem, do you have any suggestions about how to get the AERA to change their scale?

Cheers,
Burke Johnson
Educational Research and Methodology Listserv ----------------------------------------------------------------- List Service Info http://listserv.uconn.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=><https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=Im0RZaeHPaag7mAbo0n7JqaMbXwAaXFld1jQ5lhtc2Y&s=mkVg5ej182j64tRsbWleBgyn1A88Vcv_v6yKBtbLm90&e=><https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=niLm1r0FQ8i1g678LgMjCQmi53Msx9TB2IJCO1o2sYI&s=6rDOa4PShFkiwBAKt01NiZ4UeLjVgXZx3ywCTFBDXD0&e=> To cancel your subscription address click please do the following: SEND an email to the following address: mailto (colon) listserv (at) listserv (dot) uconn (dot) edu Your email should contain only the message UNSUB EDRESMETH-L. Address problems with your subscription to: [log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>><mailto:[log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>>> -----------------------------------------------------------------

Educational Research and Methodology Listserv
-----------------------------------------------------------------
List Service Info http://listserv.uconn.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=><https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=Im0RZaeHPaag7mAbo0n7JqaMbXwAaXFld1jQ5lhtc2Y&s=mkVg5ej182j64tRsbWleBgyn1A88Vcv_v6yKBtbLm90&e=>
To cancel your subscription address click please do the following:
SEND an email to the following address:
     mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
Your email should contain only the message
     UNSUB EDRESMETH-L.
Address problems with your subscription to: [log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>>
-----------------------------------------------------------------



--
Wm W. Pendleton
Professor Emeritus Emory University
3 Breakers Ct.
Salem, SC 29676

Tel. 864 944-2238
Educational Research and Methodology Listserv ----------------------------------------------------------------- List Service Info http://listserv.uconn.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=><https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=Im0RZaeHPaag7mAbo0n7JqaMbXwAaXFld1jQ5lhtc2Y&s=mkVg5ej182j64tRsbWleBgyn1A88Vcv_v6yKBtbLm90&e=> To cancel your subscription address click please do the following: SEND an email to the following address: mailto (colon) listserv (at) listserv (dot) uconn (dot) edu Your email should contain only the message UNSUB EDRESMETH-L. Address problems with your subscription to: [log in to unmask]<mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>> -----------------------------------------------------------------

Educational Research and Methodology Listserv
-----------------------------------------------------------------
List Service Info http://listserv.uconn.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=>
To cancel your subscription address click please do the following:
SEND an email to the following address:
     mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
Your email should contain only the message
     UNSUB EDRESMETH-L.
Address problems with your subscription to: [log in to unmask]<mailto:[log in to unmask]>
-----------------------------------------------------------------

Educational Research and Methodology Listserv ----------------------------------------------------------------- List Service Info http://listserv.uconn.edu<https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=> To cancel your subscription address click please do the following: SEND an email to the following address: mailto (colon) listserv (at) listserv (dot) uconn (dot) edu Your email should contain only the message UNSUB EDRESMETH-L. Address problems with your subscription to: [log in to unmask]<mailto:[log in to unmask]> -----------------------------------------------------------------

Educational Research and Methodology Listserv
-----------------------------------------------------------------
List Service Info http://listserv.uconn.edu
To cancel your subscription address click please do the following:
SEND an email to the following address:
     mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
Your email should contain only the message
     UNSUB EDRESMETH-L.
Address problems with your subscription to: [log in to unmask]
-----------------------------------------------------------------
 
Educational Research and Methodology Listserv ----------------------------------------------------------------- List Service Info http://listserv.uconn.edu To cancel your subscription address click please do the following: SEND an email to the following address: mailto (colon) listserv (at) listserv (dot) uconn (dot) edu Your email should contain only the message UNSUB EDRESMETH-L. Address problems with your subscription to: [log in to unmask] ----------------------------------------------------------------- 
Educational Research and Methodology Listserv ----------------------------------------------------------------- List Service Info http://listserv.uconn.edu To cancel your subscription address click please do the following: SEND an email to the following address: mailto (colon) listserv (at) listserv (dot) uconn (dot) edu Your email should contain only the message UNSUB EDRESMETH-L. Address problems with your subscription to: [log in to unmask] -----------------------------------------------------------------

___________
Professor Diana Kornbrot
 +44 (0) 208 444 2081
 +44 (0) 7403 18 16 12
 skype:  kornbrotme
Work
University of Hertfordshire
College Lane, Hatfield, Hertfordshire AL10 9AB, UK
Home
19 Elmhurst Avenue
London N2 0LT, UK






Educational Research and Methodology Listserv ----------------------------------------------------------------- List Service Info http://listserv.uconn.edu To cancel your subscription address click please do the following: SEND an email to the following address: mailto (colon) listserv (at) listserv (dot) uconn (dot) edu Your email should contain only the message UNSUB EDRESMETH-L. Address problems with your subscription to: [log in to unmask] -----------------------------------------------------------------