Please note that my objective is to determine the *degree *to which
participants are satisfied or dissatisfied with the current object of
evaluation (the conference). (If I only cared about satisfied vs.
dissatisfied I could use a dichotomous scale, but that is not my interest.)
Also, I do not know *a priori* what the responses will be because that is
an *empirical question*; neither does the AERA know (for this year's
conference) despite what might have occurred multiple years ago. (Check out
David Hume's problem of induction.) Next, a survey or marketing or
evaluation researcher would not recommend that a client use an unbalanced
scale, except in exceptional circumstances. In my opinion, this is not one
of those circumstances. The vast majority of books on measurement and
marketing research *exclude *discussion of the practice altogether because
it is controversial. An unbalanced scale can bring the one advantage
mentioned by Paul, but it also brings with it the problem of not being
defensible IF one wants to know the distribution of satisfaction *and
*dissatisfaction
and make claims about this to the public. That is what I want to know. I
thought that was the purpose of the evaluation being discussed here. We
cannot (we should not) just *assume *that most everyone is satisfied when
evaluating a program or other object of interest and then systematically
eliminate categories for responding (in my n=1 expert opinion). How would
you feel if you were just a little dissatisfied (or not quite sure until
you carefully considered the issue) or if you were extremely dissatisfied
and you noticed that satisfied people were given three options but you were
only given one option to place your opinion? Would you feel that the rating
scale was fair, or would you question it? That is the person that is *hurt*
by this scale. Is that fair for a conference focused on social justice? How
would a third-party evaluator or auditor or even an interested consumer
feel when shown the scale currently used, IF the purported *purpose is to
obtain empirical evidence of the degrees of satisfaction/dissatisfaction
with the conference* (and not just to differentiate responses in the scale
region of satisfaction and censor the rest)? In short, I am arguing that
the costs of using the unbalanced scale in the case under consideration are
greater than the benefits.
Perhaps some additional experts in survey research (other than us) will now
reply to this thread. We have an entire SIG devoted to survey research and
we have SIGs devoted to various sorts of evaluation. It would be useful to
hear their opinions.
*The question again*: What following scale do others recommend be used to
evaluate the degree of satisfaction/dissatisfaction with the AERA
conference, scale A, scale B, or scale C (or some other scale)? And why so?
Scale A:
1=dissatisfied
2=somewhat satisfied
3=satisfied
4=very satisfied.
Scale B:
1=Very dissatisfied
2=Somewhat dissatisfied
3=Somewhat satisfied
4=Very satisfied
or
Scale C:
1=Very dissatisfied
2=Somewhat dissatisfied
3=Neutral
4=Somewhat satisfied
5=Very satisfied
Thanks and Cheers.
BJ
On Wed, Apr 29, 2015 at 7:19 PM, Swank, Paul R <[log in to unmask]>
wrote:
> Sorry, but I disagree. What is the use of a 6 or 7 point scale where half
> of the values are rarely used. Consider the following distribution of
> responses on a six point scale.
>
> Strongly disagree. 1
> Disagree. 5
> Slightly disagree. 50
> Slightly agree. 100
> Agree. 500
> Strongly agree. 344
>
> Wouldn't you rather know about the variation in 500 agrees the 56 who
> disagreed?
>
> Paul Swank
> Sent from my iPad
>
> On Apr 29, 2015, at 6:26 PM, Burke Johnson <[log in to unmask]
> <mailto:[log in to unmask]>> wrote:
>
> That might be true Paul, but using a biased scale is not the solution. The
> solution is to add to the number of categories on the scale for finer
> discrimination.
> Cheers,
> Burke
>
> On Wed, Apr 29, 2015 at 6:04 PM, Swank, Paul R <[log in to unmask]
> <mailto:[log in to unmask]>> wrote:
> The purpose is to spread people out on a continuum. If all the responses
> are occurring in just a few categories, then the balanced scale fails. This
> has been a known problem in the scaling literature for years. In bff
> balanced scales make good sense when a substantial number of the responses
> are on the positive side of the scale.
>
> Paul Swank
>
> Sent from my iPad
>
> On Apr 29, 2015, at 5:50 PM, William Pendleton <[log in to unmask]<mailto:
> [log in to unmask]><mailto:[log in to unmask]<mailto:[log in to unmask]>>>
> wrote:
>
> Exactly. If the purpose is to equalize the numbers in each grouping, the
> question is a bad one That end can be done in coding. Using the unbalanced
> form that would seem to decrease the number of unfavorable responses to
> force that end strikes me as poor design.
>
> On Wed, Apr 29, 2015 at 6:17 PM, Swank, Paul R <[log in to unmask]
> <mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:
> [log in to unmask]>>> wrote:
> The problem with many balanced scales is that when evaluating events or
> people or objects raters tend to stick to the positive end of the scale.
> This is a particular bias in raters, especially untrained ones. When 70% or
> more of the responses are on a positive end then differentiation suffers.
> There are 2 ways to deal with this, either expand the scale to more levels,
> which may just give you a large number of levels not endorsed (the negative
> ones), or make the scale unbalanced. The latter method is more efficient.
> And by spreading out so many positive responses across more levels, better
> differentiation is achieved.
>
> Paul Swank, Professor
> Health Promotion and Behavioral Sciences
> School of Public Health
> University of Texas Health Science Center Houston
>
> Sent from my iPad
>
> On Apr 29, 2015, at 4:55 PM, Burke Johnson <[log in to unmask]
> <mailto:[log in to unmask]><mailto:[log in to unmask]
> <mailto:[log in to unmask]>><mailto:[log in to unmask]
> <mailto:[log in to unmask]><mailto:[log in to unmask]
> <mailto:[log in to unmask]>>>> wrote:
>
> Question for research and measurement experts,
>
> Are any of you concerned about the unbalanced 4-point rating scale being
> used by the AERA to evaluate the recent annual conference? Here is the
> unbalanced scale they are using:
>
> 1=dissatisfied
> 2=somewhat satisfied
> 3=satisfied
> 4=very satisfied.
>
> It seems pretty convenient and unfair to have 3 of 4 responses indicating
> satisfaction and only one indicating dissatisfaction in my professional
> opinion.
>
> I informed the Executive Director of the AERA about this problem two years
> ago and she did not make a change to a recommended scale.
>
> I had recommended that they use one of the two following scales:
>
> 1=Very dissatisfied
> 2=Somewhat dissatisfied
> 3=Somewhat satisfied
> 4=Very satisfied
>
> or
>
> 1=Very dissatisfied
> 2=Somewhat dissatisfied
> 3=Neutral
> 4=Somewhat satisfied
> 5=Very satisfied
>
> If you agree with me about this scaling problem, do you have any
> suggestions about how to get the AERA to change their scale?
>
> Cheers,
> Burke Johnson
> Educational Research and Methodology Listserv
> ----------------------------------------------------------------- List
> Service Info http://listserv.uconn.edu<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=
> ><
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=Im0RZaeHPaag7mAbo0n7JqaMbXwAaXFld1jQ5lhtc2Y&s=mkVg5ej182j64tRsbWleBgyn1A88Vcv_v6yKBtbLm90&e=
> ><
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=niLm1r0FQ8i1g678LgMjCQmi53Msx9TB2IJCO1o2sYI&s=6rDOa4PShFkiwBAKt01NiZ4UeLjVgXZx3ywCTFBDXD0&e=>
> To cancel your subscription address click please do the following: SEND an
> email to the following address: mailto (colon) listserv (at) listserv (dot)
> uconn (dot) edu Your email should contain only the message UNSUB
> EDRESMETH-L. Address problems with your subscription to:
> [log in to unmask]<mailto:[log in to unmask]><mailto:
> [log in to unmask]<mailto:[log in to unmask]>><mailto:
> [log in to unmask]<mailto:[log in to unmask]><mailto:
> [log in to unmask]<mailto:[log in to unmask]>>>
> -----------------------------------------------------------------
>
> Educational Research and Methodology Listserv
> -----------------------------------------------------------------
> List Service Info http://listserv.uconn.edu<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=
> ><
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=Im0RZaeHPaag7mAbo0n7JqaMbXwAaXFld1jQ5lhtc2Y&s=mkVg5ej182j64tRsbWleBgyn1A88Vcv_v6yKBtbLm90&e=
> >
> To cancel your subscription address click please do the following:
> SEND an email to the following address:
> mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
> Your email should contain only the message
> UNSUB EDRESMETH-L.
> Address problems with your subscription to: [log in to unmask]
> <mailto:[log in to unmask]><mailto:[log in to unmask]<mailto:
> [log in to unmask]>>
> -----------------------------------------------------------------
>
>
>
> --
> Wm W. Pendleton
> Professor Emeritus Emory University
> 3 Breakers Ct.
> Salem, SC 29676
>
> Tel. 864 944-2238
> Educational Research and Methodology Listserv
> ----------------------------------------------------------------- List
> Service Info http://listserv.uconn.edu<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=
> ><
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=Im0RZaeHPaag7mAbo0n7JqaMbXwAaXFld1jQ5lhtc2Y&s=mkVg5ej182j64tRsbWleBgyn1A88Vcv_v6yKBtbLm90&e=>
> To cancel your subscription address click please do the following: SEND an
> email to the following address: mailto (colon) listserv (at) listserv (dot)
> uconn (dot) edu Your email should contain only the message UNSUB
> EDRESMETH-L. Address problems with your subscription to:
> [log in to unmask]<mailto:[log in to unmask]><mailto:
> [log in to unmask]<mailto:[log in to unmask]>>
> -----------------------------------------------------------------
>
> Educational Research and Methodology Listserv
> -----------------------------------------------------------------
> List Service Info http://listserv.uconn.edu<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=
> >
> To cancel your subscription address click please do the following:
> SEND an email to the following address:
> mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
> Your email should contain only the message
> UNSUB EDRESMETH-L.
> Address problems with your subscription to: [log in to unmask]
> <mailto:[log in to unmask]>
> -----------------------------------------------------------------
>
> Educational Research and Methodology Listserv
> ----------------------------------------------------------------- List
> Service Info http://listserv.uconn.edu<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listserv.uconn.edu&d=AwMFaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=MLbJazHRlJtdtbj_qj01IvESP88OVkIcc-s3JCckozo&s=dd5ok_JHNpvoJXz7XZaR0WBEYyyCbr655v1PB436DB4&e=>
> To cancel your subscription address click please do the following: SEND an
> email to the following address: mailto (colon) listserv (at) listserv (dot)
> uconn (dot) edu Your email should contain only the message UNSUB
> EDRESMETH-L. Address problems with your subscription to:
> [log in to unmask]<mailto:[log in to unmask]>
> -----------------------------------------------------------------
>
> Educational Research and Methodology Listserv
> -----------------------------------------------------------------
> List Service Info http://listserv.uconn.edu
> To cancel your subscription address click please do the following:
> SEND an email to the following address:
> mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
> Your email should contain only the message
> UNSUB EDRESMETH-L.
> Address problems with your subscription to: [log in to unmask]
> -----------------------------------------------------------------
>
Educational Research and Methodology Listserv
-----------------------------------------------------------------
List Service Info http://listserv.uconn.edu
To cancel your subscription address click please do the following:
SEND an email to the following address:
mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
Your email should contain only the message
UNSUB EDRESMETH-L.
Address problems with your subscription to: [log in to unmask]
-----------------------------------------------------------------
