Dear colleagues -- thank you for your responses. Now weren't they much
more helpful to our student researchers (past, present and future) as well
as our esteemed colleagues. I know it was of much greater interest to me.
After more than two decades in the field of research, evaluation and
measurement and seeing the change in access to research participants and in
sampling, and the field at large, it was very informative. Thanks.
Best,
Barbara
On Fri, Apr 18, 2014 at 1:20 PM, Burke Johnson <[log in to unmask]>wrote:
> Perhaps this is not an either/or question (the results either do
> generalize or they do not). Are the thousands of experiments (including the
> strong designs with random assignment) based on random samples? That is
> rarely, if ever, the case. Can any conclusions be made about people based
> on experiments? I think the answer is yes, especially when we see a similar
> result across experiments in different ecological settings with different
> kinds of people. Have any good theoretical generalizations come from
> research not based on random samples? Think about Piaget. Could the study
> discussed here be viewed as *one* study needing to be "replicated" with
> other people, but it is still a study that can provide some tentative and
> useful working knowledge? Did you expect to generalize from a single
> empirical research study (other than in polls) in your past research? If
> yes, who did you generalize to? All humans? All adults in the US? All
> members of our excellent educational research methodology list? Please
> provide an example for us to discuss. Hopefully the authors of the study
> are collecting demographic information at the end of their questionnaire
> and will use that information to inform their readers of the
> characteristics of their sample. Then intelligent readers can engage in
> making what Robert Stake called naturalistic generalization, comparing
> their own/local people to the sample and then making cautious
> generalizations based on similarity; that is, the reader rather than the
> researcher might make any appropriate generalizations. Stake's idea was
> similar to Don Campbell's concept of proximal similarity, or generalizing
> based on similarity. Related to this discussion, do you ever use p values
> on empirical (including experimental) data based on convenience samples? Or
> do you only use p values when you have a random sample from a delimited
> population? I am not saying here what I think is "correct" on this issue;
> I'm just asking the question. Getting back to the idea of asking everyone
> to participate in the internet study, could the EDRESMETH group be
> considered a population and those that don't participate non respondents?
> The response rate would then apply to the population rather than to the
> sample. Would p values and inferential statistics be needed, if one assumes
> he or she is working with population data (with less than a 100% response
> rate)?
>
> I used to be a stickler about random sampling, and I still am if one is
> claiming to use a single sample to provide statistical estimates of
> parameters in a particular population. However, I think there are
> additional varieties and degrees of generalizations.
>
> Again, I LOVE random samples! I just don't think that the lack of a random
> sample necessarily invalidates a study (unless, again, the author claims to
> make a statistical generalization from a single sample to a population).
>
> In short, I think the answer to the original question (to generalize or
> not) requires a more nuanced answer.
>
> I also think we should be a little kinder to those asking us to
> participate in their research. Let's shift our discussion to something like
> this: "Can one generalize from internet research?"
>
> Cheers!
> Burke Johnson
>
>
> On Fri, Apr 18, 2014 at 9:12 AM, Dennis Roberts <[log in to unmask]> wrote:
>
>> On 4/17/2014 10:40 PM, Tanesia Beverly wrote:
>>
>>> You are invited to participate in a study investigating the perceptions
>>> a=
>>> nd
>>> attitudes about standardized test taking ability.
>>>
>>
>> What I have noticed is that more and more folks are posting requests on
>> the internet for "volunteers" to participate in some kind of a research
>> study (usually a survey). To me this is not a good way to define and
>> obtain a sample ... though it appears to be a fairly easy one. My question
>> is ... let's say the researcher gets n=87 to respond .... WHAT IS THE
>> POPULATION TO WHICH THESE RESULTS (whatever they are) CAN BE GENERALIZED?
>>
>> Educational Research and Methodology Listserv
>> -----------------------------------------------------------------
>> List Service Info http://listserv.uconn.edu
>> To cancel your subscription address click please do the following:
>> SEND an email to the following address:
>> mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
>> Your email should contain only the message
>> UNSUB EDRESMETH-L.
>> Address problems with your subscription to: [log in to unmask]
>> -----------------------------------------------------------------
>>
>
> Educational Research and Methodology Listserv
> ----------------------------------------------------------------- List
> Service Info http://listserv.uconn.edu To cancel your subscription
> address click please do the following: SEND an email to the following
> address: mailto (colon) listserv (at) listserv (dot) uconn (dot) edu Your
> email should contain only the message UNSUB EDRESMETH-L. Address problems
> with your subscription to: [log in to unmask]
--
Barbara J. Helms, Ph.D.
Education Evaluation Consultant
2144 Commonwealth Avenue
Auburndale, MA 02466
Telephone: 617-480-2260
Educational Research and Methodology Listserv
-----------------------------------------------------------------
List Service Info http://listserv.uconn.edu
To cancel your subscription address click please do the following:
SEND an email to the following address:
mailto (colon) listserv (at) listserv (dot) uconn (dot) edu
Your email should contain only the message
UNSUB EDRESMETH-L.
Address problems with your subscription to: [log in to unmask]
-----------------------------------------------------------------
