690+validity

External and internal validity (including a few of the ways to “control” for it),

more on external validity under the wikipage called "generalizability"

from Marcie's slides (wk 3)



from Fraenkel/Wallen ch. 7

validity: it measures what its supposed to measure

from ch. 8

to what extent are my results applicable to others generalizability : applicable to others special ed might be single subject study, and genealizing 1. ecological generalizability: headstart, is west coast process the same as southeast. what works in headstart will it work in another setting 2. population generalizabity

validity: not about true, but about adequacy, data to support our assertion, not about proving, soft verbs like suggest, indicates, measure covers what it's supposed to.

content, critierium, and construct

validity: appropriateness, meaningfulness, correctness, usefulness of the inferences made by researchers. reliability: consistency of scores from one admin of an instrument to another, and from one set of items to another.

validation is the process of collecting and analyzing evidence to support such inferences. inferences of use are validated not the instrument itself must give attention to way in which researcher intends to interpret the info

appropriate inferences: relevant: related to the purposes of the study meaningful inference: says something about the meaning of the info obtained thru use of an instrument useful inference: helps reserachers make a decision related to what they are trying to find out. validity: amount and type of evidence there is to support interpretations

3 main types of evidence: 1. content-related evidence of validity: content and format of the instrument; if the content that the instrument contains is an adequate sample of the domain of content it is supposed to respresent. questions need to be presented in appropraite format, have a competent judge look over instrument

2. criterion-related evidence of validity: relationship betw scores obtained using the instrument and scores obtained using on or more other instruments or measures (criterion); one instrument (the one being validated) is being compared with performance of an independent one, criterion.

2 forms of criterion-related validity: 1. predictive validity: time interval to elapse betw admin of instrument and obtaining criterion scores. 2. concurrent validity: admin of instrument and gaining scores at the same time both use a correlation coefficient (r): indicates the degree of relationship that exists bets the scores on two instruments. pos. relationship is high score on both instruments (or low and low). all c.c. are betw 11 and 21, whereas a zero mean no relationship

validity coeficient: CC of same individuals, for prediction, the higher the vc then the more likely of prediction expectancy table : two way chard, with predicor categories on left, percentage of indivduals who all withing each, useful for data in schools, easy, clearly show relationship betw 2 measures

3. construct-related evidence of validity: nature of psychological construct or characteristic being measured by the instrument, broadest of the 3, no single piece of evidence, must collect a variety of diff types

3 steps: 1. variable being measured is clearly defined; 2. hypotheses, are formed about how peple who possess a lot vs. a little of the variable will behave in a particular situation; 3. hypotheses are tested both logically and empirically a broad array of evidence, not one particular type




 * from ch. 9 on INTERNAL VALIDITY**

internal validity = any relationship observed betw 2 or more variables should be unambiguous as to what it means rather than being due to something else. (if the something else factors are not controlled or accounted for, researcher can never be sure that they are not the reason for the observed results.

also defined as a means that observed differences on the dependent variable are directly related to the independ variable and not due to some other unintended variable.

threats to internal validity: least attention when planning a study, not discuessed at all, not seen as essential

subject characteristics threat: selection of people may have unintended differences, selection bias: age, strength, maturity, gender, ethnicity, coordination, speed, intelligence, vocab, attitude, reading ability, fluency, manual dexterity, socioeconomic status, religious beliefs, political beliefs researcher must think of what might happen and minimize the effects

mortality threat, or loss of subjects as study progresses, especially with intervention studies that take place over time, absent or fail to finish instrument, if sizable differences in one group rather than another group, most difficult to control, not nec to replace subject with another (bc you don't know that they'd respond the same as those lost) explain the reasons for the loss and how its not relevant, or provide evidence that those lost are similar to those remaining

location threat: may have different results, environment might have diferent respones, best to keep location constant, try to not have them favor or jeopardize a hypothesis

instrument decay: if nature of instrument or scoring is changed someway, longer things to score (like essays) can cause fatigue of the scorer - stricter at first, lenient later, best to schedule to minimize this

data collector characteristics: gender, age, ethnicity, language, other, can affect nature of data collected, if related at all to variables may be used as alternative explanation, use same data collector, or analyze data separately for each collector,

data collector bias: standardize all procedures with training, lack info needed to distort aka planned ignorance, unaware of hypothesis, not told with group to observe or test, or how they perform

testing threat: practice on a pretest caused improvement, some kind of interaction betw a test and the intervention that could skew results, also if subjects can figure out the nature of the study,

history threat: something happens, an event in history, or coincident event, or environmental cause like construction that day, bad storm, etc

maturation threat: change during an intervention may be due to factors associated with the passing of time rather than to the intervention itself.

hawthorne effect: attitude on a study, positive attention and recognition of subjects, or novelty of a treatment remedy would be to have a comparison or control group with special or novel treatment similar to experimental group

regression threat: whenever change is studied in a group that is extremely low or high in its preintervention perfromance, espec. spe. ed. can be explained statistically, but group selected bc of low or high performance wil usually socre closer to the mean on subsequent regardless

impletmentation threat: how they are treated could be different: different people implement different methods: resercher can evaluate the individuals who do the implementing, or when implementers have a bias in favor of one over the other

meta-analysis: locate all studies on topic, statistically combinging the results with less internal validity threats, quantifying replications of a study, but replication is loose bc studies gathered may have little in common

to reduce threats: 1. standardize conditions 2. obtain info on subjects 3. obtain more info on details of study 4. choose an appropriate design

planning helps and collecting additional info before study

from Fraenkel/Wallen website
 * []**

An important consideration in the choice of an instrument to be used in a research investigation is validity: the extent to which results from it permit researchers to draw warranted conclusions about the characteristics of the individuals studied.


 * []**

__Validity__
 * It is important for researchers to use valid instruments for the conclusions they draw are based on the information they obtain using these instruments.
 * The term "validity," as used in research, refers to the appropriateness, meaningfulness, correctness, and usefulness of any inferences a researcher draws based on data obtained through the use of an instrument.
 * Content-related evidence of validity refers to judgments on the content and logical structure of an instrument as it is to be used in a particular study.
 * Criterion-related evidence of validity refers to the degree to which information provided by an instrument agrees with information obtained on other, independent instruments.
 * A criterion is a standard for judging; with reference to validity, it is a second instrument against which scores on an instrument can be checked.
 * Construct-related evidence of validity refers to the degree to which the totality of evidence obtained is consistent with theoretical expectations.
 * A validity coefficient is a numerical index representing the degree of correspondence between scores on an instrument and a criterion measure.
 * An expectancy table is a two-way chart used to evaluate criterion-related evidence of validity.


 * []**

__The Meaning of "Internal Validity"__ __Threats to Internal Validity__ (as a list:) __Controlling Threats to Internal Validity__
 * When a study lacks internal validity, one or more alternative hypotheses exist to explain the outcomes of the study. These alternative hypotheses are referred to by researchers as "threats to internal validity."
 * When a study has internal validity, it means that any relationship observed between two or more variables is unambiguous as to what it means, rather than being due to something else.
 * Some of the more common threats to internal validity are differences in subject characteristics, mortality, location, instrumentation, testing, history, maturation, attitude of subjects, regression, and implementation.
 * 1) Threats to Internal Validity
 * 2) Subject Characteristics
 * 3) Loss of Subjects (Mortality)
 * 4) Location
 * 5) Instrumentation
 * 6) Testing
 * 7) History
 * 8) Maturation
 * 9) Attitude of Subjects
 * 10) Regression
 * 11) Implementation
 * 12) Factors that Reduce the Likelihood of Finding a Relationship
 * The selection of people for a study may result in the individuals or groups differing (i.e., the characteristics of the subjects may differ) from one another in unintended ways that are related to the variables to be studied.
 * No matter how carefully the subjects of a study (the sample) are selected, it is common to lose some of them as the study progresses. This is known as "mortality." Such a loss of subjects may affect the outcomes of a study.
 * The particular locations in which data are collected, or in which an intervention is carried out, may create alternative explanations for any results that are obtained.
 * The way in which instruments are used may also constitute a threat to the internal validity of a study. Possible instrumentation threats include changes in the instrument, characteristics of the data collector(s), and/or bias on the part of the data collectors.
 * The use of pretest in intervention studies sometimes may create a "practice effect" that can affect the results of a study. A pretest can also sometimes affect the way subjects respond to all intervention.
 * On occasion, one or more unanticipated, and unplanned for, events may occur during the course of a study that can affect the responses of subjects. This is known as a history threat.
 * Sometimes change during an intervention study may be due more to factors associated with the passing of time than to the intervention itself. This is known as a maturation threat.
 * The attitude of subjects toward a study (and their participation in it) can create a threat to internal validity.
 * When subjects are given increased attention and recognition because they are participating in a study, their responses may be affected. This is known as the Hawthorne effect.
 * Whenever a group is selected because of unusually high or low performance on a pretest, it will, on the average, score closer to the mean on subsequent testing, regardless of what transpires in the meantime. This is called a regression threat.
 * Whenever an experimental group is treated in ways that are unintended and not a necessary part of the method being studied, an implementation threat can occur.
 * There are a number of techniques or procedures that researchers can use to control or minimize threats to internal validity. Essentially they boil down to four alternatives: (1) standardizing the conditions under which the study occurs; (2) obtaining and using more information on the subjects of the study; (3) obtaining and using more information on the details of the study; and (4) choosing an appropriate design.

from Marcie's study session:

internal validity: things as we are getting to run any type of study, what could go wrong as we set up a study, some we can control or account for, some we can't control