Validity and Realibility

VALIDITY

·         refers to the accuracy of an assessment -- whether or not it measures what it is supposed to measure. Even if a test is reliable, it may not provide a valid measure.

Type of Validity	Definition	Example/
Content	The extent to which the content of the test matches the instructional objectives	A semester or quarter exam that only includes content covered during the last six weeks is not a valid measure of the course's overall objectives -- it has very low content validity.
Criteria	The extent to which scores on the test are in agreement with (concurrent validity) or predict (predictive validity) an external criterion.	If the end-of-year math tests in 4th grade correlate highly with the statewide math tests, they would have high concurrent validity.
Construct	The term construct  is defined as a property that is offered to explain some aspect of human behavior, such as mechanical ability, intelligence, or introversion	early self-esteem studies - self-esteem refers to a person's sense of self-worth or self-respect. Clinical observations in psychology had shown that people who had low self-esteem often had depression. Therefore, to establish the construct validity of the self-esteem measure, the researchers showed that those with higher scores on the self-esteem measure had lower depression scores, while those with low self-esteem had higher rates of depression

Factor affecting Validity	explanation
Nature of the group	Consistency of the validity coefficient for subgroups which differ in any characteristic (e. g. age, gender, educational level, etc, …)
Sample heterogeneity	A wider range of scores results in a higher validity coefficient (range restriction phenomenon)
Criterion-predictor relationship	There must be a linear relationship between predictor and criterion. Otherwise, the Pearson correlation coefficient would be of no use!
Validity-reliability proportionality	Reliability has a limiting influence on validity – we simply cannot validate an unreliable measure!
Moderator variables	Variables like age, gender, personality characteristics may help to predict performance for particular variables only – keep them in mind!
Criterion contamination	Get rid of bias by measuring contaminated influences. Then correct this influence statistically by use of partial correlation.

Reliability

·      The degree of consistency between two measures of the same thing. (Mehrens and Lehman, 1987).

• The measure of how stable, dependable, trustworthy, and consistent a test is in measuring the same thing each time (Worthen et al., 1993)

TYPES OF RELIABILITY	DEFINITION	EXAMPLE
TEST-RETEST	The same form of a test on two or more separate occasions to the same group of examinees (Test-retest)	For example, the examinees will adapt the test format and thus tend to score higher in later tests. Hence, careful implementation of the test-retest approach is strongly recommendation
EQUIVALENT FORM	Two different forms of test, based on the same content, on one occasion to the same examinees	A examinee who took Form A earlier could not share the test items with another student who might take Form B later, because the two forms have different items.
INTERNAL CONSISTENCY	The coefficient of test scores obtained from a single test or survey	The same principle can be applied to a test. When no pattern is found in the students' responses, probably the test is too difficult and students just guess the answers randomly.
SPLIT HALF	A measure of consistency where a test is split in two and the scores for each half of the test is compared with one another.	you have the Math test and divide the items on it in two parts. If you correlated the first half of the items with the second half of the items, they should be highly correlated if they are reliable.
INTER RATER	When multiple people are giving assessments of some kind or are the subjects of some test, then similar people should lead to the same resulting scores.	Two people may be asked to categorize pictures of animals as being dogs or cats. A perfectly reliable result would be that they both classify the same pictures in the same way.

FACTORS THAT LOWER THE RELIABILITY OF ASSESSMENTS Insufficient number of tasks Remedy: Accumulate results from several assessments Poorly structured assessment procedures Remedy: Define carefully nature of tasks, conditions for obtaining the assessment and the criteria for scoring and judging the results. Dimensions of performance are specific to the tasks Remedy: Increase generalizability of performance by selecting tasks that have dimensions like those in similar tasks Inadequate scoring guides for judgemental scoring Remedy: Using scoring rubrics or rating scales that specifically describe the criteria and levels of quality Scoring judgements that are influenced by personal bias Remedy: Check scores with those of an independent judge. Receive training in scoring and rating if possible

 

The relationship between validity and reliability.

At best, we have a measure that has both high validity and high reliability. It yields consistent results in repeated application and it accurately reflects what we hope to represent.

It is possible to have a measure that has high reliability but low validity - one that is consistent in getting bad information or consistent in missing the mark. *It is also possible to have one that has low reliability and low validity - inconsistent and not on target.

Finally, it is not possible to have a measure that has low reliability and high validity - you can't really get at what you want or what you're interested in if your measure fluctuates wildly.

 

Rubris for KPD3016 Blogspot

RUBRIC ASSESMENT ON KPD3016 BLOG

Construct	1	2	3	4	5	Total
Ideas and Content	The ideas expressed are not original, often confused and are not connected to discussions around education or ed. technology.	The ideas expressed are not necessarily original, and are not usually connected to discussions around education or ed. technology.	The student expresses some original ideas but not usually connected to discussions around education or ed. Technology.	The student expresses some original ideas. The majority of ideas are related to education or more specific to ed. technology.	The student has many original ideas and expresses them clearly. The great majority of ideas are related to education or more specifically to ed. technology.
Writing Quality	Posts are of very poor quality. There is little to no evidence of reading other information in order to form new meaning of the topics at-hand.	Posts show a below average, overly casual writing style with a lack of attention to style. Students pay little attention to other reading and mostly regurgitate previous personal views.	Post show medium average. The content show student do reads but in moderate and not attempts to synthesize information and not form new meaning.	Posts show above average writing style. The content demonstrates that the student reads moderately, and attempts to synthesize information and form new meaning.	Posts are well written, and are characterized by elements of a strong writing style. The content demonstrates that the student is well read, synthesizes learned content and constructs new meaning.
Organisation	Fails to provide a consistent organizational structure and shows no awareness of the need for consistency.	Uses a loosely defined organizational structure that shows minimal awareness of the need for consistency.	Uses mediumly consistent organizational structure that shows awareness of the need for consistency.	Uses a generally consistent organizational structure, with the most current posting listed at the top.	Uses a consistent organizational structure that is easy to follow and places the most recent posts at the top of the page.
Post Frequency	The post is not updated.	The post updated once a month.	The post is rarely updated.	The post updated once a week.	The post frequently updated.
Creativity on post	Use of unattractive colour,  too detail explanation ,  upload document and not upload image	Use of unattractive colour,  too detail explanation ,  upload document, not upload image	Use of attractive colour, too detail explanation, upload document and image	Use of attractive colour, summarized article on certain post,  upload document and image	Use of attractive colour, mindmap and summarized article,  upload document and image
Blog design creativity	Use unattractive colour of background, cannot read post clearly, not using attractive side accessories.	Use attractive colour of background, cannot read post clearly, not using attractive side accessories.	Use unattractive colour of background,  can read post clearly, not using attractive side accessories.	Use attractive colour of background, can read post clearly, not using attractive side accessories.	Use attractive colour of background, can read post clearly, using attractive side accessories.
Respond from readers	Less and bad comments from readers	Bad comments from readers and rarely good comments from readers	Some good and some bad comment from readers	Good comments from readers and rarely bad comments from readers	More and good comments from readers