Topics 1 & 2: Norms & Reliability {by 6/1}
Based on the text readings and lecture recording due this week consider the following three discussion points: (1) In your own words, provide a general description of the difference between criterion/domain-referenced instruments and norm-referenced instruments. Why is it important to understand this difference? (2) Correlation does not equal causation! Share your thoughts on why this assumption is still a common mistake, especially in the mental health field. (3) Share your thoughts on why reliability is so (wicked) important for psychological assessments (this is a bit of a deep question – give it your best shot).
Your original post should be posted by 6/1. Post your two replies no later than 6/3. *Please remember to click the “reply” button when posting a reply. This makes it easier for the reader to follow the blog postings.
Dr. Adam M Volungis 
May 24, 2023 @ 12:33:34
Criterion/Domain-Referenced Instruments vs. Norm-Referenced Instruments:
Criterion/domain-referenced instruments compare an individual’s performance on a test to an established criterion or standard, focusing on what the test takers can do and know. On the other hand, norm-referenced instruments interpret an individual’s performance by comparing it to a specific group (standardized sample) that has taken the same test, indicating the individual’s relative standing within that normative sample. Understanding this difference is important as it helps professionals select the appropriate assessment approach based on their goals and the specific context of the assessment.
Correlation does not equal causation:
The assumption that correlation implies causation is a common mistake not only in the mental health field but also in various other domains. This misconception arises due to a desire for simple explanations or causal relationships for complex phenomena. However, correlation merely indicates a statistical relationship between two variables and does not prove causation. To establish causality, rigorous research designs, such as experimental studies or longitudinal investigations, are necessary to control for confounding factors, establish temporal relationships, and provide stronger evidence for causal relationships.
Importance of reliability in psychological assessments:
Reliability is indeed crucial in psychological assessments. It ensures consistency and stability in measurement when the testing procedure is repeated on a population of individuals or groups. Reliability estimates the proportion of true variance versus error variance, providing a foundation for valid measurement. Reliable assessments offer precise and consistent measurement, which is essential for making informed decisions about treatment planning, progress monitoring, and evaluating treatment outcomes. Additionally, reliability helps ensure fairness and equity in assessments by reducing measurement bias and providing consistent evaluation across individuals.
Jun 01, 2023 @ 12:51:01
Hi William! I liked the way you explained everything in your post! I especially like how you described the assumption that correlation doesn’t equal causation. It does come down to people wanting a simple explanation for complex things (if only it were that simple). Of course, we can never assume that correlation equals causation because, as you described, correlation is only about the statistical relationship between two variables and does not prove that they are related. This is why it is important to perform reliable tests and gather valid information before proving causal relationships.
Jun 02, 2023 @ 21:27:04
Hello William! I really enjoyed reading your answers to each of the questions. The way you worded your third question brought some ideas to the importance of Reliability was insightful. The point being that “Reliability estimates the proportion of true variance versus error variance, providing a foundation for valid measurement”. This is very important as being able to tell the is a measuring will be able to be used on different occasions along with the corresponding variables to each test. This can also lead to decisions regarding treatment plan, clients progression, as well as the outcomes. All of which you summed up perfectly in your discussion post!
May 31, 2023 @ 19:37:11
Hi everyone! Criterion-referenced instruments are used when an individual’s score is compared with a standard that has already been established, such as a test being graded on an A-D scale. Norm-referenced instruments, on the other hand, compare the individual’s score to that of others who have also taken the instrument. The distinction between each of these instruments is crucial when a professional is choosing the appropriate instrument for the client or situation; in some scenarios, criterion-referenced instruments are not appropriate, and vice versa.
The misconception of correlation equaling causation often occurs because correlation compares two variables, and when a strong relationship is present, it can be easy to assume that one variable causes another. However, this cannot prove causality between two variables; the only thing that can strengthen our hypothesis of causality is repeated experimental studies with evidence leaning in the direction of causation.
Reliability is extremely important to consider when carrying out assessment because it helps to determine the precision and consistency of the instrument across uses. If an assessment is found to be unreliable, it can heavily influence crucial decisions made regarding client progress and plans of treatment. Knowing the reliability of an instrument can help professionals make the best choices of assessment for their clients.
Jun 02, 2023 @ 21:41:02
Hi Chase! I appreciate your answers to all of the questions in the discussion post. In regards to the first question, I like how you included that the example of criterion-referenced instruments reflecting on test scores on a A-D range. This gives a classic everyday example that helps to better understand the importance to the rights and wrongs of using each instrument. For your explanation on the misconception I appreciate the way you worded the hypotheses for strengthening causality, and how even after a correlation has been discovered it would take multiple tests of the same nature to represent any type of causality. Finally, for your answer on reliability, I believe you an I have a similar understanding as to why it is so wicked important. Reliability helps to understand the consistency of instruments and the results that come from the tests, which then leads to better understanding clients diagnoses, progress, best course of action for treatment, and how treatment is working.
Jun 03, 2023 @ 23:02:56
Hi Chase! It is good as you mentioned that a strong relationship can lead to assume that one variable directly causes the other. I think that most people tend to do like that. You made a good point when you referred to repeated experimental studies. When the same experimental manipulation consistently produces similar results across multiple studies, it strengthens the evidence for a causal relationship. Repetition of the experiment with different samples, settings, or researchers enhances reliability of the causal relationship. By accumulating evidence from multiple replications, researchers can strengthen the case for causality and enhance the scientific validity of their findings.
May 31, 2023 @ 20:15:29
Based on my understanding, criterion / domain referenced instruments lack comparison but rather provide a cut off score / percentage that determines if the individual’s score has reached a level of mastery. The percentage cut off that establishes mastery is a standard that is determined by experts and previous data. I know in the lecture video it discussed the licensure in Massachusetts for LMHC has two criterion / domain referenced parts where if not meeting the cut off in one means failure in both but meeting the cut off in both declares mastery. Norm referenced instruments differ in the comparison aspect. A norm referenced instrument is an individual’s score compared to a standardized norm (established by other individuals scores on the same instrument). The norm referenced instrument is comparing scores between other standardized scores whereas the criterion / domain referenced instruments compares a score to an established mastery cut off.
Correlation equaling causation still seems to be a common misconception. Correlation is used to compare the relationship between two different variables. Correlation is used to provide evidence of consistency. When two variables are correlated this means that they occur in the same direction rather than that they cause each other. For instance, the example of tv violence and aggression being positively correlated indicates that tv violence and aggression move in the same direction. When one increases so does the other. However, correlation does not take into account other factors such as the family, income, culture, age, gender, and other hidden variables etc. In order to develop causation there would need to be repeated controlled studies that take into account other variables.
Reliability is so (wicked) important for psychological assessments as without reliability we will not be able to establish validity. Assessments need to show consistency. Without consistency there would be a lack of confidence with results from different instruments. In other words, we wouldn’t want to consider a diagnosis or treatment plan for a client based upon the same test yielding different results for the same individual. We want to see that the assessment we are using is yielding consistent results or else we would be making decisions that aren’t backed by enough data / information.
May 31, 2023 @ 23:05:12
Leah! I absolutely love this example of the mistake that is assuming causation because of correlation! This makes me think a lot about some of the things that I see and hear in the school system sometimes. Teachers/parents/administration will oftentimes take a behavior they are seeing in a child and assume that it is due to a factor that they have noticed occurred around the same time as the behavior without actually thinking critically about the child. For example, a child will have some more defiant behaviors in the classroom and because these behaviors arose around the same time as a change in their family life (birth of a niece, grandmother moving etc ), people will attribute the behaviors to that one change instead of thinking more broadly about what is going on. This kind of reminds me of what William said about this topic, in that people fall for this assumption because it provides simple explanations for things that are often difficult to understand. I think the ideas people form based off of thinking that correlation = causation lack depth and show that no further examination is happening! This is why it is so important for counselors to understand that there are many things to observe and measure when trying to substantiate a claim that something is causal.
Jun 03, 2023 @ 09:55:03
Hi Leah! I liked the example you used to explain that correlation does not equal causation. I think that example, in particular, has been used to explain any “violent” behaviors in children due to the violent video games they may play or movies they watch. It’s important to do studies that show the actual causation of these “violent” behaviors and never to assume a correlation. You explained this very well!
May 31, 2023 @ 20:32:07
The difference between criterion-referenced instruments and norm-referenced instruments:
Criterion/domain-referenced instruments are tests that compare an individuals score to that of previously established standard or criterion. However, when reviewing the individuals scores the importance is to focus on the scores in respects to the standard or criterion rather than the comparison of other participants scores. While Counselors are reviewing the individuals scores a sense of understanding the domain is required to then see where the standard of the results lies. Norm-referenced instruments are different to that of criterion-referenced instruments in the sense that the results are compared between each participant. These instruments can be used on a small to large scale such as a classroom or a sample of a couple 1000.
Why correlation does not equal causation:
There is a large misconception between correlation and its relationship to causation. The misconception is that correlation equals causation which is false due to correlation being the comparison of two variables and how that is represented through a showing of a strong relationship (whether that be positive or negative). Due to this, when there is a strong relationship, the misconception is that one variable causes the impact of the second variable. However, this is not true as to prove there is any causation would require continuation of research of experimental design and time requirement of similar or same correlative research.
Why is reliability so wicked important:
Reliability is so wicked important when considering the trustworthiness of testing and results within psychological assessments. Testing helps to prove consistency which then leads to the understanding of diagnoses, progress, and treatment of clients. This way the best decisions are made for each client to have a positive progress and keep growing to move forward in testing, instruments, treatments, and so on.
Jun 01, 2023 @ 11:08:27
I appreciate your comprehensive and accurate description of criterion-referenced and norm-referenced instruments. Criterion-referenced instruments indeed compare an individual’s score to a pre-established standard or criterion, focusing on the performance in relation to the specified benchmark rather than comparing it to other participants’ scores. Understanding the domain is crucial for professionals to determine the position of the results in relation to the established standards. On the other hand, norm-referenced instruments compare the results among participants, making them suitable for evaluating larger groups, such as classrooms or samples of a few thousand. Your explanation of the misconception regarding correlation and causation is spot-on. It’s important to note that correlation, which indicates the strength and direction of the relationship between two variables, does not imply causation. While a strong correlation may suggest a relationship between variables, it doesn’t establish a causal link. To determine causation, further research, experimental design, and replication of correlative findings are necessary. Reliability, as you highlighted, is indeed of paramount importance in psychological assessments. It ensures the trustworthiness and consistency of testing and results. By demonstrating consistency, reliability allows for accurate diagnoses, tracking progress, and determining appropriate treatments. The incorporation of reliable measures enables clinicians to make well-informed decisions that contribute to positive client progress and facilitate advancements in testing, instruments, and treatments.
Jun 01, 2023 @ 13:50:55
Hey Megan! Your answer to number three made me think of something that both Professor Volungis and the book mentioned and that is that assessment is not just a part of our clients intake. It is important that we utilize assessment instruments at the beginning, end and throughout treatment as well. Doing so allows us to track whether or not our decided treatment interventions are working to help our clients meet the goals that we established with them as we began our treatment. It is important that throughout, we are using reliable instruments. Not only is it important that we carefully select the best form of assessment at the start, but to maintain its use to ensure the most reliability in scores that will be reflective of the clients progress (or regression–hopefully not).
Jun 02, 2023 @ 13:22:52
Hi Meaghan! I liked your explanation of why correlation does not equal causation, especially since it most commonly occurs when there is a strong relationship between two variables. Even with this strong relationship however, we cannot be sure if one variable causes another unless we control for potential outside variables that may influence the correlational relationship. In your discussion of criterion-referenced instruments, I thought it was valuable for you to mention the importance of the counselor having a good understanding of the applicable domain being assessed. This is crucial to gain a solid interpretation from the assessment data.
May 31, 2023 @ 22:57:12
It is important for all counselors to know the difference between criterion/domain-referenced instruments and norm-referenced instruments. Criterion/domain-referenced instruments are used when an examiner is attempting to make a comparison of the individual to a set standard of the quality they are trying to assess. For example, teachers or school counselors may use a criterion/domain-referenced instrument when deciding whether or not a student has met standards of the grade they are in or are at the level that is usually expected for their next grade level. Norm referenced instruments specifically are used when an examiner wishes to compare the individual to a specific group in which they share similar characteristics. In the same respect as before, a teacher may use a norm referenced instrument to assess their student in regards to other children in their class, grade level or reading level. Norm referenced instruments helps the counselor see how their client compares to those in their groups, while domain referenced instruments compare them to a widely established standard.
Statistics can illustrate relationships between data that points us towards a greater understanding of an issue at hand. But one mistake that is commonly made is seeing a statistical correlation and assuming causation. An example of how this assumption could go awry in the field of mental health is if a counselor is simply administering testing and analyzing the data without further investigation, they may casually assume that the relationship between the clients symptoms and risk factors they have discussed is simple. For example, if the persons’ symptoms had a statistical correlation with their rates of work stress, the counselor may falsely assume that work stress is causing the symptoms and work to help them ameliorate the work stress without considering all other factors, leading to ineffective treatment.
Reliability in psychological testing is very important because without it, we would not be able to feel confident in the results that our assessment provides to us. From my understanding, reliability is the degree to which an instrument is able to provide us with a consistent measure. This is important because if we were to administer an instrument to our client that has weak reliability, without test-retest or some other form of assessing the reliability, we may never know that their score is inaccurate. This may lead to incorrect assumptions and treatment decisions that are less effective. Ultimately, ensuring reliability in assessment leads to optimal knowledge of our clients and their symptoms as well as successful treatment.
Jun 02, 2023 @ 13:15:13
Hi Maura! You make a good distinction between the two instruments. In my opinion, I feel like it would make more sense for colleges & universities to use criterion / domain – referenced instruments over norm referenced instruments. You would think that when taking a class, you would want to show mastery of material in order to pass. I found a lot of times in undergrad that our grades were scaled (based upon how all of the students did – norm referenced). Reflecting back on that, it doesn’t make much sense. There was one class I took and it seemed everyone did pretty bad on the exams, yet we all passed with B’s & up, because of how the class was scaled. If this class were to use the idea of a criterion / domain referenced instrument it would have been a reflection of how much we actually know (mastery) versus none of us knowing a lot but getting by because everyone did bad. Looking at the misconception of correlation equalling causation you point out something that is really important. In this field making an assumption, like causation, is a BIG mistake. Think of all the maladaptive outcomes that could come of that. The client would not be receiving the right treatment, their symptoms may get worse – as we could be treating the wrong thing, and it could cause the client to have a much harder time than necessary. Treatment plans are so important, which is why we need to think of the unknown variables that could be contributing rather than assuming causation. Reliability is also very important. I was thinking about the aftermath of giving a client an assessment that has low reliability. If we give a client an assessment repeatedly, and the results are consistently different, what information is this providing us with? Nothing! We have no idea what is going on, and we can’t just pick one assessment result and ignore the others. We have to have reliability with assessments, with that, we can hopefully establish validity.
May 31, 2023 @ 23:38:13
Criterion/domain-referenced instruments are designed to measure an individual’s performance against a predetermined set of criteria or standards. On the other hand, norm-referenced instruments are designed to compare an individual’s performance to the performance of a larger group, known as the norm group, which is representative of the population. Understanding this difference is important because it helps in selecting the appropriate assessment tool for specific purposes, such as goal setting, instructional planning, or decision-making.
The phrase “correlation does not equal causation” means that the presence of a correlation between two variables does not necessarily imply a causal relationship between them. While two variables may be associated or vary together, it does not mean that one variable directly causes the other. However, there is often a desire for clear causes and solutions in the mental health field. The complexity of mental health issues and the search for straightforward explanations can lead to the assumption that correlations imply causation. In addition, many studies rely on observational designs, which can establish correlations but cannot definitively prove causation. In the absence of strong experimental evidence, there may be a tendency to infer causality based on observed correlations.
Reliability is exceptionally important for psychological assessments because it refers to the consistency and accuracy of measurement. High reliability ensures that the assessment produces consistent results across different administrations, settings, and raters. Without reliability, the results would be erratic and unreliable. Also, reliability is a fundamental prerequisite for validity, which refers to the extent to which an assessment measures what it claims to measure. Reliability acts as a foundation for valid inference and ensures that the conclusions drawn from assessment results are meaningful and accurate.
Jun 01, 2023 @ 10:57:13
Hi Doan!
You did a great description differentiating between criterion / domain referenced instruments and norm referenced instruments. One part that really stood out to me in the lecture videos was the idea of mastery. With the criterion / domain referenced instruments there is a specific cut off and if you meet that cut off then you have hit the predetermined score that is indicative of the level of mastery of a material. I like that you included the idea of goal setting. I could see if you had a norm referenced group instrument and were comparing a clients performance to the performance of the larger sample that you could set up goals with a client if their performance is far below the performance of the larger sample. In regard to correlation not equaling causation, I think you are spot on. Within the field of mental health many people do look for straight forward explanations to understand mental health problems. I find that when anyone has any type of problem (mental health, medical, physical) we want to understand the root and the “why”. Not having a straight forward explanation leads to ambiguity which people do not like. However, mental health is complex and there are SO many factors to consider. To find causation there would have to be repeated studies that control so many variables- that we may not even think of. Lastly, reliability must be determined and established to even consider validity. I think you did a good job at highlighting that. The results indeed would be erratic and unreliable without reliability. Finding reliability will be a great foundation to start with when looking at assessments.
See you today in class!
Jun 01, 2023 @ 11:52:48
Your description of the difference between criterion-referenced and norm-referenced instruments is thorough and accurate. Criterion-referenced instruments assess an individual’s performance against specific criteria or standards, while norm-referenced instruments compare an individual’s performance to that of a larger norm group. In addition to this. I must say Your understanding of the importance of selecting the appropriate assessment tool based on the intended purpose, such as goal setting or decision-making, demonstrates a comprehensive grasp of these concepts.
Your explanation of correlation not equaling causation is excellent. It is crucial to acknowledge and remember that a correlation between two variables does not automatically imply a causal relationship. The intricate nature of mental health issues often fuels a desire for straightforward causes, which can sometimes lead to the misconception that correlations indicate causation. It is essential to approach correlations with caution and recognize the need for rigorous research and evidence to establish causal relationships accurately. Your acknowledgement of the limitations of observational designs and the need for strong experimental evidence to establish causality showcases your in-depth understanding of this concept.
Furthermore, your understanding of the significance of reliability in psychological assessments is commendable. Reliability ensures consistent and accurate measurement, allowing for dependable results across different administrations, settings, and raters. Your recognition of reliability as a prerequisite for validity highlights its foundational role in ensuring meaningful and accurate conclusions from assessment results. Your grasp of these fundamental concepts is great.
Jun 01, 2023 @ 12:42:58
A criterion-referenced instrument compares an individual’s score to the established standard or criterion. This does not focus on the individual’s performance but on how that individual’s performance compares to the standard. A norm-referenced instrument focuses on comparing an individual’s score to other individuals’ scores who have taken the same test. Understanding the difference between the two instruments is important because it helps test-makers make the appropriate measurements in their testing to compare their samples appropriately.
Correlation does not equal causation. When we see two variables moving together, it is easy to assume they are correlated, that one is causing the other to move with it. The assumption that correlation equals causation is still very relevant because it is easy when you look at data that is moving together to assume that they are correlated even if they have nothing to do with one another. Just because ice cream sales go up simultaneously with shark attacks does not mean they are correlated. They have nothing to do with one another, and it is wrong to assume they do. The only way to measure correlation and causation accurately is by performing tests and getting accurate data to prove it.
Reliability is essential in psychological measurements as a reliable test is consistent and valid. In order to trust a test and its results, the test needs to be done consistently so we know that the results are accurate. We wouldn’t want decisions to be made for us based on a test with inaccurate and invalid results. We would want the test to be consistent and reliable so we can trust it and then trust the decisions based on it.
Jun 02, 2023 @ 14:05:00
Hi Julianna! I really liked your example about correlation between ice cream sales and shark attacks. Of course these two variables would not logically have the power of causality over one another, where for other variable pairs, it could be easier to make that mistaken assumption. This is why it is so crucial to perform further experimental study to gain a better knowledge of how the given correlational relationship functions. I also enjoyed the way you put the importance of reliability into perspective: we as a client would never want a professional to make treatment decisions based on an unreliable assessment. You made some great points!
Jun 03, 2023 @ 21:48:26
Hi Julianna. I totally agree that understanding the difference between the two instruments is important for assessment selection. Having knowledge of these two types of instruments helps in selecting the appropriate assessment tool for a specific purpose. Depending on the goals of assessment, different instruments may be more suitable. Understanding the distinctions allows practitioners and educators to make informed decisions about which type of assessment will best serve their needs.