Topics 1 & 2: Norms & Reliability {by 6/2}

Based on the text readings and lecture recording due this week consider the following three discussion points: (1) In your own words, provide a general description of the difference between criterion/domain-referenced instruments and norm-referenced instruments.  Why is it important to understand this difference?  (2) Correlation does not equal causation! Share your thoughts on why this assumption is still a common mistake, especially in the mental health field.  (3) Share your thoughts on why reliability is so (wicked) important for psychological assessments (this is a bit of a deep question – give it your best shot).

 

Your original post should be posted by 6/2.  Post your two replies no later than 6/4.  *Please remember to click the “reply” button when posting a reply.  This makes it easier for the reader to follow the blog postings.

38 Comments (+add yours?)

  1. NikkiAnn Ryan
    Jun 01, 2022 @ 22:52:16

    Criterion/domain-referenced instruments and norm-referenced instruments are important to consider when interpreting an individual’s performance on an instrument. Both instruments compare the individual’s performance with another point of reference, however, one compares the individual’s performance with the performance of others and the other compares it to a standard within the domain being measured. With a norm-referenced instrument, the individual’s performance, or score, is compared with the scores of other individuals who have taken the same instrument. Criterion/domain-referenced instruments differ because they assess whether the individual reached a specified level of performance in the domain being measured. It is important to understand the difference between these instruments because there are different implications for interpreting the results, for instance, statistics can help with interpreting the results of norm-referenced instruments.

    Correlations are used to describe relationships, but they do not indicate causation because there are countless factors that may influence the relationship between the variables being measured. This mistake may happen among individuals in the mental health field because the researchers likely selected the variables to study because they believe one of the variables influences the other. Therefore, if they then see a relationship between the two variables, they may interpret it as one variable causing the other, especially if that is a desirable outcome for their study. For instance, taking the example of a correlation where the level of depression decreases as the number of therapy sessions increases, someone may make the mistake of stating that the therapy sessions caused the reduction in depression especially if it confirms their beliefs or theories that therapy influences depression levels; however, they cannot evaluate all the other factors that may have also influenced the decrease in depression to be certain that the therapy sessions were the cause of the decrease.

    Reliability refers to the consistency of an assessment, whether it is measuring what it is intended to measure, and it is important for psychological assessments because psychological assessments are an integral part of the counseling process. Assessments are used throughout the counseling process from assessing the problems the client is struggling with to the treatment selection and the conclusion of therapeutic services. An individual’s performance on psychological assessments has important implications for determining diagnoses and treatment interventions, therefore it is important to consider the reliability of an assessment before administering it to a client because you want to use an instrument that will give an accurate depiction of what you are trying to measure, such as depression.

    Reply

    • Ashley Torres
      Jun 02, 2022 @ 22:05:26

      Hi NikkiAnn, it is very interesting to see how lots of people infer correlation equals causation. That is a good point you made about researchers using variables they think influence each other. They are more likely to be biased in their research, especially if the outcome is what they speculated. As a result the relationship between two variables might not be accurately presented which then leads to false information.

      Reply

    • Luz Rodriguez
      Jun 06, 2022 @ 16:59:54

      I definitely agree that a mistake can happen where you can see the two relationships between the variables and sometimes a person might interpret it as one causing the other not realizing that their are other variables that can take place. I agree with the example you used and I do agree with you.

      Reply

  2. Rachel Marsh
    Jun 01, 2022 @ 23:51:29

    1-) The primary difference between criterion -referenced instruments and norm-referenced instruments is that criterion-referenced instruments compare a client’s performance to a specific benchmark whereas norm-referenced instruments compare a client’s performance to the performance of other individuals (Whitson, 2016). Understanding this difference is imperative to ensure that assessments are being administered and interpreted appropriately, given the situation. Choosing a criterion-referenced assessment in a situation where a norm-referenced assessment may be more beneficial would prevent accurate interpretation of results and would likely impact subsequent decision making. Criterion-referenced instruments are more beneficial when analyzing what a person knows about a certain subject or how well they perform a certain task (Whitson, 2016). For example, a counselor could administer a vocational assessment to ascertain a client’s ability to perform work-related tasks. Alternatively, norm-referenced instruments are more beneficial when analyzing characteristics to see if an individual performs above or below a desired population (Whitson, 2016). For example, an admissions counselor to a graduate program could compare the verbal and quantitative GRE scores of an applicant to the university average to see if the applicant’s scores fall above or below students who are currently enrolled in a program.

    2-) I believe that people tend to conflate causality and correlation for several reasons. Firstly, because correlation is necessary to establish causation people may assume that if two or more variables have an established correlation that one caused the other (Morling, 2017). In my undergraduate research methods course, I remember having this discussion with classmates when we were learning about the requirements to make a causal claim which includes correlation. Our Professor always emphasized that while correlation is necessary to establish causality, it is not the only factor.
    In our discussion, our Professor gave us a website that displays spurious correlations to illustrate her point (https://www.tylervigen.com/spurious-correlations). One correlation listed on the website indicates a strong positive correlation between per capita consumption of mozzarella cheese and civil engineering doctorates awarded. The idea that eating more mozzarella cheese will increase your ability to receive a doctorate degree in civil engineering is silly. While causation requires an established correlation, it also requires temporal precedence and ruling out extraneous variables (Morling, 2017).
    Additionally, I believe that people tend to assume that correlation equates to causation because of cognitive ease. I remember discussing this topic in a research assistantship I did during my senior year of undergrad. The project my group worked on was a correlational study. Our Professor always emphasized that our study was correlational and therefore we could not infer causality. Our study focused on mindfulness as predictors of stress and wellbeing. With existing evidence that mindfulness interventions can increase wellbeing and decrease stress, it was always difficult to remember that our study was correlational and we couldn’t say that mindfulness caused wellbeing to increase and stress to decrease in the context of our study. It made a lot of sense that the variables we were studying could directly cause one to increase or decrease, but there were other factors that we didn’t account for, nor did we look at temporal precedence to make causal claims regarding our data. Overall, the way our brains work, I think it takes more cognitive effort to remember that correlation does not equal causation, especially in cases where a cause seems plausible.

    3-) I believe that reliability is essential to ensure the assessments being used are consistent in
    scoring. Counselors use results from assessments to inform decision making regarding treatment and diagnosis for their clients. A difference of even just a few points could completely change the decision-making process, the approach a counselor chooses, and possibly diagnosis. A lack of consistency in scoring, if severe enough, could completely impact diagnosis and subsequent treatment. This would likely lead to unethical and inaccurate treatment of a client. For example, if two therapists were giving a client a risk assessment at the same time in the same conditions, they would expect that the assessment would result in the same score both times (Whitson, 2017). But after the risk assessment one therapist concluded that the client was a danger to themselves and others, but the other therapist concluded that the client was safe. In reality, if the therapist was truly a danger to themselves but after the risk assessment was falsely designated as being safe, the proper steps would not be taken to prevent them from harming themselves or others. While this example is extreme, because of the cost of even a few points, it is ethically imperative for a counselor to ensure that the methods they use are reliable. Even if done inadvertently, this could put the client at risk for further distress, lack of trust, delay them from receiving proper services, and likely cause harm.

    References

    Morling, B. (2017). Research methods in psychology (3rd Ed). W.W. Norton & Company.

    Spurious Media LLC. (n.d.) Spurious correlations.
    https://www.tylervigen.com/spurious-correlations

    Whitson, S.C. (2016). Principles and applications of assessment in counseling (5th Ed). Brooks/Cole.

    Reply

    • Patricia Ortiz
      Jun 03, 2022 @ 13:37:40

      Hi Rachel,
      I like how you concluded that because of the way our brains work it takes more cognitive effort to remember that correlation does not equal causation, especially in cases where a cause seems plausible. This is so true because the brain works like a machine that makes predictions all the time and automatically looks for “cause and effect” relationships in things.

      Reply

    • Kiara Mark
      Jun 06, 2022 @ 10:32:52

      You bring up a great point with cognitive ease. Some people are afraid of the unknown so it is easier to believe there is a cause when in reality the cause is not known.

      Reply

  3. Rylee Ferguson
    Jun 02, 2022 @ 08:45:31

    In order to interpret a person’s scores on a norm-referenced instrument you need to compare their results to other peoples. Where they score relative to others determines how well they did. On the other hand, to interpret a person’s scores on criterion-referenced instrument you need to compare their results to a predetermined standard. In other words people, hopefully experts on the test topic, have to come together beforehand and decide what score is sufficient. I think it is important to understand the difference because it is better to measure certain things with one of these instruments and not the other. For example when it comes to getting a driver’s license you do not want to give it to a kid because he drove better than the average teenager who came in that day because they all could have been terrible. Instead you want to give a license because the kid meets pre-established criteria for what is considered safe driving.

    I think assuming causation from correlations is such a common mistake because it is easy. I think when the average person hears about findings they are not analyzing them deeply but rather using thinking shortcuts to understand them as simply as possible. Also in previous classes we have gone over how this mistake is made even in articles, likely because causation sounds more interesting and catches readers more than correlations. So it is not surprising that a lot of people still confuse these topics when it is easy to make and being modeled for them online.

    Reliability is trying to measure if an instrument has any errors or if it consistently evaluates individuals. I think this is particularly important in psychological assessments because so many factors can affect an individual’s score on psych related assessments other than what the instrument is intended to measure. It is important the instrument is not too warped by these other factors or else someone might get a diagnosis might get misassigned or not given where appropriate. This is a big concern for instruments in a field where so many potential diagnoses overlap and can run together.

    Reply

    • Ashley Torres
      Jun 02, 2022 @ 22:22:17

      Hi Rylee, I agree reliability is really important because if an instrument has errors then it will not do its job appropriately. Factors can include poorly made instruments, unclear instructions, the distributor’s tone of voice, and many others. If an instrument is reliable then each time the same individual takes the tool, they should score about the same. Of course life happens and events occur that could change an individual’s scores but that does not mean the instrument is unreliable.

      Reply

    • Rachel Marsh
      Jun 04, 2022 @ 12:27:00

      Hello Rylee,

      I like how you described the use of criterion-referenced versus norm-referenced instruments. Your example of drivers license tests highlights this very well! As you stated, it is essential for professionals to distinguish between criterion-referenced and norm-referenced instruments to understand the situations they are applicable to. As demonstrated with your driver’s license test example, this could bring up serious safety issues. If someone were to give a teenager a drivers license with a norm-referenced measure and the norming population that the sample was drawn from were all unsafe drivers, then it is likely that the teen receiving their drivers license based on that assessment isn’t a safe driver. Similarly, if you were to administer a risk assessment to a client with the concern they were engaging in harmful behaviors you would also want to use a criterion-referenced assessment. In a norm-referenced assessment, the client could be more or less likely to harm themselves or others based on the general population. But that isn’t beneficial if the norming sample tended to engage in higher rates of self-harm than the average. Having a specific criteria of what constitutes self-harm would be more helpful in promoting accurate assessment and subsequent intervention. Overall, great post!

      Reply

    • Sarah Kendrick
      Jun 06, 2022 @ 21:41:26

      Hi Rylee! I like how you acknowledged that other factors can affect a measure or the individuals’ score on the measure. I also agree with Ashlee and her examples of what these factors can be. I remember when I did my research class in undergrad and administered a measure to a sample of students, it was nerve-wracking trying to be as consistent as possible with the environment we provided! In regards to diagnosing an individual, I can only assume how much more difficult it would be to ensure this consistency and precision.

      Reply

  4. Luz Rodriguez
    Jun 02, 2022 @ 12:34:39

    The difference between norm- referenced instruments and criterion / domain referenced instruments and criterion domain referenced instrument is norm referenced instrument is when it’s making a reference how we are comparing scores to other people score or results that compare performance. Were criterion domain reference instrument being more of taking the scores and comparing to another score and measure as a test measurement of performance of the person or score. Norm measures more like specific skill or knowledge while criterion / domain reference measures more of a performance on a specific area or specific comparison of the group.

    When we look at correlation between to variables, we tend to look for similarities and a relationship between them or what will influence the other. We try to make a connection between them that one may cause the other, but it does not exist there are many things that can lead to one of variables it not just one specific cause brings to fact that there no proof that second variable. We always think that one must cause the other for something to happen. We always think that we must find a relationship between the two for it to have an effect. Example one used with kids in the school : if you eat breakfast your children do better on test results in school. Some may do well testing others may not , some due to being nervous about taking test, they don’t study and remember information being tested on there so many other factors that come with this not just one.

    Reliability is so important for psychological assessments because it give us a consistent result of assessing clients and test results don’t fluctuate or change significantly though out the testing time frame. So, when you are testing the individual becomes is more persistent with the pervious results with out a large change in information that collected. When you have reliability measured the quality is more persistent, so your assessment of the client is more accurate and useful to be used or expressed to the client’s trust. It brings a positive effect to the clients trust towards you and understanding them and provide change if it is needed.

    Reply

    • Patricia Ortiz
      Jun 03, 2022 @ 13:13:40

      Hi Luz,
      I agree with you that a “reliable assessment brings a positive effect to the client’s trust towards you and understanding them and provide change if it is needed”.That is so true! we should be aware of that. We have to make sure those assessments are reliable and appropriate and that they provide a piece of accurate information, so we can determine a proper solution to the problem of the client!

      Reply

    • Rylee L Ferguson
      Jun 05, 2022 @ 23:12:13

      Hi Luz, I liked your discussion of how people tend to want to believe a relationship is simple and one thing causes the other but that in reality other factors make it much more complicated. I think the example you used illustrated this well. I also found it interesting how you tied in rapport with reliability. I had not thought about how using faulty or outdated tests could impact how much a client trusts you and wants to work with you. Very insightful.

      Reply

  5. Ashley Torres
    Jun 02, 2022 @ 21:52:16

    Criterion/domain-referenced instruments and norm-referenced instruments are instruments used to measure performance. Norm-referenced instruments consist of a group of people taking the same instrument/assessment. Once the instruments have been completed, an individual’s score is compared to other individuals in the same group. On the other hand, criterion-reference focuses on an individual’s performance with a certain criteria rather than comparing their scores to the rest of the group. Criterion-referenced measures a domain and then those scores are analyzed to determine how much material an individual knows. It is important to understand the difference because they measure different things and it is crucial to administer the appropriate assessment based on the situation. Norm-referenced measures if an individual scores average, below average, or above his groups average. Criterion-referenced is more focused on the performance of an individual with a specific domain.

    Correlation does not equal causation and this assumption is a common mistake because people believe if two things are present at the same time, then one must cause the other. We find it very interesting when there is a relationship between variables because it gives an “answer” to why things occur. An example regarding mental health could be, the more time young people spend on social media the more they’re depressed. Here we are assuming social media is the cause of young people’s depression which is simply not true. We need to take inconsideration other variables like health, sleep, genetics, abuse, and more. Young individuals can spend lots of time on social media and be depressed but that does not mean social media is causing depression.

    Reliability is really important for psychological assessments because they need to accurately measure their subject. Reliability ensures consistency of an assessment in order to accurately measure what they intend. The assessment needs to measure a domain and each time an individual takes the assessment they should score the same, as long as major changes have not happened. If the individual is not scoring consistently then the assessment is not accurately measuring the specific domain, therefore the client will not receive an accurate diagnosis or treatment.

    Reply

    • Amanda Bara
      Jun 03, 2022 @ 11:28:49

      Ashley,
      You did a really great job explaining the difference between criterion/domain-referenced assessments and norm-referenced assessments. I agree that it is important to understand these assessments and that they are used in the right situations. Conclusions can be misleading or confusing if the right assessment is not used. I also wanted to highlight your comment on correlations and how people think that if there is a relationship that it is an “answer” to a question. There is a lot of influential factors that determine this so called answer that a correlation does not include. Nice discussion post!

      Reply

    • Tuyen Phung
      Jun 04, 2022 @ 10:00:22

      Hi Ashley,
      Your discussion provides clear answers to the questions. I like the way you explain the mistake in correlation. If people are not aware of the fact, it usually makes them believe that one causes the other. I did think that when I had not learned about the knowledge. Especially when I read any title referring to two variables. I think that many people usually have the mistake that they are not aware of. The mistake can occur in mental health practice when people think that one causes the other, and then, they try to solve one as a way to recover the other without paying attention to other factors. Overall, your post is clear and really makes sense.

      Reply

    • luz Rodriguez
      Jun 06, 2022 @ 17:14:23

      As we go through life we always taught one thing causes another, the fact that when we think of correlation doing the same thing because out thoughts are programed like this one thing causes another but there is so much more that is involved. I like the example you used with social media so many different causes for depression through media.

      Reply

  6. Tuyen Phung
    Jun 02, 2022 @ 22:02:25

    There are several differences between the two instruments. A criterion-referenced instrument is used to compare an individual’s outcomes or status to a stable standard while a norm-referenced instrument is used to compare an individual’s outcomes or status to other individuals in a group. In addition, a criterion-referenced instrument is used to evaluate the individual’s mastery of a specific outcome while a norm-referenced instrument is used to evaluate the individual’s rank in a group. Personally, I think that it is necessary to know the difference in assessment because it can help professionals determine which instrument should be used for evaluating a specific proficiency of individuals and which one should be applied to evaluate the level of individuals’ skill in a group.
    A mistake in determining that one variable causes the other variable in correlation often happens in understanding correlation. The mistake occurs because of several reasons. Specifically, two variables used in correlation usually relate to each other in some ways. Moreover, both variables can appear simultaneously, so it can cause confusion as causality in correlation. Furthermore, two correlated variables may appear closely and depend on each other. As a result, people tend to refer to one as a cause of the other. The mistake easily occurs in the mental health field because variables such as behaviors cannot be visibly measured like physical factors, which can be measured, weight, and seen visibly.
    Reliability is important in psychological assessment because it becomes a determinant factor of whether an assessment is accurate and appliable or not. If there is no consistency of measurement in a group of people with the same test and its repetition, people would not see it credential and trust to use it. Also, if an assessment is not reliable, people may face challenges to find accurate results related to the tested patterns. Like people believe in the effect of a specific kind of medicine because of the repetitive recovery of other people with the same disease, they will have more confidence in using a psychological assessment with the consistency of measurement.

    Reply

    • Amanda Bara
      Jun 03, 2022 @ 11:32:17

      Tuyen,
      You did a great job explaining these terms in your post. I liked how you said that reliability is important in determining if an assessment is applicable or not. There are so many assessments out there but it really is critical to use the ones that will make conclusions and help a counselor understand the treatment necessary for a certain individual. If we can not rely on assessments to make inferences on where individuals stand then we can not go forward in treatment. Nice post!

      Reply

    • NikkiAnn Ryan
      Jun 05, 2022 @ 09:56:55

      Hi Tuyen,

      Your discussion of the two types of instruments is clear and I like the way you highlight the importance of confidence in using a psychological assessment that is reliable. I also like that you discuss that behaviors cannot be measured the same way as weight for instance, and how this relates to the mistake people sometimes make in the mental health field when they interpret causation from a correlation. I think that because thoughts, feelings, and behaviors cannot be measured and tested in the same way that other fields use measurement impacts various aspects of the mental health field.

      Reply

    • Sam Keller
      Jun 07, 2022 @ 23:12:48

      Hi Tuyen!

      I really liked the point you made that factors we can’t see are harder to measure because they are internal compared to something external such as color or weight. I also like our point that seeing two things at the same time can make us think that there is a causal relationship between the two.

      Reply

  7. Amanda Bara
    Jun 03, 2022 @ 11:21:22

    Criterion/domain-referenced instruments look at one person’s score and compare it to a set standard or a cutoff score. Norm-based instruments compare one person’s score to a sample of other individual’s scores that have taken the same assessment. It is important to understand the difference between criterion/domain-referenced instruments and norm-based instruments in order to interpret the results of a score properly in its context. If an instrument is criterion/domain-referenced than the individual’s score is looking at what the individual knows and is not considering other people. Norm-based instruments see where an individual is at based on how other people score on the assessment.
    The term, “correlation does not equal causation” is heavily mistaken in the mental health field because of the strong relationships between variables concluded in research. People make conclusions based on relationships however they do not take into consideration the other factors and influences that may be surrounding these correlations. For example, an individual who committed suicide listened to a song about someone also committing suicide. People may infer that this song CAUSED the individual to go through with her suicide but there are so many other factors that are influential to one ending their own life. Although there may be a correlation between that song and someone’s suicidal actions it does not mean that it ultimately caused her to end her life. Other factors such as childhood trauma, depression, anxiety could be influential factors.
    Reliability is important for psychological assessments because it allows the individuals implementing/interpreting the tests as well as those taking the tests to be confident that it makes appropriate conclusions. Psychological assessments are an important part of the counseling process and allow counselors to identify a client’s strengths/weakness and progress throughout sessions. Having assessments that are reliable and show consistency as a testing procedure will allow for progress in treatment and confidence in making treatment decisions. Without reliable measurements, a counselor might be treating someone inappropriately and making decisions based on inaccurate data.

    Reply

    • Rachel Marsh
      Jun 04, 2022 @ 12:13:35

      Hello Amanda,

      I completely agree with your insights regarding how people often equate correlation with causation, especially in the mental health field. You bring up a great point when you say that the strength of relationships can influence how people are more likely to equate correlation with cause. When we find very strong relationships in research, it can be difficult not to argue that one variable caused another. As you brought up, other factors likely contribute to certain events. The only way to establish causality between two variables is to control for those external factors, which isn’t something we realistically can do in a clinical setting (Whitson, 2016). Overall, great post!

      References

      Whitson, S.C. (2016). Principles and applications of assessment in counseling (5th Ed). Brooks/Cole.

      Reply

    • NikkiAnn Ryan
      Jun 05, 2022 @ 09:54:29

      Hi Amanda,

      I think the example you used with the song “causing” someone to commit suicide demonstrates the importance of not using terms related to causation when it is really a correlation. You are right, there are so many possible underlying reasons and other influential factors that may lead someone to commit suicide that we cannot say that a specific song “caused” it. Additionally, I like the way you discuss the importance of reliability at the end of your post when you state that a counselor may be making decisions based on inaccurate data if they are not using a reliable measure. If a counselor is not using accurate information when they are making decisions about the treatment plan not only are they likely not helping the client, they may be doing more harm than good.

      Reply

    • Rylee Ferguson
      Jun 05, 2022 @ 23:17:09

      Hi Amanda. I liked how you highlighted the importance of context in the difference between norm and criteria referenced assessments. I also agree with the idea that having reliable assessments in psychology is critical to developing a proper treatment plan. I think it would be unethical to use assessments with poor reliability as it could lead to clients being misdiagnosed and even given unecesary medication.

      Reply

  8. Patricia Ortiz
    Jun 03, 2022 @ 12:52:31

    (1) In your own words, provide a general description of the difference between criterion/domain-referenced instruments and norm-referenced instruments. Why is it important to understand this difference?

    Criterion/domain-referenced instruments are designed to measure a person’s capacity to “meet” the expected standard or to be considered capable or qualified to perform a certain skill. For example: to achieve and accomplish specified learning standards.

    Norm-referenced instruments are made to “compare” a person’s capacity and scores to a selected group of people and report whether they did better or worse.

    While Criterion/domain-referenced instruments are designed to evaluate the level of achievement of a person, Norm-referenced instruments try to fit a person into a category.
    Criterion domain/referenced instruments do not compare a person’s performance to another person on the same task and Norm-referenced instruments do.
    Some examples of Criterion/domain-referenced instruments are driving tests, language tests, and academic achievement tests.
    An example of Norm reference tests is the SAT or IQ tests in which each individual is compared to another and categorized with a percentile.
    It is important to know the difference between them because it is critical for a counselor to know about assessments and their purpose to well assess a client (if an assessment is needed) either to get an insight or overview of the patient, such as their IQ, personality, level of achievement on a determined task, etc. Or make a treatment plan if that was the case.
    Also, we should know whether we need to compare our clients to the same group as them or just know what their mastery of a specific skill is.

    (2) Correlation does not equal causation! Share your thoughts on why this assumption is still a common mistake, especially in the mental health field.

    Correlation does not equal causation because the correlation between two variables not necessarily means that one variable causes the other. In the mental field, two variables could be correlated and one could cause the other, but also could be that some other factor is causing the variable to happen. This is a common mistake because if we see two things happening at the same time, we are used to thinking that one automatically causes the other. One example could be that when it rains it is more likely to thunder, but it is not the rain that causes the thunder. One example applied to mental health is when a person is depressed, we automatically think that they may have experienced some upsetting events that lead to the disorder when in reality depression can happen for a lot of factors such as -biopsychosocial factors- and not solely due to “upsetting events”.

    (3) Share your thoughts on why reliability is so (wicked) important for psychological assessments (this is a bit of a deep question – give it your best shot).

    Reliability is the consistency of the results in an assessment. This characteristic is important because reliability is what tells us the level of precision of the instruments used, and that the scores on a test are trustworthy and we can rely on them to make valid and coherent conclusions.
    This is so important for psychological assessments because in this field we need instruments and assessments that are reliable enough to be used in describing and evaluating people and making predictions about their behavior.

    Reliability helps us to select appropriate instruments either for the investigation or solution of a problem and these provide us with the most and best information about the problem. If the evidence we have is not reliable, the problem will not be properly settled.

    Reply

    • Tuyen Phung
      Jun 04, 2022 @ 10:17:41

      Hi Patricia,
      Your explanation of norm and criterion-referenced measurement, the mistake about the causality of correlation, and the importance of reliability in mental health practice are very clear. It makes more sense when you take examples and connect them to actual practice. I would like to focus on the measurements. I also think that each measurement has its own purpose and application. However, each measurement should take culture and context into account. Norm-referenced measurement should consider the variability of its takers and criterion-referenced measurement should consider their actual situations.

      Reply

    • Kiara Mark
      Jun 06, 2022 @ 10:42:48

      Your point about whether we need to compare our clients to the same group as them or just know what their mastery of a specific skill brings up a great point as to culture and ethnicity. It made me think about what would the best comparison be. Is it just people who took the same test or does characteristics like age and ethnicity play a factor?

      Reply

    • Jonas Horan
      Jun 06, 2022 @ 15:59:43

      Assuming that people who are depressed must have experienced something traumatic is an interesting example! I think that conflating correlation and causation happens a lot with our personal experiences. If we experienced a certain thing as a result of something else, its easy to assume that the causal relationship is the same for other people.

      Reply

    • Kat
      Jun 06, 2022 @ 21:20:54

      Hi Patricia! You are right, in psychological research it is always possible that one correlated variable caused the other in a study. However, it is much more about deciphering directionality when it comes to causation. Correlational studies assess two or more variables without following up with participants over a long period of time. Thus, it is impossible to decipher whether one variable caused the other in correlational studies.

      Reply

    • Sam Keller
      Jun 07, 2022 @ 23:16:22

      Hi Patricia!

      I really liked your point that the end result of an instrument not being reliable in our field is that a problem won’t be properly resolved. Because of the importance of our work this can have devistating consequences on the people we are trying to help.

      Reply

  9. Kiara Mark
    Jun 06, 2022 @ 10:26:56

    Criterion/domain referenced instruments are used to compare an individual’s performance to a stated criterion or standard. This instrument usually provides information on a specific knowledge or skill and whether or not the individual has fully learned said knowledge or skill. The results are also interpreted in terms of domain so the counselor needs to understand the domain being tested. An example, of this would be state testing. In most high schools in order to graduate you must pass state tests according to what they decide is passing. Norm-referenced instruments are used to compare an individual’s score to others who have taken the same test. For example, by using this instrument a student would be able to determine how they did on a quiz in comparison to their peers. Even though both instruments are used to measure performance they should not be used interchangeably. Each instrument needs to be used correctly so the data can be used in the correct setting.

    Correlation does not equal causation is a common mistake made especially in the mental health field. Correlations are used to examine the relationship between two variables. They do not indicate causation because there are various factors that can influence the relationship between the variables being examined. When looking at the data for a correlation people may assume that since the variables move together one causes the other. However in order to do this lurking variables must be ruled out. For example, there could be a positive correlation between ice cream sales with increased heat. Some one might say the heat causes people to get ice cream. If this were true then it would be a good idea to open an ice cream store near a sauna rather than just an area with hot weather.

    Reliability is important for psychological assessments. According to the Classical Test Theory reliability is the degree to which there was error in the instrument. In any result from an assessment device is a combination of a person’s true ability plus error. There are two different types of errors; random error and systematic error. Reliability does not measure systematic error only random error. An example of systematic error would be if there was a typo in a test and everyone taking the test reads that error. Random error is error that is not consistent, such as an examinee having a fever or test anxiety while taking a test. Their performance could be affected because they are not performing what is counted as their standard. It is important to have reliability in psychological assessment because it tells the counselor how well the assessment went based upon the individual. Counselors interpret the results from an assessment and determine whether someone needs treatment and how they should be diagnosed. If there is any inaccuracy in the assessment it should be retaken so the client is getting the best care.

    Reply

  10. Jonas Horan
    Jun 06, 2022 @ 15:53:44

    1) While criterion/domain-referenced, instruments compare results to a body of knowledge that determines the scoring, norm-referenced instruments compare an individual’s results to other people who have taken the same test or assessment. This is the difference between being graded against a set of correct answers on a multiple choice exam, and being ranked by percentile, such as on an SAT. Many norm-referenced instruments do not necessarily have right answers at all, such as personality inventories, and are simply a measure of a trait or tendency compared to others. These two types of instruments have different strengths and should be used in different ways. For example, in a classroom setting, a norm-referenced system of exams could result in many students being guaranteed a failing grade, regardless of if they had achieved acceptable mastery of the material. They would simply have scored more poorly than a certain number of classmates. Similarly a criterion/domain referenced instrument would not necessarily be appropriate when measuring more abstract constructs such as introversion. It is difficult to define such constructs independently from a comparison to other individuals, making a norm-referenced instrument more appropriate.

    2) Perhaps the reason why the assumption of causation after establishing correlation is so easy to fall into, is simply because the social sciences deal with such complex variables. Working with correlations can be frustrating at times, especially in the social sciences. The mechanisms of causation found in the social sciences are often so complex and diffuse, that correlation can be the only indication of a causal relationship. I also think that statistics tend to make claims feel more legitimate than they actually are, and are easy to abuse. We like to have our claims backed up with numbers, even though numbers don’t always mean what they want them to mean.

    3) The problem of reliability (along with validity of course) is one of the defining problems of psychology. Because psychology must legitimize itself as a science, developing reliable instruments is essential. However, the complexity and subjectivity of psychological phenomena makes this a difficult task. There is no blood test to verify if someone has depression or another mental illness. Thus reliability in psychology isn’t just a mechanical question as it is with physical instruments, it’s a question that can only be solved statistically.
    Obviously, if we didn’t have reliable instruments, the field of psychology would be seriously impaired, both from a theoretical standpoint, and a practical one. Reliability elevates psychological measurements above subjective judgment, and establishes some level of objectivity. While clinical judgment is still significant and necessary, reliable instruments give clinicians more information, while also rendering more accountable for their diagnoses and treatment results.

    Reply

    • Sarah Kendrick
      Jun 06, 2022 @ 21:25:23

      Hi Jonas! I like how you stated that “statistics tend to make claims feel more legitimate than they actually are, and are easy to abuse. We like to have our claims backed up with numbers, even though numbers don’t always mean what they want them to mean.” I don’t think I conveyed this as well but outside of lack of appropriate knowledge for the wording used, I do think we see what we want out of the results. How articles are published is also a big problem in that they want people to read them and will make results sound more conclusive just to get the readers’ attention.

      Reply

  11. Kat
    Jun 06, 2022 @ 21:01:21

    1. A client’s scores are compared to other individuals’ scores who have all taken the same instrument in a norm-referenced instrument. The group of individuals whose scores are later compared to a client’s scores is called a norm group. This type of group can be large or small. Contrastingly, a criterion-referenced instrument does not test whether or not a client compares to a group of individuals. Instead, a criterion-referenced instrument looks at how the client compares to a predetermined standard in a specific domain. In addition, in criterion-referenced instruments there is another predetermined score which shows whether a client has reached a level of mastery.

    2. I think that there are a lot of reasons why people make the mistake of believing correlation is causation. For starters, everyone has hypotheses that probably include some form of directionality or causation. Therefore, although correlation only means that two variables have been connected or shown to have a relationship, it can be easy for people to jump to predetermined conclusions.

    3. Reliability is crucial for psychological assessments to ensure that whatever is being analyzed is accurately testing the variable of interest. It is important to decipher what is true variance versus error variance. There can also be a lot of unsystematic errors in psychological assessments specifically where a client might answer a certain way based on what they think the assessor would want. Thus, since reliability measures unsystematic errors, it’s important to consider in psychological assessments.

    Reply

  12. Sarah Kendrick
    Jun 06, 2022 @ 21:01:55

    Criterion/domain-referenced instruments refer to measures in which individuals’ scores on this measure are compared to an already established standard within a specific domain. Norm-referenced instruments refer to measures in which individuals’ scores are compared to other individuals’ scores who have taken the same measurement (the norming group). It’s important to understand the difference between these instruments as one may not accurately interpret the results of these measures. Thinking back “fondly” on my MCAS years, this state standardized testing is an example of a criterion-referenced instrument. While I won’t pretend that I understood the scoring back then outside of whether or not I was “proficient,” I now understand that my scores were compared to a set standard for Massachusetts for each domain or subject (Biology wasn’t my best…). My scores/performance were not compared to others’ scores like in the general classroom setting rather, my performance was compared to the set standard.

    I think people continue to mistake that correlation does not equal causation for a few reasons. It’s very difficult, at least in the mental health field, to actually prove that X causes Y. I think that in correlations, people may get too excited about their findings and want to say that it’s a causal relationship. I also think that it’s easy for people to simply assume that X causes Y, without thinking about the other variables or evidence and that it’s easy to assume causation if there is correlation. It’s easy to believe perhaps that drinking five 2-liter bottles of soda a day may lead to tooth decay. For someone who does not understand the specific language of the mental health field, they may read this as drinking that much soda a day causes tooth decay. Without evaluating all of the other variables (an individual’s overall oral hygiene/hygiene practice, what else their diet consists of, etc.), people may be quick to jump to a causal conclusion.

    Reliability is so important for psychological assessment because it indicates the consistency of scores of a measure as well as determines if a measure is free of error. When attempting to measure something, it is important that the instrument you are using is precise across the items within the instrument as well as how the results of the instrument vary across multiple tests. If a measure is inconsistent and the results are different after repeated tests, then it’s not a good measure as you cannot trust the results. If you’re trying to measure something such as depression, it would be difficult to determine if an individual has changed (minimally) if their results differ even after being tested twice in a week.

    Reply

    • Kat
      Jun 06, 2022 @ 21:12:06

      I like what you said about why reliability is so important. It is definitely always important to ensure that there are consistent results regarding an instrument. I also think it is important especially for psychological research because in psychological research there is never a perfect score of reliability. A score of one is a perfect score for reliability, but psychological research never reaches that and can only come close in well orchestrated studies and assessments.

      Reply

  13. Sam Keller
    Jun 07, 2022 @ 23:06:00

    Criterion-referenced instruments are where a test judges performance using a set of predetermined standards. A common example is grades in school. There is a predetermined grade percentage point where if you get above it you pass and if you get below it you fail. Another example is getting your driving license. You can’t get partial credit, either you know enough to be safe on the road or you don’t. Norm referenced instruments are measuring participants based off of a scale or score gathered by testing a large pool of people. It is basically comparing the person taking the test to a bunch of other people taking the same test. It is important to be aware of in psychological testing because it is the difference between saying ‘you have more than 5 traits out of 9 for major depressive disorder, therefore you make the cutoff for a diagnosis’ and ‘compared to a group of your peers you have significantly higher depression scores’. One is based on others while the other is based solely on the person’s individual traits.

    Correlation does not equal causation is the entire argument Dr. Doerfler was making that there is no proof that depression is caused by a serotonin deficiency. People see that SSRIs work for depressed people and assume that because of this the cause of the depression was the lack of serotonin. However the lack of serotonin could be caused by other environmental factors such as having a terrible day job.

    Reliability is king (or queen) in testing because you want to make sure that if you test a person 100 times you will get as close to the same result as possible. If the trait you are measuring is a constant and your test is giving you variable results that means that there is a flaw in your test. If I have a 100 lb weight and put it on a scale 5 times and get 40, 110, 89, 56, and 201 then I can’t trust the scale. When we are working with important factors such as people’s mental health you really need to be able to trust the scale.

    Reply

Leave a Reply to NikkiAnn Ryan Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Adam M. Volungis, PhD, LMHC

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 75 other followers
%d bloggers like this: