Topics 1 & 2: Norms & Reliability {by 5/24}

Based on the text readings and lecture recording due this week consider the following three discussion points: (1) In your own words, provide a general description of the difference between criterion/domain-referenced instruments and norm-referenced instruments.  Why is it important to understand this difference?  (2) Correlation does not equal causation! Share your thoughts on why this assumption is still a common mistake, especially in the mental health field.  (3) Share your thoughts on why reliability is so (wicked) important for psychological assessments (this is a bit of a deep question – give it your best shot).

 

Your original post should be posted by 5/24.  Post your two replies no later than 5/26.  *Please remember to click the “reply” button when posting a reply.  This makes it easier for the reader to follow the blog postings.

41 Comments (+add yours?)

  1. Jennifer Vear
    May 19, 2021 @ 19:50:56

    1. Criterion/domain-referenced instruments are instruments where a score is not compared to another’s performance but are compared to an established standard. That is, testing what the test-taker knows and can do instead of how they compare to others. Norm-referenced instruments, on the other hand, are instruments where a score is compared to other test-takers/norm groups. This determines where a test-taker stands in relation to a normative sample.

    This difference is extremely important because the criterion/domain-referenced instruments will tell an individual about themselves, making it solely about them, not in reference to anyone else. For example, how one person scores on their personality test should not have any weight or significance in determining how another individual might score on their personality test. The test uses a standard for what a certain personality looks like and that is the only comparison that will matter to determine where an individual will fall.

    Norm-referenced instruments rely on where an individual falls in relation to their peers or a norm group. This is important for determining, for example, whether or not a child is learning enough in second grade in order to pass onto third grade. If this child scores significantly lower on their evaluation than the other children in his grade and age range, then this child might need to stay back and no move onto third grade, or they might need intervention to determine where they are in their learning process.

    2. As we know, correlation does not equal causation. However, especially in the mental health field, people seem to make this mistake frequently. For example, one might be able to make the correlation that their medication made their anxiety symptoms decrease. However, this correlation does not equal causation. The decrease in anxiety symptoms could have also been caused by that individual leaving a toxic relationship, slowing down on their caffeine intake, or learning and using breathing techniques. Another example would be that someone can make the correlation that their significant other left them because they did not know how to cook. This does not equal causation because the individual could have left their significant other for a number of reasons other than the fact that they could not cook. These two examples could have a number of possibilities and a number of correlations. However, that does not mean that the one correlation is the end-all-be-all reason for causation.

    3. Reliability refers to the consistency of a measurement. In various aspects of life, especially the mental health field, consistency is important due to a number of factors. For one, if someone is taking the BDI test for their depression and their results and the administration are not consistent in the delivery or the outcome, the score of the patient will not be valid or have any significance. In order for that test to be significant, it needs to tell something about the patient. However, if it does not consistently or accurately test for depression, then it will not be a useful tool in assessing the patient’s level of depression for treatment. Also, if an individual is taking a test that does not pertain to their age, gender, ethnicity, or nationality, then the results would not accurately depict the appropriate response for that individual, and will therefore not help them. Finally, reliability is important for test-retests because if the tests are not consistent in their instrument, practice effect, and time that is given, then the reliability of that test will decrease.

    Reply

    • Yoana Catano
      May 23, 2021 @ 06:09:33

      Hi Jennifer, I like the example you gave for causation as a common mistake in the mental health field, I am wondering if a linear thinking is easier for people to explain problems, specially related to the medical field where medication solves specific symptoms, like a headache, and it can be associated to emotional pain. In counseling is very usual to find us helping people to understand when a problem is correlated without meaning causality.

      Reply

    • Katie O'Brien
      May 23, 2021 @ 21:01:53

      Jennifer,
      While reading your correlation/causation response and the example of taking anxiety meds and seeing a decrease in symptoms, I tuned into how you mentioned the other factors that might contribute to the decrease in symptoms – lowering caffeine intake, breathing techniques, etc. At least in personal experience, I find that in a lot of cases, the solution isn’t to jump to medication and do nothing else. Often times clients would likely be encouraged to also begin things like meditation and adjusting how much coffee they drink, probably even concurrently with beginning their meds. Effective therapy may involve a combination of all of these things – saying one truly “causes” causes improvement would be difficult, as meds and those techniques probably work together to make an improvement. The degree to which these factors “work” may also differ person to person.
      Intersting point!

      Reply

    • Lindsay O'Meara
      May 26, 2021 @ 16:37:25

      Hi Jennifer,

      I agree with you that there can be many external factors that might contribute to beliefs that are generally not proven. Correlations are helpful, but definitely lack the evidence to support their claims. Various internal, external/environmental and situational experiences need to be explored as well.

      Reply

  2. Morgan Rafferty
    May 21, 2021 @ 09:50:11

    1.) A criterion/domain-referenced instrument does not compare the test-taker’s results to other individuals’ results. This type of instrument compares the test-taker’s results to an established standard.
    A norm-referenced instrument compares the test-taker’s results to other test-taker’s results. Individual performance is compared with the performance of other individuals who took the same test.
    It is important to understand this difference because results might be interpreted differently for each of these two methods.
    For the criterion-referenced method, a test-taker can discover how they performed in relation to others. For the norm-referenced method, a test-taker can discover if they have mastered the topic or have fallen short.

    2.) It is so tempting to assume that when two variables are statistically significant correlation exists between two variable, that one variable CAUSES the change in the other. This is a major RUH-ROH as Dr. Volungis would say. 🙂
    If a correlation exists between use of expensive skin care lotion and fewer facial wrinkles, we are tempted to assume use of lotion causes fewer wrinkles. We cannot make this assumption. An experiment is the only way to show a causal relationship exists. The individuals who use the expensive skin care lotion might also drink more water than the average person; consume a highly nutritious diet; exercise more than the average person; take expensive vitamins, etc etc,
    In terms of mental health, we might assume that someone’s life circumstances are causing their disorder. There is a correlation between low SES and rates of schizophrenia. Does low SES cause schizophrenia though? Or do schizophrenics wind up more likely to be lower SES as a result of inability to earn a good income or work at all?

    3.) Reliability is so wicked important when it comes to psychological assessments because it would be horrendous to inform someone that they suffer from a mental disorder when they really do not. If these types of assessments were not reliable, mental health providers would be unable to successfully gauge improvement/decline in a client’s status. The assessments would be useless. What is the purpose of administering an assessment that is not reliable? The risk of providing inaccurate results is enough to not bother with the assessment at all.

    Reply

    • Yoana Catano
      May 23, 2021 @ 05:57:54

      Hi Morgan, it is important to add to the last question that the difference between reliability and validity is that while reliability is the consistency that shows an instrument when is applied repeatedly to an individual or group (consistent in time), validity is the evidence that provides an indication that the instrument measure what is it intended to measure (evidence of their purpose). Reliability is the precursor of validity but if an instrument has reliability it doesn’t mean it has validity.

      Reply

    • Frayah Wilkey
      May 23, 2021 @ 14:25:11

      Morgan,
      I think that your reply to #2 is really comprehensive and I appreciate that you’ve brought up so many examples. It’s easy to fall into the trap of assuming causation especially when we’re dealing with the human services. I also like that you brought up the example of skin care- companies frequently use specific wording so that consumers assume causation. I know that I did before I had a better understanding of stats and experimental research!

      Reply

    • Valerie Graveline
      May 23, 2021 @ 18:53:59

      Hi Morgan!
      I really enjoyed your example about correlation/causation and expensive skin care lotion/fewer wrinkles. I think this example really showcases why people everyday find it so easy to believe correlation = causation, because they want a quick-fix and might not realize that there are so many other factors that contribute toward- in this case- fewer wrinkles. It’s unfortunately something we run into constantly when trying to buy anything, always being bombarded with reviews that seem to back up this idea of correlation = causation. I know for myself it’s hard to get away from that since it’s so ingrained in our daily lives. Great example!
      Valerie

      Reply

      • Morgan Rafferty
        May 24, 2021 @ 09:25:36

        Thanks Valerie. I appreciate your feedback.
        I just caught wind of a “correlation but NOT causation” example on the news this morning. Scientists were reporting about a very small number of adolescents who have recently experienced heart inflammation after receiving the COVID vaccine. I appreciated the news anchor clarifying the report by saying: it cannot be said at this point that the COVID vaccine CAUSED heart inflammation in these young teens; further observation/data collection is necessary before that is determined. The number of teens with heart inflammation might have experienced this physical ailment anyway; without having gotten the vaccine. The timing post-vax is suspect enough to look into it further though. (but again, it’s like a dozen teens….very very small number).

        Reply

    • Lindsay O'Meara
      May 26, 2021 @ 16:40:30

      Hi Morgan,

      A misdiagnosis going off of unreliable testing is such a scary thought! I agree with you that reliability is such an important factor in psychological testing. Having consistency ensures that we are taking the appropriate steps with clients.

      Reply

  3. Morgan Rafferty
    May 21, 2021 @ 09:53:05

    edit #2.) It is so tempting to assume that when a statistically significant correlation exists between two variables, that one variable CAUSES the change in the other.

    Reply

  4. Kaitlyn Tonkin
    May 22, 2021 @ 12:08:09

    1. With criterion-referenced instruments, an individual’s score is compared to an already established standard. Examiners using this test are interested to see how the individual scores against a standard. An example of a criterion-referenced instrument are the AP tests. Conversely, norm-referenced instruments are a type of test that compares an individual’s score with other scores of the same test. The individual’s performance is compared to everyone else’s performance on the same test. The other’s test scores serve as a reference point or the norm for the examiner to compare against. It is important to understand the difference between these two tests because there are some measures that should not be compared to others and some tests that should not be compared to a standard. Additionally, criterion-referenced instruments have a mastery component to them, meaning this type of test is oftentimes used to measure level of mastery (which I think can be problematic, but that’s a different discussion). However, testing against a norm is useful for determining if a child is falling behind in the classroom, or even excelling. It can show useful information to clinicians and teachers about where students stand compared to one another. This can be helpful if a child has an undiagnosed learning disability. Without a norm to compare to, it would be difficult for teachers, parents, or clinicians to see what was going on. I think this can also be important for diagnosing a client with a mental health disorder.

    2. Correlation does not equal causation – this has been drilled into our brains for as long as I can remember, yet people still jump to conclusions and operate under the assumption that just because something is related, it means that one caused the other. I think this is such a common mistake still, and even in the mental health field, because it requires less testing and it’s easier to make a blanket statement. For example, a child may have experienced a traumatic event in their childhood and now is a rebellious teenager. One might jump to the conclusion that the traumatic event has caused the child to be a rebellious teenager when in reality, there are many other variables that come into play. Examples include how the child was treated after the event, who the teenager spends frequent time with, how they are being treated at home currently, and so on. All of these other variables make it difficult to come to a conclusion, so since it’s easier to just say the traumatic event caused the rebellion, people tend to lean towards that. This can clearly cause issues and lead to improper diagnoses and treatment which can be deadly.

    3. Reliability is the consistency of measurements when a test is repeated – does the test produce similar results each time it is tested? Reliability is important in psychological testing because it is important that depression, anxiety, ADHD, schizophrenia, etc. tests consistently measure what they are meant to. It would not be good if someone was to go see their doctor or therapist and be given a test that does not always measure for anxiety. The problem here is that they would not be able to be treated properly. Psychological testing, as we talked about during our first class, is useful in measuring the client’s progress and determining what treatment is best for them. However, this cannot be done if clinicians are not equipped with reliable testing measurements.

    Reply

    • Jennifer Vear
      May 22, 2021 @ 12:42:31

      Hi Kaitlyn!
      You make a really great point in your explanation about how the correlation between rebellious teenagers and a traumatic event in their life does not mean that one caused the other. There could be various factors as to why people make certain decisions in their life and it is not always one reason as to why it happens. With social media, we live in a society where people are very quick to make assumptions on various topics and about other people without really getting to know them. I believe that this has a lot to do with the fact that it is human instinct to not like things that are unfamiliar, odd, or unknown. So, we bridge that gap by making assumptions about people so we feel a bit more comfortable or so that we at least have some sort of explanation without having to really get to know that person. As future therapists, we will be able to bridge that gap with our clients and really get down to the nitty-gritty of their lives and the experiences that they have been through. This concept is so important for us to learn so that we are more open and do not make basic judgments about people based on little pieces of information and little knowledge. Learning this concept is truly a huge part of learning to be more open-minded!
      Great example!

      Reply

    • Katie O'Brien
      May 23, 2021 @ 21:11:49

      Kaitlyn,
      The dreaded AP tests!
      Something in your post that I think is important is how a norm-referenced test might indicate to a teacher that a student is falling behind in relation to their classmates, possibly indicating a learning disorder. I think the next step in then diagnosing that disorder might then be criterion-referenced instrument, to see further if there is a significant issue. For example, if the child is behind in class relative to classmates, but overall performs adequately on the criterion-referenced assessment – maybe not a learning disorder, but the program / school may be too “advanced” or quick moving for the student. Maybe that student is better served by a slower-progressing program. On the other hand, the norm-referenced instrument scores might lead to a criterion-referenced assessment that supports the theory of a learning disability, if the student cannot meet the standard.
      It’s interesting to think of how the two types may be used together to fill in blanks that the other type might not cover.

      Reply

  5. Yoana Catano
    May 23, 2021 @ 05:31:16

    1. The difference between criterion/domain-referenced instruments and norm-referenced instruments, lies in what is the score compared to. A norm is a group of individuals that have taken the same instrument and the individual’s score will determine what percentage of percentile is the performance of the individual in comparison to the norming group. For example, what is the age of sexual debut or onset of alcohol use to determine risk in problematic behaviors; it compares an individual response according to a group previously identified.
    On the other hand, when the score is compared with a criterion, it means that a stablished standard, domain or objective has been defined and the individual can reach the standard or master the level according to what is indicated in the test. For example, a test for a social worker license is designed to measure the minimum competency acceptable to practice social work and it will give the examiner the idea of how an individual’s performance is compared with a stablished knowledge or practice skills.
    A criterion-referenced instrument is highly used, however there are questions related to how to define a criterion in a performance if there are no universal agreements in certain areas or how to determine if a person didn’t get the mastery level due to other circumstances or if there is a false positive.
    It is important to understand the difference because to compare scores with a norming group could only give the individual an idea of how their performance should be in that specific group, but a criterion-based will give a different result in the same performance, comparing with themselves to an objective matter. A kid could be “very smart” in a classroom when the level of cognitive performance is low, but they could be just normal if they are tested with a National Assessment of Educational Progress.
    _____________
    2. Correlation and causation could be mistaken in mental health, because causation involves correlation but not the other way around and human behavior is usually multicausal and complex. Lacasse & Leo (2015) provided an example about how an advertisement for Fluoxetine (Prozac) used a study about depression, to explain that depression appears when the serotonin drops, that way, they will demonstrate their medication will help with depression by bringing the serotonin levels back to normal. This made the authors of the study to start a fight with pharmaceutical companies about that the information they were proving was not accurate. The chemical imbalance they were using commercially was a metaphor according to the authors, and many clinicians and patients convinced of this explanation mostly because of the causal relation is easier to understand and probably to sell to people.
    It is also easy to think the depression is the causal of suicidal actions, but the possibility of having suicidal action from other causes like hallucinations or religious reasons, makes this category more correlational than causal.
    _____________
    3. In counseling, a proper assessment will facilitate the treatment choice; the right instruments used to evaluate are important, according to what we want to measure but most important is that the test that we use, will perform what it’s supposed to, and that it can be repeated with no much variability. If we get the same result repeatedly, this is determined by the reliability as one of the instrument’s psychometric qualities.
    Reliability is based on true component and error component, according to Whiston (2013) no one has been able to develop a perfect instrument in the field of psychological and educational assessment, thus it is important to include a percentage of error in a test, that could be indicated by factors like emotional distress in the individual or irregularity in some questions, and will facilitate to provide certain clarity about how to interpret the results. That is why reliability is so (wicked) important for psychological assessments, it is necessary to know the percent of reliability (correlation and consistency) to provide the most accurate hypothesis and therefore the most appropriate treatment.
    To estimate reliability test re-test can be used, alternate or parallel forms and measure of internal consistency. Reliability ranges from 0 to 1, being 1 the perfect consistency, thus, if an instrument has a high degree of error, it doesn’t have much practical value and it would no help clinicians to decide if the instrument is appropriate to use, also it brings the necessity of validity, even when it is not determined by the reliability.

    References:
    Lacasse, J. R., & Leo, J. (2015). Antidepressants and the chemical imbalance theory of depression: A reflection and update on the discourse. the Behavior Therapist, 38, 206-213
    Whiston, S. C. (2013). Principles and applications of assessment in counseling (4th ed.). Belmont, CA: Brooks/Cole.

    Reply

    • Frayah Wilkey
      May 23, 2021 @ 14:29:15

      Yoana,
      Thank you for giving such an in depth explanation of reliability and why it’s important. I know that you have a lot of experience in the field so I appreciate your perspective on how it impacts therapeutic treatment. Without reliable results, we can’t really trust that the treatment is effective or the right choice.

      Reply

    • Kaitlyn Tonkin
      May 23, 2021 @ 19:48:20

      Hi Yoana,

      I really liked the point you made about how the two different types of instruments could test the same thing, for example, a student’s intelligence, and produce the same results, but mean two different things. That was something I hadn’t thought about, and I feel like your example outlines it perfectly! I also think this example is quite relevant and something that happens often, especially in classrooms. Thank you for this insight!

      Reply

    • Angela Petrella
      May 25, 2021 @ 12:03:02

      Hi Yoana! I love the examples you provided in your explanation to help better understand all of the information we have been learning. Your response was super helpful for me to read and I feel like I have a much better grasp on this topic now!

      Reply

  6. Lisa Andrianopoulos
    May 23, 2021 @ 12:54:22

    1)Criterion reference and norm based instruments differ by how individual performance is compared. In a criterion/domain-referenced instrument, performance is compared to a predetermined criterion or standard. Tests of mastery are an example. For norm based instruments, individual performance is compared to other people’s performance on the same instrument. Intelligence tests are an example of norm based instruments. It is important to understand this difference because fist, it goes to how you interpret your results. A score of 80 on a criterion referenced test is very different from a score of 80 on a norm referenced assessment. The examiner must know the difference in order make a proper interpretation. Second, when selecting an instrument you will need to know which type of assessment you will need in order to properly evaluate what you are looking to assess.

    2)Correlations describe the strength of a relationship between two variables. It is NOT, A is the direct cause of B. Rather, it gives us a sense of whether if A occurs, is B more or less likely to occur beyond a random chance. This does not apply causation. Particularly when a correlation is high however, (e.g, greater that .90), a lay person (i.e, one not versed in interpretation of reliability coefficients) may infer causation and then restate it as truth. We are a society that jumps to determine causes, perhaps as a method of trying to figure out how to protect ourselves. Determining a cause is perhaps a natural self-defense. However, it is extremely rare to find a perfect correlation (1.0). In mental health in particular, mental disorders tend to be on a continuum and there is much overlap, confounding variables and variables that work in concert with each other. Typically there is no single cause. Nevertheless, individuals are constantly looking for an explanation or even a place to put blame. It can be easy to look at a statement, summary of research or a lay person’s explanation and think you’ve figured it out! This is dangerous because it can lead to false assumptions, misinformation and/or improper treatment.

    3)Reliability refers to the consistency of an instrument. For a test instrument to have good reliability, an individual should score similar on a 2nd administration of the same instrument (given after a short interval). This is important when looking at things like the effectiveness of an intervention or general change over time. If a test is not reliable, then how do you know that any change in scores is a result of intervention/progress (or regression) or just fluctuations in the test itself? Without good reliability, the examiner loses confidence and credibility in the interpretation of his/her results. This has implications for research results, treatment outcomes and informing treatment moving forward.

    Reply

    • Valerie Graveline
      May 23, 2021 @ 19:58:45

      Hi Lisa!
      You make a great point about how determining a cause may be a self-defense mechanism ingrained in us as a way to protect ourselves. With this said, your point about people constantly looking for some kind of explanation made me think about how sometimes people attempt to self-diagnose in an attempt to put a label on “what’s wrong with them”. I feel like this happens frequently with mental health, but also with everyday health problems where someone might look up their symptoms and “diagnose” themselves as having something potentially extreme simply based on what they read on the internet. I think people might find some relief in finding a name for their issue but unfortunately it can lead to, as you said, false assumptions and misinformation.

      Reply

  7. Frayah Wilkey
    May 23, 2021 @ 14:21:06

    1. Norm-referenced instruments refer to the practice to comparing a person’s score to the scores of other people who have completed the same measure. It’s simply comparing one person to the ‘norm’ which is based off others’ results. Criterion-referenced refers to the practice in which an individual’s score is compared to a pre-established standard- others’ results have no bearing. I think it’s an important distinction to understand because it can change the scope of how we’re looking at someone’s results and how the measure is administered. As the text mentions for example, in a criterion-referenced instrument, the person reading the results needs to have an understanding of what domain was being measured for the test. Then for norm-referenced, we should be asking what group was used to create the norm? These components can alter how results are viewed and therefore be impactful on the individual so we have to know beforehand what the difference is.
    2. To the general public, I think that the relationship between correlation and causation can be confused. Until I took a statistics class in undergrad, I would often assume things about results of studies or stats I saw on the news which I think can be largely attributed to an error that many of us make. It’s easy to infer causation, especially with certain wording of results. It’s perhaps even possible that results are phrased in such a way to suggest correlation=causation because it could further the research goals in some way. But I think a lot of people just assume unconsciously because that’s where our thought processes naturally go, so it’s important to untrained ourselves of that and to delve deeper into results and question them.
    3. I think that reliability is one of the most important facets of the application of psychology. If we’re not getting consistent results, then it seems that therapeutic efforts would be futile. Reliability allows us to ensure that treatment plans are informed and that any changes made are based on reliable results. Any scientific based practice requires reliability to be credible and useful.

    Reply

    • Sergio Rodriguez Pineda
      May 24, 2021 @ 01:04:32

      Hi Frayah,

      You pointed out a crucial role of reliability in psychology, and it is evidence-based practice. Because, commonly, therapists use any questionary or test to show a number, a diagnostic, or a result in their jobs but do we really make a difference when we present results based on an inconsistent test? How those nonreliable assumptions will impact a client’s life?. Thanks for outlining this!!!

      Reply

  8. Valerie Graveline
    May 23, 2021 @ 18:42:10

    1) Norm-referenced instruments compare an individual’s performance directly with other’s performances, whereas criterion-referenced instruments compare an individual’s scores to a consistently-set standard. The main difference between the two instruments focuses on what the individual’s performance is being compared to. With norm-referenced instruments, the interpretation of an individual’s performance is reliant on the population of which the “norm” is based on. With criterion-referenced instruments, there is a “mastery” component that is reliant on an assumed universal-understanding of what constitutes “mastery” in a specific domain. It is important to understand the difference between these two instruments because with comparing an individual’s performance to other’s scores, it is essentially measuring whether the individual is scoring below, above, or equal to the average of a specific population. In any given population, the average scores may not be considered “mastery” as criterion-referenced instruments focus on. With this said, it is very different to be comparing an individual’s scores to the “average” versus assessing whether or not they mastered a particular domain.

    2) The assumption that correlation equals causation is a common mistake that society is drawn to, perhaps because it is the easiest possible answer. In the mental health field, the idea that correlation equals causation is especially something to avoid because for any given client, there are a multitude of factors involved with their mental health and treatment at all times. To serve as examples, there is not one single factor that causes an individual’s anxiety, there is not one treatment alone that is leading to a decrease in an individual’s depression. In the mental health field, treatment involves an integration of techniques, skills, medication, etc. Similarly, with an individual’s issues being addressed in treatment, it is incredibly unlikely that there is one single cause to their issue and once that issue is removed, everything will be fixed. Correlation equals causation is the easiest possible answer and is essentially saying there’s a “quick fix” to any given problem, but unfortunately it is not that simple.

    3) Reliability is crucial to psychological assessments considering if there is no reliability, then the validity of the measure has no real standing. Aside from the idea of validity, in the mental health field it is important for the psychological assessments utilized to have good/high reliability since the results of some assessments could potentially impact a client’s life. With that in mind, there is little room for error. Also, reliability is necessary in order to successfully track a client’s progress. If the measure the client is taking gives them a completely different result every time, then there is no way to properly track what progress is being made.

    Reply

    • Kaitlyn Tonkin
      May 23, 2021 @ 19:52:47

      Hi Valerie!

      I think you are absolutely right when you say that people are drawn to saying correlation equals causation because it is such an easy answer. I enjoyed the point you made about there not being a single cause to someone’s anxiety, depression, etc., and consequently, there is not a single fix. This is a big misconception I feel like a lot of people have about therapy and any kind of mental health treatment, especially if they are wary about treatment. As future mental health professionals, I feel like it is our job to dismantle this misconception. Great explanation!

      Reply

    • Angela Petrella
      May 24, 2021 @ 09:45:46

      Hi Valerie! I loved how you responded to question number 3. You explained this so clearly and I couldn’t agree more with you on how if a test is unreliable, then the validity of the measure has no real standing. I also really liked how you mentioned that reliability is important when tracking one’s progress! This is so important to make sure that a client is actually benefiting from the treatment they are receiving.

      Reply

    • Jennifer Vear
      May 24, 2021 @ 12:08:17

      Hi Valerie!

      I really like how you explained the second question. Especially the part where you mentioned the integration of techniques, skills, and medication for patients’ treatment. Also, how a lot of patients will want a “quick fix” so they make these assumptions about correlations because they do not know that it can sometimes take more than just medication or just therapy to cause a change in their life. It takes a lot of the patient’s own willpower and motivation to be able to make various changes in their life for the better. There is no such thing as a “quick fix” or no one would need therapists!

      Great example!

      Reply

  9. Katie O'Brien
    May 23, 2021 @ 20:50:22

    1. Criterion/Domain-referenced instruments are instruments in which the individual’s (person taking the instrument) score is compared to some established standard or criterion, comparing the individual’s performance to that of said criterion, not to the scores of other individuals taking the same assessment. This would be the case, for example, on college exams. In order to receive an A, a student must achieve a score of at least 90%, in other words, the established criterion for an A on the exam. On the other hand, norm-referenced instruments are those in which an individual’s score is compared to the scores of other individuals who took the same instrument. For example, say a professor grades on a curve. A students’ score is interpreted comparatively to the scores of the other students in her class.
    It is important to understand the difference between the two when evaluating client results and then presenting them to the clients or their families. For example, maybe a student has not done as well on her Grade 2 Math tests compared to her classmates, (using norm-referenced instruments) but at the end of the year, has reached the appropriate grade 2 math standards for moving onto grade 3 anyways. (Using a criterion/domain-referenced instrument.)

    2. Correlation does not equal causation. I remember a conversation during undergraduate regarding this topic. My professor told us 2 statistics: the rate of violent crime goes up in the summer, and ice cream sales also go up in the summer. While those two are correlated, it seems outlandish to think an increase in icecream sales then increases violent crime in an area, or that violent crimes would increase ice cream sales. Upon closer examination, we see another variable: temperatures also go up in the summer. Increasing temperatures might have a closer relationship to increased violent crimes (tempers flare in the heat), and cold treat sales. It’s just a funny anecdote I always think about!
    Anyways, I think there’s a tendency to claim causation because in world where there are so many factors impacting so many other factors all of the time, it can be difficult to see which variables truly have effects on others, the extent to which they do, and how those variables might interact with even more variables, and around and around. In cases where things are tricker, say mental disorders, it would likely be easier to simply say one thing causes another – it would likely help to simplify therapies afterwards, too. For example, rather than look closer at the multitude of factors that might contribute to schizophrenia, one might be tempted to say “genetics cause schizophrenia” because then you aren’t dealing with so many nuances in it’s causes and treatment.

    3. Reliability is important for a couple of reasons. If we want to give a client an assessment to measure their intelligence, we need to be confident that the score we interpret and explain to the client is a true reflection of their intelligence, not some error or variance due to issues with the instrument we are using. If we give the same assessment to clients multiple times and their scores keep changing, likely it’s because the instrument isn’t very consistent, not because the clients intelligence, depression levels, etc. keep fluctuating. As a therapist, if you’re using an unreliable instrument, it might lead to misleading results – one session, a client might score as incredibly depressed and the next they are completely un-depressed and fine – leading the therapist to think therapy is working or the client is improving, when they truly might not be. This could really impact the course of therapy.

    Reply

    • Sergio Rodriguez Pineda
      May 24, 2021 @ 00:54:52

      Hi Kathy,

      I think I really like the example your teacher gave you. It made me think how often we make quick assumptions and determinations daily and in the mental health field with kids. For example, it’s pretty common that kids can be considered “problematic students” because they don’t pay attention to the teacher or anything on the board in class. Those kids can easily get prescriptions for ADHD when the professionals could find the real case analyzing another factor or variables like the hard situation at home, conflict with parents, being abused… or a simple reason like a non solved vision problem like myopia that makes the student feel tired when trying to look the board class thus the students will find easy to engage in activities nonrelated to class.

      Reply

    • Morgan Rafferty
      May 24, 2021 @ 09:18:28

      Hi Katie – I love your ice cream sales / crime rate example. That’s a great one! I teach PSY 101 and enjoy using the finding that shows a positive correlation between number of churches in a city/town and crime rate. I pose to the students in dismay: “Can it be true? The existence of more churches causes an increase in crime?! How can this be?” We discuss how higher number of churches also means a town/city with a larger population which in turn is also correlated with higher crime rates.
      It is easier to see that correlation does NOT mean causation with these ridiculous type of examples; but not so easy to avoid the trap when the examples seem logical. I appreciate your post! And wonder who you had for psychology professors at HC. 🙂 We will have to chat later about that. My favorite and my advisor = Prof Andrew Futterman.

      Reply

      • Katie O'Brien
        May 24, 2021 @ 12:27:58

        That’s another really great example! Wow. I wish I could remember the site he referenced – it was full of “out-there” correlations exactly like we talked about! So helpful in this conversation. I will have to look and see if I can find it.

        We certainly do! I never met a Professor Futterman! The professor with the crazy ice cream example was Professor Hallahan 🙂

        Reply

    • Giana Faia
      May 25, 2021 @ 21:26:53

      Hi Katie,

      Great explanations of criterion-referenced and norm-referenced, I like how you provided clear examples of each. Your example of correlation does not equal causation is also a great way to describe how two variables can be related, but are not the cause of something. I also agree with your point on why this assumption is still a common mistake. It is definitely easier to say x causes y, however, as we know this is not the case and there are many factors that can contribute and play a role.

      Reply

  10. Sergio Rodriguez Pineda
    May 24, 2021 @ 00:33:07

    1). Norm-referenced instruments are developed with norms base on another group of individuals that had taken the test already and supply a sample that allows comparing an individuals’ score to others individual’s score. The client’s results will be compared to the norming group constituted by a large population who already took the test and help to establish a normative group with a different percentual range where the induvial tested can be compared. Criterion/domain-referenced instruments are not being compared to other individuals. The test results are analyzed if the individual knows the domain by getting a score equal or superior to the cutoff score. How can we get that information? The ideal scenario is that a group of experts will determine that score and get to know if mastery de domain or not.
    One of the fundamental reasons to understand the difference between both types of tests is the results’ analysis. Norm-referenced instrument results will make sense when compared to a normative group of the domain. For example, trying to evaluate the score in attention will be alarming just if the score is high or below compared to the norm group. In contrast, the Criterion/domain-referenced instruments consider the induvial results apart from any other individuals or population’s score. It will be analyzed to determine if the individual obtained the required score to state that the person achieved the domain acknowledgment. A mistake in that analysis can lead to the wrong treatment or diagnosis, which means a significant not favorable impact on a human being’s life.

    2). Correlation indicates the relationship between two variables and how they can be affected by each other. Still, it does not mean that we know which variable causes the effect on the other. Therefore, it’s important to understand that there can exist a correlation between variables without causation. In psychology, it is common during research or a case analysis to try to identify the cause of a behavior problem or aim to find the casualization of that problem. If, for example, I want to prove that an active lifestyle will predict a high level of cognitive function in the old population, I will have to consider that probably a high level of cognitive functioning can lead to an active lifestyle. Thus, the relationship between the variables is a correlation and not a causation. Other variables like academic background or economic status would clarify the absence of causation between the variables.
    On the other hand, Mental health tends to fall under the explicative model (causation relationship). However, Mental health has to be analyzed from different perspectives, including bio-phyco-social perspectives and variables that will explain the correlation between multiple factors, not just causes.

    3). Reliability is the consistency of the result’s measurements when the testing procedure is repeated on a population of individuals or groups (Whiston, 2013). Reliability is essential for psychological assessments because it is the primary indicator of identifying an instrument’s psychometric qualities, like interpreting a client’s scores and determining an instrument’s validity. High reliability means that scores are precise, reproducible, and consistent from one testing occasion to another.
    That means that reliability will be one of the most (if not the most) relevant aspects that a counselor would like to analyze before taking to choose or apply a test to a client. Thus, before selecting the test, reliability will provide the counselor an idea of what grade of accuracy the patient’s process would determine once it applies a test before and after the therapy has been developed.
    Another reason is that if a person or client wants to know about unsystematic errors in the application of an instrument, the reliability coefficient will provide the amount of error in a score when an individual or group of people get tested. It includes factors like errors during the presentation of the test, harsh weather conditions, noisy environment, or particular conditions of the induvial (e.g., feeling sick, tired, or anxious). It is essential to consider that there are diverse ways to obtain the reliability of a test. It could be test-retest, alternate, or parallel form, and internal consistency could be calculated with the Split-Half technique.

    References:
    Whiston, S. C. (2013). Principles and applications of assessment in counseling (4th ed.). Belmont, CA: Brooks/Cole.

    Reply

    • Francesca Bellizzi
      May 25, 2021 @ 17:47:11

      Hi Sergio,

      I really enjoyed you blog post 🙂 Great explanation of what reliability is and why it’s so important! Frankly, I think that if somebody with no knowledge of the topic (or its importance) read your response it would be straightforward. Likewise, it is apparent that if a clinician were to use something is not reliable then the subsequent diagnosis would potentially also be unreliable.

      Reply

  11. Angela Petrella
    May 24, 2021 @ 09:42:16

    1. The difference between criterion referenced instruments and norm-referenced instruments is how individual performance is compared. When using norm-referenced instruments, an individual’s test score is compared with the scores of other individuals who have taken the same instrument. In a criterion referenced instrument, an individual’s test score is compared to an established standard. It is important to know the difference between the two because when interpreting results using a norm-referenced instrument, the information provided can only tell us how that individual tests within a specific group. On the other hand, when using a criterion referenced instrument, we can make interpretations based on ones level of mastery considering a generalized standard has already been established.

    2. Often times in mental health the idea that correlation equals causation is a common misconception. It is important not to make this assumption, as often times there are several variables that come in to play when understanding a specific result. With that being said, it can never be fully certain that one specific variable is the causing factor of an outcome, but rather that even when there is a strong relationship between said variables, there are still other factors to consider.

    3. Reliability is important for psychological assessments because if a test is unreliable, so is the data that is concluded from it. Ultimately, this can lead to inadequate treatment for a patient, or even a misdiagnosis. That is why it is so important to test the reliability of instruments used and reduce the amount of possible error so that individuals can ensure they are receiving the best care for their diagnosis.

    Reply

    • Lisa Andrianopoulos
      May 24, 2021 @ 18:51:09

      Hi Angela,
      I agree with what you said about correlations. As Dr. V said in class today, a perfect correlation is virtually unheard of! In fact, if you get a perfect correlation the first thing you should do is question your research, lol. There’s always error and extraneous variables to consider. If you can never be 100% certain then you can never say with 100% confidence that one causes the other.

      Reply

  12. Lindsay O'Meara
    May 24, 2021 @ 14:10:56

    1. A criterion or domain-referenced instrument is when a person’s score is compared with a standard score. For example, if you were to need an 85% on a test to pass, the individual would have to meet that standard score. Whereas a norm-referenced instrument is when a person’s score is compared to other peoples’ scores who have taken the same instrument. It is important to know this because a criterion or domain-referenced instrument is based on a predetermined score, and a norm-referenced instrument is used to compare a score to a group of people.
    2. The assumption that correlation is causation is still a common mistake, especially in the mental health field, I would assume is because it is easy to look at a significant result and believe that there is causation. A significant finding in a correlation might convince someone that there is cause there, but there can be no causation in correlations. So, even though it might be a strong correlation, there are a lot of factors that may not be accounted for. There is no experimental control in correlations so there is a limited understanding of how much a significant result could mean.
    3. Reliability is very important for psychological assessments because it is important to be consistent to make sure that individuals are assessed properly. If an assessment were to have unsystematic errors it could contribute to assessments coming back inaccurately. Essentially it is important to have consistency in assessments, and to take precautions to make sure that you safeguard your assessments and results.

    Reply

    • Lisa Andrianopoulos
      May 24, 2021 @ 18:43:17

      Great point about reliability Lindsay. How can you properly assess and/or be confident in your results if you can’t rely on the instrument you’re using? This has HUGE implications for research. If the researchers are not using reliable instruments then how could they ever know if their results are due the the variable they are investigating. There is no credibility.

      Reply

  13. Giana Faia
    May 24, 2021 @ 14:44:14

    1. With a criterion-referenced instrument, also called domain-referenced, the test scores of individuals are being compared to a fixed standard or criterion. An example of this is in order to get an A on a test, you must receive a 90% or above. Here, the focus is not on the individual’s performance compares to the performance of others. Along with this, there is a predetermined mastery component. With norm-referenced instruments, the scores of individuals are compared to other individuals scores. An example of this is your test score being compared to the scores of others in your class. While one compares your performance with a developed standard, the other compares your performance to other people’s performance. This difference is important because criterion-referenced focuses on the individual’s performance and whether they are meeting criterion or not. Whereas, norm-referenced focuses on where the individual is compared to the rest of the norm.

    2. Correlation is used to estimate the relationship between two variables, not to state that ___ causes ___ . This is a common mistake because other factors that are involved are often forgotten and not taken into account. If a student is getting poor grades and the teacher insists it is only because they do not study, then the teacher is assuming lack of studying is the cause of poor grades. However, the teacher is not taking into account other outside factors such as the student’s demanding job, their parents going through a divorce, plus other schoolwork load. In the mental health field, it is easier to say ___ is happening because of ___ because it provides a clear explanation. However, there are usually many factors contributing to the situation.

    3. Reliability refers to the consistency of the scores. Reliability is important for psychological assessments because each time an assessment is given, it is important to get the same or similar scores each time. If the same person is getting inconsistent scores, then the reliability might be questionable. It is important to get consistent scores to ensure that we are properly diagnosing and properly treating clients.

    Reply

    • Francesca Bellizzi
      May 25, 2021 @ 15:43:25

      Hi Giana,

      Great explanation about outside factors that are often forgotten and not taken into account. It’s funny that this assumption continues to be made in the mental health field, when every one I talk to in the field preaches that “correlation is not causation”. I wonder if there has to be more to why this keeps happening than just simply forgetting to take something into account. Maybe it is potentially due to the characteristics of the mental health field?

      Reply

  14. Francesca Bellizzi
    May 24, 2021 @ 16:54:47

    1. Simply, the difference between criterion/domain-referenced instruments and norm-referenced instruments is that criterion/domain-referenced instruments compare a person’s score to a predetermined score/scale; where as norm-referenced instruments compare a person’s score to a group of others who have also taken the test. Understanding this difference is important because criterion/domain-referenced instruments compare the individual to a basic level of mastery on a particular subject and individual “skill”, yet the other compares an individual to a specific population and whether they fall above or below average.

    2. I feel that this assumption is still a common mistake because most want to be able to give an explanation as to why something is occurring. In mental health specifically, there are general “guidelines” to situations that are not easily explicable that allow people to make basic assumptions and that – as a whole – the domain of mental health seems to be interested in cause. Likewise, there can be basic human error when wanting to interpret results that lead individuals to believe that “results = answer” instead of “results = possible relation”

    3. Reliability is important for psychological assessments because if they are reliable then they are testing and evaluating what they are designed to test / evaluate for. Similarly, it assures that the scores researchers receive are based on the actual test and not any external variables. A higher level of consistency helps assessors Overall, reliability is important within a psychological assessment to ensure that individuals are being diagnosed and treated properly – if thinking clinically.

    Reply

    • Giana Faia
      May 25, 2021 @ 21:08:30

      Hi Francesca,

      I like how you mentioned that in the field, people want to be able to provide an explanation for why something is occurring and finding the cause. I feel like that is always the initial thought, to find the cause but in this field there are usually many contributing/ external factors. I also thought it was an important point that you mentioned about basic human error when interpreting results.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Adam M. Volungis, PhD, LMHC

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 69 other followers

%d bloggers like this: