Retracting a study doesn’t mean its conclusion is false

Remember the RAND study that found that crime increased in areas after medical marijuana dispensaries were shut down? RAND made no causality claims, only correlation, yet the results enraged certain political entities, most notably the L.A. city attorney’s office.

After a lot of pressure, RAND retracted the study. In their news release, they said that the reason they voluntarily retracted it was that they “determined the crime data used in the analysis are insufficient to answer the questions targeted by the study.”

They did not find contradictory information, or any reason to necessarily doubt the findings, but rather decided that the amount of data was insufficient to make the conclusions solely based on that data. They claimed they intended to redo the study with more data, but we’ve heard nothing since.

What’s interesting, of course (though not at all surprising) is the way the retraction has been seen by some as a kind of proof that the findings were false – that, in fact, the closing of dispensaries were not followed by an increase in crime.

That has resurfaced with Scientific American’s article Doh! Top Science Journal Retractions of 2011

#5: Los Angeles marijuana dispensaries lead to drop in crime.

Keep smoking. The RAND Corporation retracted its own report in October after realizing its sloppy data collection.

The tone is annoying, and “sloppy” is a possible overstatement, but the retraction is news, so I have no issue with that.

But it doesn’t stop there. Nick Schou at OC Weekly pens a particularly ignorant column: Rand Study On Medical Pot Among Year’s Worst, Scientific American Says (Notice how it magically escalated from “top five retractions” to “among year’s worst”?) which is then positively linked to by the drug czar’s office.

Keith Humphreys goes even further and claims that the retraction means that the finding that “Closing medical marijuana dispensaries increases crime” was “inaccurate” and “false.”

Humphreys bizarrely accuses others of believing that “a retraction is the ultimate confirmation that a study’s results are true” while at the same time ignoring the fact that he has used it as the ultimate confirmation that a study’s results are false.

The retraction of a study means absolutely nothing regarding the truth or falsity of the conclusion, only that the study has now been made irrelevant to that discussion.

This entry was posted in Uncategorized. Bookmark the permalink.

25 Responses to Retracting a study doesn’t mean its conclusion is false

  1. darkcycle says:

    Spinning like a top.

  2. Duncan20903 says:

    .
    .
    “Never let the facts get in the way of disseminating an effective piece of hysterical rhetoric”
    ~~ The motto of the Know Nothing prohibitionist

  3. claygooding says:

    I am sorry,I missed something,,how did a think tank owned by a pharmaceutical executive become scientific?

    Rand is nothing but a tool for ONDCP/government research,if what they do is science I am a friggen brain surgeon,,,where did I leave that scalpel?

  4. Servetus says:

    There are laws against people impersonating doctors, lawyers, and police officers; but no law against someone impersonating a scientist. This legal oversight is the only reason Keith Humphreys is not sitting in a jail cell or out on bail.

  5. claygooding says:

    Fresh off the press,,at CNN,,no comments

    U.S.: States Say It’s Time To Rethink Medical Marijuana
    http://www.cnn.com/2012/01/01/us/medical-marijuana/index.html?iref=obnetwork

    Carreno said petitions to reschedule a drug take years to review. The DEA does its own analysis, then refers the requests to the FDA and the Department of Health and Human Services, which review their own research.

    “Then they send recommendations back to us, and based on the recommendation we get, we make a decision,” she said.

  6. Duncan20903 says:

    .
    .
    Oh let’s not be so hard on Mr. Humphries. He really does have a talent for making bullshit look like it makes sense. When you consider what he has to work with he’s certainly an elite level con man and is well deserving of the Bernie Madoff Trophy and inclusion in the Grifters’ Hall of Shame.
    ———-
    I heard that Sarah Palin’s boyfriend is going to endorse Mitt today. I hope he makes Mitt have Ms. Palin for VP too. Gosh she made the work of our nation’s comedians so much easier, and who can argue with the assertion that this country needs more laughter? Does anybody remember laughter?

  7. claygooding says:

    Only people that smoke pot laugh anymore,,it is a lost art.

  8. claygooding says:

    I was perusing the tv guide and could not help but notice that O’Donnell’s show is no longer listed,,did he get fired for his diatribe against congress and it’s failure to legalize pot?

    http://www.youtube.com/watch?v=gZTceWVtERo

    • claygooding says:

      Found him,,he got into an argument with the host of the show that took his prime time slot about Santorum.

      I used their comment to inject a little barb at msnbc:<)

    • Swooper420 says:

      That’s a great clip. Thx for posting a link to it. The guy wastes no words, that’s for sure!!!

  9. darkcycle says:

    The “reality based” community. What a fanciful place.

  10. Matthew Meyer says:

    Yeah, it’s like:

    A:[This study suggests, with some confidence, that closing dispensaries increases crime]

    NOT-A

    REAL CONCLUSION: [this study does not allow the confidence we require to show that closing dispensaries increases crime]

    PROHIB CONCLUSION: [closing dispensaries decreases crime]

  11. claygooding says:

    The Regulate Marijuana Like Alcohol Act of 2012
    http://www.marijuana.com/drug-war-headline-news/187457-co-regulate-marijuana-like-alcohol-act-2012-a.html#post1540505

    Regulate Marijuana Like Alcohol Act aims 160,000 signatures at 2012 ballot

    http://blogs.westword.com/latestword/2012/01/regulate_marijuana_like_alcohol_act_signatures_ballot.php#Comments

    ​In July, the folks behind the Regulate Marijuana Like Alcohol Act of 2012 launched a petition drive to land the measure on the November 2012 ballot — and at 11:30 p.m. today, they’ll celebrate the effort at a press event before formally submitting nearly 160,000 signatures. Proponent Mason Tvert notes that backers overshot their original goal.

  12. “They did not find contradictory information, or any reason to necessarily doubt the findings …”

    This isn’t true. They appeared to find plenty of reasons to doubt the findings. The LAPD’s own crime stats, which RAND did not use (and really, they’re only crime stats for the city of L.A.), showed consistent decreases in crime in all areas affected.

    What’s more, I first doubted the study in my LA Weekly reporting — the City Attorney’s questions came later.

    The study claimed that crime went up around dispensaries that closed, yet hardly anyone, not even the CA, the LAPD, or the city, could determine what dispensaries had actually close and, in fact, there was ample evidence to suggest that many if not most did not, which would lead to the opposite conclusion — that crime went up at dispensaries that stayed open.

    Except that, as RAND has admitted, the crime stat data was dubious and did not come from the LAPD.

    “Never let the facts get in the way of disseminating an effective piece of hysterical rhetoric.” Indeed.

    • Pete says:

      Sorry, Dennis, I had not followed your reporting on the issue. I was basing my statement that you quoted on the press release that RAND put out where they talked about the data being “insufficient” and did not mention that other data had been demonstrated as “contradictory.”

      Regardless of the outcome, I welcome the proper use of data and the proper findings for the study, and hope that RAND will actually do so, although I find it likely that it will go into a black hole. Perhaps someone else will do a study, perhaps not.

      The issue itself isn’t all that important in the scheme of things, but it is a curiosity.

      It doesn’t change the point of my post, however — that merely because a study has been retracted doesn’t a priori mean that the findings were wrong. And what has been happening in the places I linked is that they are using the fact of the retraction as proof of an inverse finding.

      • Duncan20903 says:

        .
        .
        Gosh, if I believed in using ad hominem arguments I’d point out that Mr. Romero is a columnists who regularly ignores factual evidence, publishes hysterical rhetoric as if it were factual, and is a strict adherent to the motto of the Know Nothing prohibitionists. I’d even say that he’s cut from the same cloth as Mr. Humphries and Mr. Sabet minus the name recognition. But I’m proud of myself for being above all that. I will note that at least Mr. Romero seems to only have the financial interest of whatever income he receives for writing his column in the local throwaway.

        Perhaps we should ask LAPD Chief Charlie Beck what he thinks.

        The opinion of the Sacramento Police?

        Don’t the people of San Francisco know how to complain?

        So far the best the prohibitionists can come up with is a withdrawn study. Pathetic, but par for the course.

        • Duncan20903 says:

          .
          .
          I must apologize and take back the gratuitous insult about the LA Weekly. They provide a source of much needed source of information of where to go to find relaxation for overstressed Los Angelenos:
          http://laweekly.backpage.com/Domination/
          http://laweekly.backpage.com/TranssexualEscorts/

          What would the perverts of Los Angeles do without the LA Weekly? They’d be driving around abusing their naughty parts in public is what. I assure you, it wouldn’t be pretty. The LA Weekly is a true servant of the public good. Mr. Romero can be proud of his employer, no doubt.

        • Duncan20903 says:

          .
          .
          Well now I have to fly to LA. I’m certainly glad I Googled the LA Weekly! 420 friendly MILF domina’s aren’t exactly a dime a dozen you know. Va Va Va Voom! Time to play carnival.

          6ft tall very Curvy w/Big Booty, blonde MILF/cougar. INCALL WEST COVINA f0ur20 friendly – 35 Picture (WEST COVINA)

    • darkcycle says:

      Still sounds like they need new or revised data, Dennis. The conclusons are retracted, but the question is not resolved and the hypothesis still stands. Provide me with enough data and I’ll resolve your question for you toot suite. Until then don’t claim that this retration means anything but “we don’t know”.

  13. Duncan20903 says:

    .
    .
    Arizona lost.

    Governer Brewer is drinking to excess and singing “poor poor pitiful me” tonight. She can’t win a lawsuit against the Feds to save her life. Oh sorry, those two statements aren’t connected in any way other than being about Governess Brewer.

    /snip/
    U.S. District Judge Susan Bolton said the case isn’t legally eligible for court consideration because the state hasn’t established a genuine threat of prosecution of state employees for administering the law, as Brewer had claimed.
    /snip/

  14. Probably the correct term for what RAND did here is: publication bias. From wiki:

    “Publication bias is the tendency of researchers, editors, and pharmaceutical companies to handle the reporting of experimental results that are positive (i.e. showing a significant finding) differently from results that are negative (i.e. supporting the null hypothesis) or inconclusive, leading to bias in the overall published literature. “

    • Duncan20903 says:

      .
      .
      No, that can’t be it. We’re talking about RAND which is a pay to play pseudo research company which produces “scientific research” on demand. Their company motto is, “the customer is always right and we prove it.”

      Would it be that hard to believe that the report being discussed was produced to be withdrawn? When it was withdrawn they said they were going to

      It’s very amusing that in his post above Mr. Romero claims that RAND didn’t use LAPD supplied data then claims that’s the only source of data. Are we therefore supposed to infer that RAND used PFTA crime data? Upon examination of the study we find that RAND obtained data from CrimeReports.com. So where did CrimeReports.com get their data? Why from the LAPD, where else would they obtain it? It’s the only source of crime statistics for L.A.

      CrimeReports works with thousands of law-enforcement agencies to help reduce, prevent and solve crime by enabling officials to easily open and manage a controlled dialog with citizens. We offer an online family of crime fighting tools including a public crime map, alert messaging, anonymous tipping and data analytics.

      Here’s a link to Mr. Romero’s article about the study.

      Here’s a link to Mr. Romero’s article about the retraction.

      I’m certainly not interested in arguing about this issue with Mr. Romero. If he wants to assert that RAND is incompetent and therefore the results of their studies should be ignored why in the world would I do so? That would mean that we see eye to eye in this instance. Unfortunately I suspect that Mr. Romero would apply the competence label on a study by study basis with his opinion varying by whether he likes the results of a particular study or not.

      I also think there’s some measure of that occurring on our side of the table and would suggest it would be better to try to avoid that in the future. Just as we should have been highly skeptical to the point of dismissal of Mr. Obama based on his choice of Skidmark Joe as his VP the same skepticism should apply to studies by RAND. Of course hindsight is 20/20 but learning from experience can certainly help clarify foresight.

      The reaction to the RAND study is yet another example of how willing the Know Nothing prohibitionists are willing to devour their own if they stray from the approved hysterical rhetoric, and that’s always amusing. It’s otherwise meaningless but I enjoy being amused so I guess they’re good for something. I’m also gratified that Mr. Romero chose to plagiarize me above. I’m shocked that he even read one of the comments, much less two. Perhaps he bumped into it accidentally. Regardless, it’s always nice to be acknowledged by people who give me an urge to genuflect in their presence.

      Who is it that’s not letting the facts get in the way of the dissemination of effective hysterical rhetoric again?

Comments are closed.