1. Welcome! Please take a few seconds to create your free account to post threads, make some friends, remove a few ads while surfing and much more. ClutchFans has been bringing fans together to talk Houston Sports since 1996. Join us!

How a Bush-promoted Christian prison program fakes success by massaging data.

Discussion in 'BBS Hangout: Debate & Discussion' started by Oski2005, Aug 5, 2003.

  1. Oski2005

    Oski2005 Member

    Joined:
    Nov 14, 2001
    Messages:
    18,100
    Likes Received:
    447
    http://slate.msn.com/id/2086617/

    Faith-Based Fudging
    How a Bush-promoted Christian prison program fakes success by massaging data.
    By Mark A.R. Kleiman
    Posted Tuesday, August 5, 2003, at 9:35 AM PT

    The White House, the Wall Street Journal, and Christian conservatives have been crowing since June over news that President George W. Bush's favorite faith-based initiative is a smashing success.

    When he was governor of Texas, Bush invited Charles Colson's Prison Fellowship to start InnerChange Freedom Initiative, a fundamentalist prison-within-a-prison where inmates undergo vigorous evangelizing, prayer sessions, and intensive counseling. Now comes a study from the University of Pennsylvania's Center for Research on Religion and Urban Civil Society reporting that
    InnerChange graduates have been rearrested and reimprisoned at dramatically lower rates than a matched control group.



    For those who know how hard it is to reduce recidivism, the reported results were impressive. Colson celebrated the report by visiting the White House for a photo op with the president. House Majority Leader Tom DeLay issued a triumphal press release. The Journal smacked critics of faith-based programs for "turning a blind eye to science" by opposing InnerChange. The report heartened officials in the four states that have InnerChange programs and buttressed President Bush's plan to introduce the Christian program in federal prisons.

    You don't have to believe in faith-healing to think that an intensive 16-month program, with post-release follow-up, run by deeply caring people might be the occasion for some inmates to turn their lives around. The report seemed to present liberal secularists with an unpleasant choice: Would you rather have people "saved" by Colson, or would you rather have them commit more crimes and go back to prison?

    But when you look carefully at the Penn study, it's clear that the program didn't work. The InnerChange participants did somewhat worse than the controls: They were slightly more likely to be rearrested and noticeably more likely (24 percent versus 20 percent) to be reimprisoned. If faith is, as Paul told the Hebrews, the evidence of things not seen, then InnerChange is an opportunity to cultivate faith; we certainly haven't seen any results.

    So, how did the Penn study get perverted into evidence that InnerChange worked? Through one of the oldest tricks in the book, one almost guaranteed to make a success of any program: counting the winners and ignoring the losers. The technical term for this in statistics is "selection bias"; program managers know it as "creaming." Harvard public policy professor Anne Piehl, who reviewed the study before it was published, calls this instance of it "cooking the books."

    Here's how the study got adulterated.

    InnerChange started with 177 volunteer prisoners but only 75 of them "graduated." Graduation involved sticking with the program, not only in prison but after release. No one counted as a graduate, for example, unless he got a job. Naturally, the graduates did better than the control group. Anything that selects out from a group of ex-inmates those who hold jobs is going to look like a miracle cure, because getting a job is among the very best predictors of staying out of trouble. And inmates who stick with a demanding program of self-improvement through 16 months probably have more inner resources, and a stronger determination to turn their lives around, than the average inmate.

    The InnerChange cheerleaders simply ignored the other 102 participants who dropped out, were kicked out, or got early parole and didn't finish. Naturally, the non-graduates did worse than the control group. If you select out the winners, you leave mostly losers.

    Overall, the 177 entrants did a little bit worse than the controls. That result ought to discourage InnerChange's advocates, but it doesn't because they have just ignored the failure of the failures and focused on the success of the successes.

    The Penn study doesn't conceal the actual poor outcome: All the facts reported above come straight from that report. But the study goes out of its way to put a happy face on the sad results, leading with the graduates-only figures before getting to the grim facts. Apparently, the Prison Fellowship press office simply wrote a press release off the spin, and the White House worked off the press release. Probably no one was actually lying; they were just believing, and repeating as fact, what they wanted to believe. It's hard to know for sure what those involved were thinking: Study author Byron Johnson canceled a scheduled interview at the last moment. The White House didn't respond to requests for comment.

    InnerChange program manager Jerry Wilger says he doesn't know much about research, but he doesn't think it's fair to count the performance of the people who dropped out of his program against him, a fair-sounding objection that misses the point entirely. If InnerChange's 177 entrants were truly matched to the control group but ended up having more recidivism, then either the apparent success with the graduates was due to "creaming" or the program somehow managed to make its dropouts worse than they were to start with. If the program genuinely helped its graduates and didn't harm its dropouts, and if the whole group of entrants was truly matched to the controls, then the group of 177 should have done better than the controls. And they didn't.

    So, the feel-good winners-only analysis simply isn't worth the paper it's printed on. Only the full-group analysis (known technically as "intent-to-treat," a holdover term from its origins in medical research) has any real value. And on that analysis, the program has a net effect of zero or a little worse than zero. That makes it a loser.

    John DiIulio, an intellectually serious advocate of faith-based programs who was the first director of the Bush administration's faith-based initiatives and the founder of the Penn research center, acknowledges frankly the results weren't what a supporter of such programs would have hoped for. But he points out that a single study almost never provides a convincing yes or no answer on a program concept. "The orthodox believers point to a single positive result and say it proves faith-based programs always work. The orthodox secularists point to a single negative result and say it proves faith-based programs never work. They're both wrong."

    The poor result of InnerChange doesn't mean that no faith-based prison program could work, but it does mean that this one hasn't, at least not yet. It joins a long line of what seemed like good ideas for reducing recidivism that didn't pan out when subjected to a rigorous evaluation. Maybe my own pet, literacy training, wouldn't do any better in a real random-assignment trial. But that's why you do evaluations; they tell you things you didn't want to hear. If you're honest, you listen to them.

    And if you're smart, you don't listen the political advocates of "faith-based" this and that when they say they're only asking us to support programs that have been "proven" to work.

    Curse my bad memory, but I seem to remember reading of non-faithed based programs with much better success rates that are being cut. I could be wrong though, wrong like the report that the article discusses
     
  2. ron413

    ron413 Member

    Joined:
    May 29, 2002
    Messages:
    3,915
    Likes Received:
    104
  3. FranchiseBlade

    Supporting Member

    Joined:
    Jan 14, 2002
    Messages:
    51,813
    Likes Received:
    20,472
    Hey if the facts don't fit what you want them to be, then just rework things until they do.
     
  4. MadMax

    MadMax Member

    Joined:
    Sep 19, 1999
    Messages:
    76,683
    Likes Received:
    25,924
    That's unfortunate. I work with and am good friends with a gentleman who has a non-profit that does prison ministry. He particularly has focused on juvenile offenders. The stories he tells are heartbreaking, but there are also many, many good stories that come out of it. It's unfortunate that a group would fudge numbers and thereby create a negative light on those individuals who are working their collective butts off in this field.
     
  5. glynch

    glynch Member

    Joined:
    Dec 1, 2000
    Messages:
    18,087
    Likes Received:
    3,605
    Unfortunately this is just par for the course with the gang who massages the intelligence or policy studies till the results fit their desired outcome.

    See my thread discussing similar political misuse of intelligence and studies in such areas as the Iraq War, global warming and the effect of tax cuts.

    Anecdotal stories told about faith based volunteerism are indeed heartwarming. It would be nice if possible to have some empirical evidence that they are more effective as claimed than other programs.

    Until then, it is discouraging to have government services privatised, as all too often it then just becomes another way to kickback government money to your contributors. It is no accident that it is the old GOP Watergate figure whose program gets the funds and publcity.

    cooked evidence
     
  6. bnb

    bnb Member

    Joined:
    Jul 7, 2002
    Messages:
    6,992
    Likes Received:
    316
    This is why you always have be wary of conclusions from studies, headlines, and reporting if you don't understand the methodology and sample used.

    This isn't just a Bush practice. You see it all the time.
     
  7. JuanValdez

    JuanValdez Member

    Joined:
    Feb 14, 1999
    Messages:
    35,066
    Likes Received:
    15,245
    There is something I don't quite understand about the control group. He says the control group is matched. Does he mean the control is matched to the 75 'graduates' or to the 177 original volunteers? If it was matched to the graduates, but Kleiman wants to compare all the volunteers to the control, then his picture would be skewed, wouldn't it?

    For example, he mentioned that you needed to find a job on the outside to be considered a graduate. Does this mean the control group also consisted of ex-cons who got jobs? And, if so, should he really compare recidivism of a group of volunteers with under 50% employment to a control group with 100% employment? Or am I reading this all wrong?

    Obviously, it wouldn't make much sense for Kleiman to try a comparison like that. But, it also wouldn't make much sense for the original study to compare the 75 graduates to a control group that didn't match in something as basic as employment.
     
  8. bnb

    bnb Member

    Joined:
    Jul 7, 2002
    Messages:
    6,992
    Likes Received:
    316
    I too was confused by his conclusions on an intuitive sense. He seems to state the obvious when he says:

    Yet he continues with conclusions that the inmates exposed to the program did worse than the 'control.' What was the 'control group?'

    It certainly seems the original study was spun, and I tend to think that this rebuttal is guilty of much of the same.
     
  9. rimbaud

    rimbaud Member
    Supporting Member

    Joined:
    Nov 3, 1999
    Messages:
    8,169
    Likes Received:
    676
    From the Penn website:

    "Johnson examined IFI data previously reported by the State of Texas and also conducted his own fieldwork, interviewing scores of program staff and participants. The 177 participants in the study included 75 IFI graduates who completed all phases of the program and 102 non-graduates (51 were paroled early, 24 voluntarily quit the program and 27 were removed for disciplinary or medical reasons). All participants were released prior to Sept. 1, 2000.

    IFI participants were compared to a matched group of 1,754 inmates who met the IFI selection criteria but did not participate in the program. There were no statistically significant differences between the groups. "

    So, the control group consisted of the kind of inmates that would be accepted into the position but instead went out on their own.
     
  10. bnb

    bnb Member

    Joined:
    Jul 7, 2002
    Messages:
    6,992
    Likes Received:
    316
    Thanks Rimmy.

    As I said, the results surprise me -- I would have expected ANY intensive program to have some sort of beneficial effect -- but it doesn't seem the 'control group' comparisons were unfair.

    I hate manipulated results...
     

Share This Page