[Truehoop] PER on Trial

Hayesfan · Mar 25, 2009

Long read, but I found it interesting.

John Hollinger's Player Efficiency Rating PER has been in heavy rotation on ESPN.com the last couple of days (today with new Value Added and Estimated Wins added rankings, yesterday with a case for LeBron James as MVP).

And yet the existence of PER -- the idea that a basketball player's contributions could be usefully expressed in a single number -- gives some people fits.

Watch the Celtics play defense. That'd be my case, if I woke up in some kind of stat geek Alice in Wonderland, and had to prove PER worthless or be turned into a sliderule.

The Celtics' defense is a friggin' symphony of subtlety. It humbles the NBA not because of one player doing one or two things that are obvious and measurable -- but because five players do the right combinations of big and little things, reading and reacting, as a unit.

On just one play, you might see Kendrick Perkins blitzing a ball-handler in a way that's really convincing, inspiring an escape dribble that lets Paul Pierce catch up, taking the easy shots out of the pick and roll. All the while, Perkins never really committed, and before the ball-handler has even completed his escape, Perkins slides back to the paint, filling space, letting Kevin Garnett stay locked onto the talented big man. And now as the shot clock winds down, and the ball has been swung, Ray Allen gets his hands -- they're not really going to call a hand-check on that, are they? -- into his opponent's deadly right arm, forcing the star player left, to a spot on the floor where Rajon Rondo -- back turned, pretending he hasn't been noticing -- has been waiting all along to pounce. Tick tock tick tock.

That's a missed shot for the offense, that's a thing of beauty for the Celtics. That's basketball, in its vast complexity.

And you expect me to believe that one of those players gets an 11 for his performance, and another gets a 16? Who are you to pretend that grades for such things can be handed out -- particularly by people who haven't ever done these things themselves?

What is the point of that, anyway?

Surely real basketball people need no such tools. The Celtics' own stat guru -- Mike Zarren -- was pretty pointed at last year's MIT conference that simply ranking players in terms of overall quality is just about the last thing teams ask their stat people to do. They are looking into much more sophisticated and refined questions than that.

I'd close my argument, before that mythical court (who would be the judge, by the way? Dean Oliver in a Mad Hatter costume is my first draft pick) making noises about Celtic assistant coach Tom Thibodeau. That's his defense we're talking about. He's one of the most important people in the NBA -- his work is the very bedrock of one of the best teams of the decade -- and yet just about none of what he does that matters shows up in PER.

How, therefore, can PER matter?

There I would rest my case.

And then, if my opponent (are there opponents in this court? I'm trying to win and I don't even know the rules) has a halfway decent intern, I'd get crushed.

Because a good intern would have spent some time perusing Harvey Pollack's 2008-2009 Statistical Yearbook. In between a lot of other stuff (Page 15: Last season, Tyson Chandler was the dunker of nearly 9% of the NBA's total alley-oops. Page 234: Shaquille O'Neal led the league by winning 72% of his jumpballs and Antonio Daniels has never fouled out. Page 243: Joe Crawford has reffed twice as many Finals games as any other active referee.) that intern surely would have found page 65.

A damning piece of evidence, to be sure.

This is where Pollack would appear to condemn me to life as a sliderule, by sharing that very same Tom Thibodeau's own homebrew Player Efficiency Ratings. Kevin Garnett had the highest score, at 68.84. The formula is given as "field goals made doubled minus field goals missed plus free throws made minus free throws missed times .5 plus 3-point field goals times three minus 3-point field goals missed times 1.5 plus rebounds and assists minus personal fouls plus steals minus turnovers plus blocks." All that is divided by minutes played.

This is where mythical Mad Hatter judge Dean Oliver is about to spit up on his colorful costume.

"Oh," he's laughing, "this is rich ... basketball is too subtle to express with numbers? And Tom Thibodeau proves it? What, then, of Thibodeau's fourth-grade level equation?"

(Neat fiction writer's trick I have going here, by the way. I'm calling a powerful NBA coach's equation bush league, but kind of making it seem like it's maybe the dean of NBA statistics speaking.)

As long as we're in a mythical court seemingly without rules, I would here approach the bench and promptly petition to switch teams. I do not want to be a sliderule, and my old team is clearly about to lose.

If Oliver resisted I'd reach into my very official looking briefcase and produce my well-thumbed copy of his own 2004 book "Basketball on Paper," and propose to read aloud.

In defense of PER, I would point to page 83, where there is a chart exposing some of the nitty gritty ingredients of an older version of PER, as reverse-engineered by Oliver. In PER, Oliver suggests, an assist is weighted at .79 of a point. A steal is worth 1.2 points.

In Thibodeau's system, an assist is worth one point. So is a steal. And a rebound.

Now, you can ask -- Why is it that Thibodeau's system is so simple, and Hollinger's so complicated? We all know the answer, right? It's because Thibodeau's numbers are designed to be easy for a basketball coach to compute, while Hollinger's numbers are designed to be useful. Another way of saying the same thing: It's because John Hollinger actually studied this stuff in college.

The numbers that make up PER are not shots from the hip. They are actually tested and honed. Two years ago Dan Rosenbaum and Dave Lewin wrote a paper (the basics of their presentation are online) where they put several stat systems to the best test imaginable: They went back in time, and cleverly measured how well these systems predicted what would actually happen on the court. As in, before the game, the different systems predicted that this or that team was stronger. After the game, which one had made a more reliable prediction?

Of all systems measured -- adjusted plus/minus, Wins Produced, points per game, and several others -- only PER was found to be more useful than crude models of human-based decisions in predicting what would actually happen in games (as measured by adjusted plus/minus -- it's a little confusing).

Which means that thanks to Hollinger's clever weighting of the different elements available (all those .72s and 1.2s, which Hollinger tweaks every year) PER could replace a very crude model of your GM and your team might actually improve -- no matter how subtly Kendrick Perkins may hedge on the pick and roll. Rosenbaum's paper (no longer online, I'm afraid in its entirety) was not kind to most stat systems, but included this in the conclusion:

The results of this paper should not discourage the use of statistical analysis in basketball. John Hollinger's PER metric, for example, performs quite well. The important thing to take away is that statistical analysis must be done carefully and rigorously, with an appreciation of the complex and dynamic interactions that are at the heart of the game of basketball. The non-academic statistical analyst community in basketball, led by Dean Oliver, author of Basketball on Paper, challenges NBA decision-making, but with a degree of restraint and respect for conventional wisdom.

"My null hypothesis is usually traditional coaching or management wisdom. So a hot hand exists, defense wins championships, and statistics are irrelevant until I prove otherwise (which I think I've done in many cases). Others may choose a different null hypothesis, but I think mine makes sense because I work with coaches and management and I'm not Billy Beane -- it is my burden to prove things, not theirs." (Oliver, 2005).

(After this bit of brown-nosing ... how can I lose?)

Now, here's the kicker: Tom Thibodeau's system probably works just fine for him. Remember when Rosenbaum talked about "the complex and dynamic interactions that are at the heart of the game of basketball?" Thibodeau has all those decades of basketball knowledge to fill in the gaps and provide context. I sincerely doubt he really advocates using his system to make every, or indeed, any key decisions. His PER is merely a time-saving tool -- the Cliff's notes of a thousand box scores.

From that, you can proceed using real first-hand basketball knowledge to further hone your thinking -- or to toss out the results entirely in some cases.

We all know, for instance, that both PER and Thibodeau's system reduce that beautiful Celtics' defensive play to the rebound that occurred after the fact -- the meaty parts of the defense aren't even charted. So clearly, such rankings must be consumed with a certain context.

Which Hollinger and nearly every stat expert out there -- including your honor the judge -- advocates. Real basketball knowledge is king. Statistical systems like PER are tools to help cut through the reams of available information.

To the extent that such tools are useful, however, let them be good ones, like PER.

And with that, I'd make my way out of the court, very pleased not to be a sliderule.

League-Wide Issues, John Hollinger, Dean Oliver, Dan Rosenbaum
Click to expand...

Edit:

My apologies... here's the link http://myespn.go.com/blogs/truehoop/0-38-359/PER-on-Trial.html

pmac · Mar 25, 2009

link?

Easy · Mar 25, 2009

They keep pitching "real basketball knowledge" against statistical analysis. What is "real basketball knowledge"?

baller4life315 · Mar 25, 2009

The results of this paper should not discourage the use of statistical analysis in basketball. John Hollinger's PER metric, for example, performs quite well. The important thing to take away is that statistical analysis must be done carefully and rigorously, with an appreciation of the complex and dynamic interactions that are at the heart of the game of basketball.
Click to expand...

I don't recall Hollinger ever arguing his PER formula is Gospel. It is what it is: an all-in-one rating that attempts to measure a player's per-minute contributions. Offensively, it paints a great picture of a player's value to a team. Defensively, and Hollinger even admits this, the PER formula is unfair to the Bruce Bowen's of the world that play excellent defense but don't really record many blocks or steals.

roslolian · Mar 25, 2009

baller4life315 said:

I don't recall Hollinger ever arguing his PER formula is Gospel. It is what it is: an all-in-one rating that attempts to measure a player's per-minute contributions. Offensively, it paints a great picture of a player's value to a team. Defensively, and Hollinger even admits this, the PER formula is unfair to the Bruce Bowen's of the world that play excellent defense but don't really record many blocks or steals.
Click to expand...

I think the reason for this is you can see the result of a great offensive play (i.e. score or assist), while the result of great defensive plays is intangible. A lot of times a good defensive play results in an aborted or abandoned offensive set, and there really is no way to record that, right?

Theoritically speaking if you have the data you can make a somewhat accurate defensive rating for every player in the league, but to do that you must track the exact players he had been guarding/matched up and for how long.

baller4life315 · Mar 26, 2009

roslolian said:

I think the reason for this is you can see the result of a great offensive play (i.e. score or assist), while the result of great defensive plays is intangible. A lot of times a good defensive play results in an aborted or abandoned offensive set, and there really is no way to record that, right?

Theoritically speaking if you have the data you can make a somewhat accurate defensive rating for every player in the league, but to do that you must track the exact players he had been guarding/matched up and for how long.
Click to expand...

Right. I'm not questioning the PER formula itself. As we both mentioned, it's pretty obvious why the PER formula is unfair to the Bruce Bowen's of the world. My issue is not with Hollinger's inability to measure intangibles but with the writer's implication that Hollinger's formula isn't good enough because he's not taking "real basketball knowledge" into consideration or "appreciating" whatever correctly.

AntiSonic · Mar 26, 2009

The fact that a guy who's never played a second round playoff game can have a higher career playoff PER than legends like Barkley, Abdul-Jabbar, Bird, Magic freaking Johnson proves how useless the number is.

Hayesfan · Mar 26, 2009

AntiSonic said:

The fact that a guy who's never played a second round playoff game can have a higher career playoff PER than legends like Barkley, Abdul-Jabbar, Bird, Magic freaking Johnson proves how useless the number is.
Click to expand...

Perhaps you aren't quite understanding what PER indicates. It's not saying that he's had more playoff success.

pmac · Mar 26, 2009

Hayesfan said:

Perhaps you aren't quite understanding what PER indicates. It's not saying that he's had more playoff success.
Click to expand...

Well, most people consider playoff success proof of how good a player is.

One of the few things i do like about PER is that it makes it tough to objectively dismiss a player having a great year, like Wade is this year. Despite his team only being 5 games over .500 he has been amazing. Yeah, it's easy to score alot on a bad team but what about the rebounds, assists, steals, blocks, etc.

Don't worry Antisonic, if Tracy Mcgrady plays a few more seasons after he comes back from this surgery his career PER numbers are bound to plummet.

Easy · Mar 26, 2009

pmac said:

Don't worry Antisonic, if Tracy Mcgrady plays a few more seasons after he comes back from this surgery his career PER numbers are bound to plummet.
Click to expand...

LOL. That will get AntiSonic into high heaven for sure.

Forums

[Truehoop] PER on Trial

Hayesfan Member

pmac Member

Easy Boban Only Fan
Supporting Member

baller4life315 Member

roslolian Member

baller4life315 Member

AntiSonic Member

Hayesfan Member

pmac Member

Easy Boban Only Fan
Supporting Member

Share This Page

About ClutchFans

Rockets Content

Support ClutchFans!

[Truehoop] PER on Trial

Hayesfan Member

pmac Member

Easy Boban Only Fan Supporting Member

baller4life315 Member

roslolian Member

baller4life315 Member

AntiSonic Member

Hayesfan Member

pmac Member

Easy Boban Only Fan Supporting Member

Share This Page

Easy Boban Only Fan
Supporting Member

Easy Boban Only Fan
Supporting Member