A Peek Behind the Curtain

Posted by Rob Mahoney on March 10, 2010 under Commentary | 15 Comments to Read

I’ve received a handful of emails and comments over the last week or so asking me to chime in on the Mavs’ surprisingly low standing in John Hollinger’s power rankings. For the most part I’ve stayed away from it, and for two reasons:

  1. They’re power rankings. No offense to John or Marc Stein or anyone else who compiles power rankings anywhere on the internet, but they just don’t interest me. They’re arbitrary descriptors of how teams have done over the last week or so with emphasis on ranking rather than analysis. There’s just not much you can do with a short blurb that’s going to shed much light on a team’s success. If you dig power rankings, then by all means. I just don’t like getting too caught up in which team is put where.
  2. Unlike other power rankings, Hollinger’s are completely transparent. He spells out his methods explicitly, even giving you the exact formula by which his rankings are calculated, and the reasoning behind the weighting of each of the factors. He also explains why he chose specific measures over others, and the whole process is based on plugging numerical data into Hollinger’s equation.

Given those two things, I’m not exactly sure what all of the fuss is about. The Mavs have been winning games, but not all of the wins have been convincing. Nail-biters over teams like the Wolves and the Heat are not going to boost the Mavs’ standing in a data-centric ranking system, unless it’s completely dependent on win-loss record. Hollinger’s rankings are not, and he tells us as such in his explanation.

Instead, he looks to point differential, which just about any source will tell you is a better indicator of future success than overall win-loss record. Although talent and performance are definitely factors, close games are more likely to be influenced by luck; a bounce of the ball here or there can completely shift the balance, and though it’s certainly impressive that the Mavs are able to execute in high-pressure situations, it’s not necessarily all that indicative of the Mavs being a dominant team. Cue Kevin Pelton of Basketball Prospectus:

Better teams definitely tend to win more close games than weak ones, but the relationship is hardly perfect. The correlation is just .454 (a correlation of 1 or -1 means two variables are perfectly in sync, while a correlation of 0 means no relationship whatsoever). There are two extreme schools of thought on close games–those that believe they are primarily decided by luck and those that feel they are primarily decided by teams and demonstrate their true ability. Neither position is supported by the data.

Instead, what the results tend to show is that the difference between good teams and bad teams is mitigated in close games. Look at the best-fit regression line on the chart. The slope is nowhere near 1, and the difference between the expected record in close games for the very best teams (about .600) and the very worst teams (about .400) is much smaller than the difference between them in games that are not decided down the stretch.

When you think about it, this makes perfect sense. Say you were coaching (or cheering on, if you prefer) an underdog team playing a powerful opponent. If I gave you the opportunity to advance directly to the final five minutes of a close game, even if you were trailing by a couple of points, you would take that scenario because anything could happen over the shorter period. The same logic can be applied to explain why we see more upsets in the one-and-done NCAA Tournament than in the NBA’s best-of-seven postseason. The smaller sample draws all teams toward .500.

The data just doesn’t support the idea that better teams perform better in close games in any kind of conclusive fashion, for exactly the reasons that Pelton describes. It’s still bloody impressive that the Mavs have been able to win 12 games in a row, but it would obviously be more impressive if every win came by way of a double-digit margin.

As you well know, they haven’t. Most have been competitive well into the fourth quarter if not until the very end, and that certainly doesn’t reflect well on the Mavs.

Even given their recent tear, the Mavs’ point differential isn’t receiving any kind of considerable bump. Wins are obviously more helpful than losses in that regard, but it’s not as if a six-point win over the Bulls or a five-point win over the Bobcats is going to make some kind of profound impact on their overall standing. Dallas has overtaken Denver and is headed towards L.A. in terms of win-loss, but they very nearly snatched the record for consecutive wins by an unimpressive margin.

That may be enough to score brownie points with the likes of you, I, and Stein, but the numbers aren’t so kind. Hollinger warns us of that very thing in the explanation of his methods:

Since this is an entirely automated ranking, you’ll notice certain “human” factors missing. It doesn’t know which players are about to come back from injury or which teams have been playing without their best players for the past 10 games. Along the same lines, it doesn’t take into account injuries, trades, controversial calls or any other variables — just the scores, please. Nonetheless, it can be very useful because it allows us to see what the landscape looks like when we remove our usual filters. [Ed. note: Emphasis mine]

It’s not accounting for the fact that Erick Dampier has been out of the lineup, that Brendan Haywood has missed games due to injury, that Jason Terry underwent facial surgery, and that all the while, Caron Butler and Haywood are still integrating themselves into the system. Those aren’t critiques of Hollinger or his work, merely the system. They’re limitations he doesn’t try to hide or obscure in any way, just things that can’t really be taken into account mathematically.

Plus, looking to the methods again:

…I weigh a team’s full-season results by two-thirds and its most recent games by another one-third, so the overall ranking gives greater weight to recent games. You’re probably wondering at this point what I mean by “recent.” It varies depending on where we are in the season. For the first 40 games of the season, it means a team’s past 10 games. From that point forward, however, it means the most recent 25 percent of a team’s schedule. The net result is that, after the first 40 games, a team’s most recent 25 percent of its schedule will account for 40 percent of its ranking. [Ed. note: Again, emphasis mine.]

The last 25% of the Mavs’ games are more heavily weighted for the purposes of the rankings, and that’s a big part of the reason why Dallas is ranked so low. You’d think that the winning streak would off-set the rest of that 25% sample, but it just doesn’t. Dallas is 10th in point differential (+3.06) over that stretch, but the real kicker may be their strength of schedule: the winning percentage of the Mavs’ opponents over the last 25% of games is just .467, which ranks 23rd in the league. Given the weakness in Dallas’ point differential (10th) and strength of schedule (12th) overall, what did everyone honestly expect would happen in the midst of a 12-game winning streak in which all but one win have been by 10 points or fewer?

The Mavs are still a work in progress. We don’t yet have an accurate grasp of how the defense will perform with two healthy centers, or what kind of rhythm the offense can get into with all hands on deck. So the fact that Dallas is 9th in offensive efficiency and 12th in defensive efficiency, while hardly heart-warming, doesn’t really distress me. We likely won’t have an accurate macro statistical picture of this team going into the playoffs, because the data from earlier in the season is practically defunct given the roster moves, and the game data available offers a pretty small sample size.

These Mavs are contenders. They’re not on the level of the Lakers and are probably only on equal footing with the Nuggets, but that puts them in a position to do serious damage. I guess that’s where my views really diverge from Hollinger’s…or at least his views from about a week ago. All of the Mavs’ problems haven’t miraculously disappeared nor have all of their weakness morphed into strengths. But I think they’re good enough to beat the Nuggets in a seven game series, and good enough to give the Lakers a run for their money. If that’s not contending in the West, then what is?

  • finzent

    Nice to see someone not getting too upset about the whole Hollinger thing.

    Personally, I am not that annoyed by the Power Rankings (although his piece in which he defended the outcome of the Mavs in the ranking strikes me as very unconvincing). I have to say, though, that I was amazed by his assessment of the trade, which can only be described as borderline insane. I don’t have the link at hand, but I remember that he saw Butler/Haywood as an (“at best” I think his words were) marginal upgrade over Gooden and Josh, and that he conjured up lineup issues with the new personnel that made no sense at all.

  • Andrew

    I think most people probably aren’t as offended by Hollinger ranking the Mavs so low which, as you say, is automated, as they are that in the last month he’s written three columns either about how the Mavs’ trade didn’t help them at all http://insider.espn.go.com/nba/insider/columns/story?columnist=hollinger_john&page=hollingerwizardsmavs-100213 or how the Mavs are not contenders and destined for a fall http://insider.espn.go.com/nba/insider/columns/story?columnist=hollinger_john&page=PERDiem-100302 , and the one you listed above.

    If he’d every just say that his power ranking has flaws, that the Mavs should probably be rated higher, or simply refuse to address the issue with more than “It’s automated, but it’s pretty good”—rather than writing several columns about how he’s still right, despite what’s going on–it’d be much less annoying. ESPECIALLY, as Finzent says, about the trades. He used the trade machine to show the Mavs would get one win better after the trade—I myself have used that thing to get Lebron, D-Howard, and Durant for the Mavs and only improved by ten. Sometimes you have to use your eyes.

    I WOULD like to point out, as Hollinger might have, that the Mavs’ strength of schedule is better than 4 of the 5 times ranked directly ahead of them, probably indicative that strength of schedule in the last twenty five should be taken more seriously, even, than point differential, and that both 10th in point differential and 12th in strength of schedule are better than the 13th they’re ranked.

    Mostly, though, it’s an interesting problem. Hollinger is flat out wrong that the Mavericks, who are not second in the West, having beaten Hollinger’s 8,5,4 and 1 in the last ten are worse than the Bucks or the Trailblazers–that, as you say, probably the third best team in the West is so much worse than so many teams they’ve had no trouble with. That some one could BE so obviously wrong using only stats during the same week that’s kind of being viewed as the coming out party FOR stats (second coming out, or whatever), presents an interesting issue for the stat-head field.

  • Andrew

    Wow, that had beaucoup de typos, huh. Back to my cave…

  • Pingback: The Mid-Afternoon Milk Mustache, featuring the wrath of Bateer | Stacheketball, an NBA Blog

  • harry

    I think as logical as Hollinger is he’s human as well. When someone says ‘Hey! Behind the Bucks!?! Really?’ he takes that as a critism of his rankings and his science, and sometimes (just sometimes), he finds data to support his theory instead of finding a theory to support the data.
    I find that I like the work Hollinger does with his calculator but can find his articles a bit off putting. When someone points on something obviously odd (like the Bucks being better than the currently constucted Mavs)he really could just go ‘Yeah, huh? Strange, right? Here’s what the numbers say and why’ instead of getting all girlfriend wierd and defensive.

  • Andrew

    Totally with you, Harry. It’s a problem of his own making.

  • Sam

    Yeah totally don’t care about his rankings or playoff odds. But when he uses those rankings and stats as the sole basis of his articles and doesn’t take any thing else into account, I can’t stand it.

  • http://twitter.com/fatboyznodrog Aleks

    I mostly agree with Rob’s statement, but one thing is curious here: Over the last few years the Mavs have a record of winning more close games than statistically expected. In the extreme case (pure luck) the winning percentage in close games should be 50%, but the Mavs almost reach their overall winning percentage in these games (around 65%) – in contrast to all other contenders who are just slightly above 50%. This over several years, which means hundreds of games, is a significant trend. I would like to get a more detailed statistical picture on this, but it seems to me that the Mavs are an statistical outlier with respect to close games. The only theory I have to explain this is the perennial excellent free throw shooting, as FTs become increasingly important at the end of games.

  • Chris

    in my oppinion the formula used would become MUCH MORE precise when all margin issues would become viewed logarithmical (MAR10 = SUM ( Sign_i * ln (MAR_i) / 10; i= 1…10 –> Example 1 won game and 4 lost by Margins -3, -2, +20, -1, -2 => MAR5 of 0.5 while the usual methode would give a MAR of 2.4

    I’d like to explain why.
    In close games each contender does everything to win it while in big margin games (15 and more) the loosing team usually has given up while the winning team has run hot. Therefor in my eyes the big number wins should count a BIT less, yet they should count.

    In my eyes this view is better


  • http://espn Richard

    I love how angry Mavericks fans are at Hollinger over the rankings. Have the Mavericks won a championship after having been ranked low on Hollinger’s rankings? No they haven’t. The teams that have been in the upper portion of Hollinger’s rankings have ended up in the NBA finals or the Conference finals. The Mavericks have had decents records, but poor performance in the playoffs (2006 not withstanding). In the last 10 years the Mavericks have averaged 56 wins per year, yet have only 1 NBA finals appearance, and 1 conference finals appearance. The other strong teams in the west over the last decade have done much better with a similar win totals.

    Spurs 2 conference finals appearances, 3 NBA finals appearances.

    Lakers 6 NBA finals appearances.

    So the Mavericks have performed worse than their regular season record would indicate. And Hollinger’s rankings do a great job of showing the best predictor of future success. It isn’t the team’s record, but the point differntial.

    So if the Mavericks go on to win the title, or at least reach the WCF then you can talk about how the Mavericks are being treated unfairly. Until then quit crying and hope the Mavs prove Hollinger’s rankings fallible.

  • Clay

    I’m sure the author has much more knowledge of statistics and trends than I do but I do have a couple of questions regarding his “close game” logic. First off, are all close wins created equal? I mean, is a come from behind win treated the same as a back and forth struggle or a game where you jump to a large lead then rest your starters? There are several ways to a close game and I’ve always felt broadcasters were right when they referred to some contests as “not as close as the score would indicate”.
    Also, I was brought up on the idea that teams that can win in close games display a type of discipline and mental toughness that translates well in the playoffs. I’ve always felt that close wins were nothing to worry about as long as you were able to beat the top tier teams, which the mavs definitely have. I’d also be interested if anyone knows anything about the margin of victory for some of the other great teams from recent seasons. I’d be shocked if they were all high.

  • Toby

    I’m with you, too, Harry. I would love Hollinger to write a piece talking about the limitations of stat-based analysis, but instead he chooses to defend just about eveery outlier. Too many time he just assumes his own conclusions when making an argument–”of course Caron Butler isn’t a big improvement over Howard. Just look at their PER numbersI’m with you, too, Harry. I would love Hollinger to write a piece talking about the limitations of stat-based analysis, but instead he chooses to defend just about every outlier. Too many times he just assumes his own conclusions when making an argument–”of course Caron Butler isn’t a notable improvement over Josh Howard. Just look at their PER numbers and adjusted plus/minus.” Blah.

    I also think he’s been awfully quick to judge this new-look Mavs team. They’ve gone 13-1 since the trade with a point differential of 6.2, putting them in the same neighborhood of other top teams. I understand that he has to write something every day and that plays a big role in what he chooses to write and when to write it, but he can’t claim to be Mr. Objective in one breath and then evaluate teams without a decent-sized set of data.
    and adjusted plus/minus.” Blah.

    I also think he’s been awfully quick to judge this new-look mavs team. They’ve gone 13-1 since the trade with a point differential of 6.2, putting them in the same neighborhood of other top teams. I understand that he has to write something every day and that plays a big role in what he choosese to write and when to write it, but he can’t claim to be Mr. Ojbective in one breath and then evaluate teams without a decent-sized set of data.

  • Sridhar

    The way I see it is you can never satisfy everyone. Just have to prove it continuously until you win the championship.

    Personally I do accept that winning in the playoffs is much much greater than the regular season. However, I am glad that Mavs average 56 wins per season for the last decade rather than be a perennial loser.

    Coming back to the win streak and the margin of victory, lets assume we beat teams and end up in blow outs, then they will pick something else. They will say “Lets see how they do in close games” or If we beat a team in December, then they will say lets see how we are playing in March/April when its close to playoff time.

    Some of these may be valid, but the end fact is to each his own.

    I am rooting for them to make some noise this playoffs, statistics be damned.

    Rowdy Proud and Loud. Go Mavs Go!!!!

  • Pingback: Defense Is Our Backbone » The Two Man Game

  • http://twitter.com/PitchBlackMind Brian D

    I agree with most of the above posts and some of Rob’s analysis. Like others who have commented, my problem isn’t so much with Hollinger’s stats (they are what they are, and he explains them), it’s his blind defense of them.

    What sane person could think (and then publicly defend) the Butler-Haywood trade as lateral move?

    I know you probably have an obligation not to insult ESPN personalities, being on the TrueHoop network, but come on Rob.