clock menu more-arrow no yes mobile

Filed under:

Jim O'Brien: An attempt to measure coaching effectiveness.

One of the "Holy Grails" of basketball statistics is a metric which can isolate and measure the effectiveness of a coach. This is an extremely difficult proposition because of the incredible number of variables. Other than Wins and Losses, coaches don't directly produce any quantifiable statistics. Almost everything that can be measured is produced by players, and obviously a player's production can't be traced or attributed directly to coaching. Setting aside the lack of accurate metrics, statistician's even disagree on the basic premise that coaches have any effect on their team's production at all. For the curious, an interesting post by Ryan McCarthy at Slate explains this disagreement in more detail. Despite these challenges, the question of a coach's effectiveness is one that needs to be asked repeatedly. Although there may not be a satisfactory answer, there is data available which can give us some indirect evidence.

This brings us to the case of Jim O'Brien. O'Brien has just finished his 3rd year as head coach of the Pacers. Each season ended in disappointing fashion, with the Pacers finishing outside the playoffs. A lot of voices from the fan base called for his firing during the season. In spite of this and the team's record, he received a vote of confidence from Larry Bird last week, who affirmed that O'Brien would be the Pacer's Head Coach, at least until his contract expires at the end of next season.

Although he has brought a few positives, I have a fairly negative view of the job O'Brien has done with the Pacers. Except for a few hiccups this season, I think he has handled a difficult situation with poise and professionalism. He also is not working with the most competitive roster in terms of talent. That being said, he has not gotten the job done. We have missed the playoffs three years in a row, despite one of the most top-heavy eras the Eastern Conference has seen (It's not like we are missing the playoffs with 48 wins like in the Western Conference). The development of young player has been stagnant. The team simply hasn't improved during his tenure.

Of course these assertions are based mostly on my personal observations. For a true, honest and complete examination of the job Jim O'Brien has done, we will need to look at as much information as possible. There is quite a bit to look at, so jump at your own risk . . .


 Not being able to rely on a specific metric, I had to put together several different statistics. Below is a Table of Contents of what I included..

  1. Team Stats (Win-Loss, Efficiency Differential, Expected Wins) for O'Brien coached teams.
  2. Statistics for Plays Run out of Timeouts
  3. Blown Leads/Comebacks
  4. Substitution Patterns
  5. Player Development
  6. Summation

 1. Team Stats (Win-Loss, Efficiency Differential, Expected Wins) for O'Brien coached teams


  • To get a more detailed picture of O'Brien's effectiveness I decided to include his career coaching stats.
  • The * represent years where he only coached part of the season.
  • PW and PL stand for Pythagorean Wins and Losses. This is a method for projecting a Team's Wins and Losses using the amount of points they scored. The PDiff. category compares Pythagorean Wins to actual wins. A positive PDiff. means the team won more games than they should have, a negative means the opposite. Check out Basketball-Reference for more information. All three seasons with the Pacers they have won fewer games than predicted. Since this formula is based on Points Scored, the implication would be that they are losing a few close games each year because of execution.
  • When looking at these statistics I wanted to add some measure of the quality of the players on each team. I came up with the idea of averaging the PER's for each of the players on the team. I multiplied each player's PER by the number of minutes played. I added all of these totals and then divided it by the total number of minutes played by the entire team. I haven't seen anything like this before, and to be honest I am not sure how statistically valid it is. To put these numbers into context, the PER of an average NBA player is 15.00. Using my method, the average PER for the Cleveland Cavaliers this season was 16.40. If you are interested in the spreadsheets you can view them here. Average PER

The picture these stats seem to paint is of average results from slightly below average teams. I have to admit that I had mostly forgotten the moderate success O'Brien had in Boston and Philadelphia. It seems like those were teams that overachieved with average rosters, but that has not been the case in Indiana. The team has posted a negative Rating Differential all three seasons under O'Brien. The other memory jogger was that several of O'Brien's teams were very good defensively. Despite his current reputation as an offensive minded run and gun guy, he has never had a team finish in the Top 10 in Offensive Rating. He has however had 3 teams finish in the Top 10 in Defensive Rating.


2. Statistics for Plays Run out of Timeouts

The idea to include this data came from a Bulls By The Horns post. Chicago fans have been similarly frustrated with the perceived X's and O's deficiencies of Vinny Del Negro. This post mentioned the idea of examining all of the plays Del Negro and the Bulls had run out of timeouts. I decided to try the same thing for O'Brien and the Pacers.
I went through and coded the results of each play the Pacer's ran coming out of a timeout this season. I also included plays they ran at the beginning of a quarter. If a timeout was called on a foul but before a player shot their free throws, I would include the offensive play that came right after the free throws. Now not all of these instances were set plays that were directly called by O'Brien. My thinking is that this represents a large sample size, of situations were O'Brien's impact would be the largest on the offensive set. If he wasn't calling a specific play I would assume he would be giving some general offensive directions, "we need to attack the basket," "let's get back to the pick and roll," etc. This should represent the scenarios were his impact on the offensive decision making of the players on the floor would be the most direct. For the sake of comparison, I also tracked the same information for the Pacer's opponent in each game and included that data as well.


  • Collecting this data took a considerable amount of effort and time; and I really had no idea of what I was going to find. The numbers raise some questions, but don't look nearly as bad for the Pacers as I was expecting.
  • The first thing which caught me off guard was the Points Per Possession found in the third table. The numbers for the Pacers and their opponents seem unbelievably low. The fact that the Offensive Ratings (Points per 100 possessions) were so low was really surprising. The lowest in the league this year was 98.1, put up by New Jersey. Usually this statistic is calculated using a formula to gauge the number of possessions. Since I actually had the number of possessions counted, I just used that. Assuming I haven't made some significant error in reasoning or data collection, which is not outside the realm of possibilities, this looks great for O'Brien and the team. A +8.8 difference in this rating is significant. The highest difference between Offensive Rating and Defensive Rating this year was a +9.25 put up by Orlando.
  • The Pacers turned the ball over at about the same rate as their opponents, and had their baskets assisted on at about the same rate as well. Where this seems to make the Pacers look great is at the defensive end. Setting aside a 62.5% FG% allowed at the rim, they held their opponents to very low percentages from everywhere else on the floor. I am obviously jumping to conclusions here, but it looks like the Pacer's defense was much more effective when it had a chance to recover, reset and get into position.
  • The most disappointing part of this data was that the Pacers took nearly 50% of their shots in these situations from 16ft. or further. An especially distressing fact when their 30.1% 3PT% is factored in. Whether by design or ignorance, this seems to be a huge indictment of Jim O'Brien's ability to get his team running efficient offensive sets. I know this isn't a surprise to anyone who watched the Pacers this year, but it is a little more striking with some numbers to reinforce it.
  • As I collected this data, I also coded the results by player. Hopefully, by next week I can finish collecting some tables to look at each player's performance in these set offense scenarios. If anyone is interested in looking at the raw data for specific games or performance against specific teams you are more than welcome. Pacers Time Out Offense

3. Blown Leads/Comebacks

In early 2006, Bill Simmons wrote a column lamenting the coaching failures of Celtics Head Coach, Doc Rivers. In his column he proposed creating a "Bad Coaching Index." The statistics he proposed including were ones he felt were particular weaknesses of Rivers'. The folks over at 82games jumped right on it, did some serious number crunching, and put together a 2-part analysis. (Part 1, Part 2). Unfortunately, these numbers are all for the '05-'06 season, when O'Brien wasn't a head coach. However, I borrowed a few ideas from their analysis for the next two sections.

The first is to look at blown leads and comebacks. I only looked at the 4th quarters and overtimes of Pacers games. The data is separated by how large the lead or comeback was.


  • With a small amount of data, and not much to compare it to, these numbers don't tell us a huge amount.
  • The Pacers did have an impressive number of comeback wins this season. However, most of them were small deficits, and often came after they had already surrendered a lead themselves.
  • The most distressing number here is that 26% of the time they Pacers had a lead in the 4th quarter, they went on to lose the game. I imagine similar data from last season would be even more depressing. Although they have generally played hard, under O'Brien the Pacers have consistently failed to execute down the stretch of games the last 3 seasons.

4. Substitution Patterns

This is another statistic which comes from the Bill Simmons Bad Coaching Index. The Pacers used 400 different 5 man lineups this season (this number includes the variations in the positions the players played.) Obviously this number is heavily influenced by the rash of injuries the Pacers faced this season. Here is a look at the Pacers compared to the rest of the league.


As I said above these numbers are heavily influenced by injuries, as well as mid-season trades and free agent signings. When put these numbers together for the 05-06 season they did see some connections

"So first of all it's only fair to point out some obvious things: teams that make trades or have serious injury issues are much more likely to use lots of different 5-man units. Likewise, teams with lots of depth or bad teams trying to figure out a winning lineup with what they've got on the roster will experiment more often. Still when you check the units used against winning percentage, you'll find a -.39 correlation, which is a reasonably strong way of saying that more successful teams tend to use fewer different units. "

  • There are a lot of factors which affect this list. But the 9 of the 10 teams which used the fewest lineups this season all made the playoffs.
  • This number for the Pacers is certainly inflated because of the injuries suffered by Danny Granger, Tyler Hansbrough, Jeff Fosterand others. I still think there is some room to criticize Jim O'Briens substitution patterns. O'Brien stuck with T.J. Ford much too long, even admitting as much himself late in the season. Minutes for Dahntay Jones dried up in the middle of the season because he wasn't a good enough shooter to space the floor. When it became apparent the team wasn't any better with a "spaced floor" his minutes came roaring back. Beyond injuries, there just never seemed to be a consistent rotation with the available players. This may have made us slightly more competitive, but I think it hurt our player development. Segue . . .

5. Player Development

For me, this has been the most frustrating aspect of O'Brien's coaching tenure. At least 75% of being a fan is the hopes you carry for your team, be they rational or irrational. Almost all of those hopes are pinned to the hypothetical development of specific players. Can player X take the next step and become a team leader? Can player Y develop into a reliable 3 point shooter? Can player JS stop taking 3 pointers altogether? With a few very small exceptions, these player development hopes for the Pacers haven't materialized under Jim O'Brien. I understand that the responsibility for skill development is shared with the rest of the coaching staff and the players themselves. Still, O'Brien is ultimately responsible for making sure his players improve and for putting them into situations which maximize their skills. Below is a table of some players who have joined the Pacers under O'Brien's tenure. I included their Pacers stats and their stats for the last season with their previous team.


  • Nesterovic and Rush the Elder were two players who came in and performed reasonably well for O'Brien. Kareem Rush was a decent shooter and got the opportunity to fling up long jumpers to his heart's delight.
  • Jones and Watson are two other players who joined the Pacers as veterans. At that point in their careers not much was expected from either in terms of skill development. They are both players with defined strengths and weaknesses and it was up to O'Brien to make the most of them. Watson did a serviceable job as a starting point guard. I still find it troubling that despite playing a starting role and handling the ball much of the time, his Ast/40 and A/TO ratio declined. Partially by necessity, and partially from a loosely structured offensive system Jones was given way to much freedom to shoot long jumpers.
  • The real troubling area is the development of Rush the Younger and Hibbert. Rush's troubles this year have been well documented and discussed. He becomes passive on offense. He drifts on the perimeter. He passes up open shots and struggles with his confidence. It seems to me that very little intentioned effort was spent this year to put him into specific situations where he would be successful. He struggled with passivity and confidence and the solution is for him to find his shot in the flow of the offense? This seems like a recipe for disaster. O'Brien should have been calling plays designed just for him. Lots of them.
  • I have basically the same complaint about the way Hibbert has been handled. Hibbert's per game numbers jumped this year mostly due to a drop in his foul rate and a resulting increase in his minutes. This is an accomplishment for which Hibbert and the coaching staff should be commended. The problem is that his per minute numbers completely stagnated. He actually attempted fewer shots per minute this season, and despite an increase in assists, his usage rate declined as well. When he was a rookie struggling to stay on the floor, a concerted effort was made to get him opportunities in the post. Now that his fouling challenges have been met, I guess we can take him for granted and not worry about feeding him in the post?
  • To me the deficient development of both Hibbert and Rush can be largely traced to tactical errors in the method and frequency of how they are utilized during a game. Unfortunately, this ends up in O'Brien's lap.

6. Summation

When I started collecting stats for this post I expected all the numbers to reinforce my own conception: That Jim O'Brien was a terrible coach and should be fired before next season. The truth is apparently more complicated. In his three years in Indiana he has been working with below average teams in terms of talent. Despite that he has gotten the Pacers to play hard and on occasion play really well. They have started each season badly, but never quit on him and finished each season strongly. O'Brien has also had a hand in changing the perceived culture of the organization from one of gun toting-thugs to hard working over-achievers. Except for a few poorly chosen comments to the media this year, he has been professional and positive about his team and their situation. I also understand, with the teams current financial challenges, how unpalatable it would be to let him go now and pay the salaries of two head coaches next season. 

Despite the positives, I think he has not done everything the job required of him. Their has not been sufficient development of young players. There is still too much disorganization and chaos on the floor, especially on the offensive end. Too often tactical advantages and opportunities are passed or ignored. In my heart I don't think he is a coach who is can lead the Pacers to a championship and I don't want to waste even one more year floundering for the sake of finances or stability.

I think we expect O'Brien to be fired because in the current NBA climate, it is so rare for a coach to be given extended opportunities to build success. You win or you're gone and that hasn't been the case with O'Brien. I have accepted the fact that he will have one more opportunity to lead this team. Whether he is merely a keeping a seat warm for our next coach, or about to find his stride, I wish and hope for him to be successful this summer and next season. I honestly expect more of the same next year, but hold out hope that things can and will be different. For better or worse he will lead the Pacers into the upcoming season, and for the sake of the team and fans alike I wish him the best.