The PCA (DIPS-based) pitching metric.

[SABR Matt]

Given the level of skepticism and otherwise interest in how I get my pitching rankings/ratings, I've decided to post the exact methodology I use to evaluate pitchers in the interests of seeking feedback and full disclosure of the method.

This isn't a "simple" method so it's going to take a bit to explain. If you're interested, please do bare with me - I'll give a concrete example pitcher record to illustrate.

Here is the pitching record I will analyze:
Tom Glavine

Focus your attention on his 2001 season. It's fairly instructive so we'll use it.

STEP ONE - PURE DIPS ANALYSIS
Statistics of interest at this point in the analysis:

IPOuts - 659
H - 213
K - 116
BB - 97
HR - 24
HBP - 2
WP - 2
BK - 0
PkO - 9
A - 40

From the IPOut, H, HR and K data, we know that Glavine put 732 balls into play (IPOuts + H - K - HR).

I track pitcher assists because I am working on the assumption that if a pitcher gets a "touch" on defense, the ball must have been hit poorly and that must be considered a success for the pitcher. The BABIP is probably WAY lower for balls which a pitcher could potentially make a play anyway. Note that pick-offs almost always result in an out...a pitcher-assisted out, so from this point forward, we consider pickoffs as part of pitcher assists.

In 2001 in the NL, pitchers recorded 2841 assists on 67240 balls in play. The Natioanl League batting average on balls in play when not fielded by the pitcher is .312 so given normal skill at preventing BIP hits and an average defense we expect Glavine to allow 216 (.312 * 692 BIP not fielded by Glavine) in play hits.

Doubles make up a little less than 23% of the in play hits, so 50 of his 216 in play hits shuold be 2B.

Triples make up 2.4% of the in play hits in 2001 in the NL, so 5 of his 216 hits should be 3B.

That leaves 161 singles.

We also have to adjust his HRA to account for the impact Turner Field had on HR rates. I calculated a 5-year weighted park-HR adjustment of 1.00296 for 2001 (that's including all of the other parks in which the Braves played), meaning that the net average park in which the Braves played their games slightly increased the odds of a HR relative to the rest of the NL.

If we divide his 24 HRs by 1.00296 we get 23.9 or essentially still 24 so no change there.

With an expectation to allow (216-189) 27 more hits than he actually allowed, he now has 27 fewer defense-independent IPOuts.

That gives him this defense independent line:

IPOuts - 632
1B - 161
2B - 50
3B - 5
HR - 24
BB - 97
K - 116
HBP - 2
WP - 2
BK - 0
PkO - 9
A - 40

STEP TWO - ACCOUNTING FOR REAL IMPACTS ON BALLS IN PLAY

How much of the difference between league average BIP hit rates and Glavine's actual performance is the Braves' defense and how much is Glavine?

The Braves as a team allowed 4419 balls in play, of which 190 were fielded by pitchers for an assist leaving 4229 BIP of interest and 1210 in play hits. But Glavine affected that figure by his own pitching, so we need to take out 692 non-pitcher-assisted BIP and 189 BIP hits leaving us with these team totals:

3537 BIP
1021 In-Play Hits
.289 BABIP on non-pitcher-asisted BIP

If Glavine were league average, pitching in front of this brilliant team defense, he'd record (.289*692 BIP) 200 in-play hits. He allowed 11 fewer than that.

Now we need to "give back" some of the hits we added above.

8 singles and 3 doubles get taken off his existing defense independent line to accoutn for the fact that he shaved 11 hits off the board.

That leaves his hit counts and out counts at:

IPOuts - 643
1B - 153
2B - 47
3B - 5
HR - 24

The rest of the stats stay the same.

STEP THREE - TURNING THE NEW LINE INTO RC

Pitcher Assists are treated as automatic outs (given the usual linear weight for an out).

Now I used less accurate LWs in the original PCA analysis than the ones I just finished calculating that were based on multi-linear regression, but I expect to implement this methodology with accurate LW data in the near future so I'll use the better LW data in this example.

In 2001, the following LW existed in the NL:

In-Play Out (this includes both FC and straight out) - -0.257
K - -0.281
1B - 0.460
2B - 0.771
3B - 1.060
HR - 1.370
BB (including both unintentional and intentional) - 0.291
HBP - 0.328
WP - 0.279
BK - 0.292

To get RA (above average) for Glavine, we just multiply the number of each of these events by their LW:

In-Play Outs: (643 - 116) = 527 * -0.257 = -135.44 +
K: 116 * -0.281 = -32.60 +
1B: 153 * 0.460 = 70.38 +
2B: 47 * 0.771 = 36.24 +
3B: 5 * 1.060 = 5.30 +
HR: 24 * 1.370 = 32.88 +
BB: 97 * 0.291 = 28.23 +
HBP: 2 * 0.328 = 0.66 +
WP: 2 * 0.279 = 0.56

Add those all up and you get 6.21. In 2001, this method asserts that Tom Glavine was 6.21 runs allowed worse than the national league average pitcher (a far cry from his defense-assisted 124 ERA+).

If you want an ERA-like number, you can convert 6.21 RAAA to a total RA metric by knowing that the NL RA/Out is 0.176 (that does include errors) which means in Glavine's defense-neutral 643 IPouts he should have allowed 113 defense-neutral runs (we're expecting a league-average rate of runs produced by errors here) and our estimate is that he actually allowed 119.21 defense-neutral runs for a DNRA (defense-neutral run average) of 5.01 (.186 * 27 outs) against a league average DNRA of 4.75.

His DNRA+ would be (4.75 / 5.01) * 100 or 95. A slightly below average pitcher in 2001.

Two notable flaws with this that I aim to correct in future renditions where data allows.

I didn't adjust for the pitcher's tendency to generate groundballs/flyballs and the expected changes that produces in the average result of a ball in play, especially the change in the double play rate.

I didn't account for the pitcher's impact in extra base hits on balls in play compared to his team's extra base hit rates. I didn't originally have that data, and I still don't have it prior to 1957 so the standing assumption must be that all pitchers allow the same rate of extra base hits per in play hit prior to 1957. In the PBP era I can change the method to account for sensible impacts on ball in play XBH rates, though I expect those tweaks to be small most of the time.

Thoughts?

Comments?

Criticisms? Any ideas you have that might improve the method, I'm open to suggestions.

[misterdirt]

Here's Glavine's actual line for 2001:

495 outs
116 K's
9 SB's
7 CS's
3 PkO's
2 WP
87 BB's
10 IBB's
2 HBP's
5 Error's on play
1 FC
156 1B
32 2B
1 3B
24 HR
28 DP's

Total BIP's is 714 not 732. The difference is due to not counting double plays. His actual value is a pitcher is going to be much higher than you calculated for several reasons. He has lower WP and PB than the average pitcher. He has a much lower success rate for SB than the average pitcher. He has much lower rate of XBH than the average pitcher. I haven't calculated the exact number but I suspect he also has a higher DP rate than the average pitcher.

I know you are trying to create a methodology that you can use for the years that you don't have PBP data, but you should calculate some of the years that you do have PBP data for both ways; using the actual numbers and your estimated numbers. That should show you the range of error that your estimates can create. Then you can try to fine tune your estimates to lower that error.

[SABR Matt]

That is in the plans MrD

The XBH though you need to be careful, becasue the Braves' defensive strength during this time period was their outfield. Andruw Jones was a one-man double/triple wrecking crew.

The way I intend to handle XBH in the PBP era is by doing exactly what I did for in play hits...placing Glavine in an average context, then adjusting for his actual impact on XBH compared to how the Braves did in that regard when he was not pitching. The same will have to happen for DP% (per ball in play), because the Braves also turned a mean double play during that time period...not necessarily because of pitching skill entirely.

[SABR Matt]

Success rate on SB is the other thing I haven't yet included, and that would need to be adjusted relative to the team as well. Some pitchers might have a low SB% because their catchers are really good at throwing out runners, although Glavine is probably not one of them since his catchers have generally sucked.

All of these things that are not pure-DIPS must be team-relative because the team is a biasing factor that needs to be considered.

[SABR Matt]

Hmm...I just noticed your pickoff number is different than mine...trying to determine how I got the wrong number in my database...that's not good.

[Tango Tiger]

STEP 3:

In 2001, the following LW existed in the NL

If you are going for precision, the LWTS is not what you should use for pitchers or teams. BaseRuns is. (Or better yet, Markov.) If Clemens gives up 2.5 runs per game, it doesn't matter what the 4.50 RPG pitcher did in the same season. You want to know how a 2.5 RPG pitcher would do, given the base-advancement probabilities of that season.

I understand why you want to use LWTS. But, it's wrong. That said, you'll only be off by a little bit.

***

As for pickoffs, it depends how you treat it. I treat each of the Retro events as mutually exclusive. It's either a SB or CS or PK.

However, some people don't do that. Read this thread, plus the follow-up comments:

http://www.insidethebook.com/ee/inde...the_cs_and_pk/

[SABR Matt]

Well the LW are based off of the markov chain for an average league...

And no, I don't think it's appropriate to use the pitcher himself to establish the run environment to use to diagnose the pitcher himself. That's like statistical incest, Tango. If you do that, all you will get is a QED..."gee...Tom Glavine gave up 3.6 R/G in 2001...based on a 3.6 R/G environment, he should...GIVE UP 3.6 R/G!" Perhaps you can explain how this is not true, because it seems to me that if you're using a pitcher's run allowed data to diagnose the pitcher, it's circular logic.

The point of this pitching analysis is to determine how a pitcher would fare if plunked into a completely average environment...how he himself would impact the league.

[SABR Matt]

And in the PBP era, I actually intend to treat Pickoffs separately...prior to PBP, we have pitcher assist totals which I can use to "cover" pickoffs...the methodology was developed to handle pitchers from any era, but obviously some fine tuning can occur where data allows.

[Tango Tiger]

That's not what I'm talking about. The run value of Mussina's singles is not the same as Jeff Weaver, even if they are both pitching at Yankee Stadium. If you decide to plunk them both to 1968, you don't use the 1968 run value of a single for each of them.

To get the runs per game value for Mussina, you need to enter his inputs (hits, hr, walks, etc) into the Markov chain, and let that tell you what his RPG is. You neither need, nor want to know, the LWTS values.

[misterdirt]

Your big error is not allowing for DP's and OOB's in calculating BIP. You have to create a factor for that to use when you don't have PBP data. Also, when you do your comparisons you shouldn't use 0 to represent an average pitcher. Starting pitcher's should be compared against the average for other starters and relievers against relievers.

Only 31 of Glavine's assists were on non bunt ground balls to him. Retrosheet also credits him with 62 total assists. I don't know why there is a difference.

Handedness of pitcher is going to have an effect on the percentage of hits that are triples.

IBB's are an available stat without PBP data for 2001 so the actual numbers should be used.

I think it is wrong to be adjusting for pitcher assists. You have no way of knowing whether the balls hit to a pitcher were easy grounders or hard smashes, just as you have no way of knowing whether the outfield PO's are easy fly balls or line drives. The ability of the pitcher to induce "easy" BIP outs is always going to be an unknown factor for the data you have available to you.

If Glavine were league average, pitching in front of this brilliant team defense, he'd record (.289*692 BIP) 200 in-play hits. He allowed 11 fewer than that.

You don't know this because you haven't calculated this. You have calculated the team defense rate of BABIP (.289) not for a league average pitcher, but for Atlanta's pitching minus Glavine.

[SABR Matt]

That's not what I'm talking about. The run value of Mussina's singles is not the same as Jeff Weaver, even if they are both pitching at Yankee Stadium. If you decide to plunk them both to 1968, you don't use the 1968 run value of a single for each of them.

To get the runs per game value for Mussina, you need to enter his inputs (hits, hr, walks, etc) into the Markov chain, and let that tell you what his RPG is. You neither need, nor want to know, the LWTS values.

Oh I see what you're saying...

Well now that's interesting...to take the luck element out though, wouldn't you need to enter average ratios for state-to-state transitions for each event type? Like Maddux might have gotten lucky one year and instead of 30% of his singles with a man on first moving runners from first to third, only 15% did that. (just pulling that percetage out of my rear end) I would think I would need to come up with league-average transition probability matrices for each event type and then multiply each of them by the probability of each event occuring when a pitcher is on the mound and then add them together.

[Tango Tiger]

Right, you would need to have those state-to-state transition rates. In fact, that's how I come up with Tables 8 and 9 in The Book. I keep the state-to-state transition rates constant for each event. I simply modify the frequency of the events.

It's not totally accurate, since if you are in a low run scoring environment, you will take more chances on taking the extra base, as opposed to playing it safe if you are in a lineup filled with boppers. So, if you are crazy, you could modify the state-to-state transition rates based on the expected run environment (which would put you in a circular logic).

[SABR Matt]

Your big error is not allowing for DP's and OOB's in calculating BIP. You have to create a factor for that to use when you don't have PBP data. Also, when you do your comparisons you shouldn't use 0 to represent an average pitcher. Starting pitcher's should be compared against the average for other starters and relievers against relievers.

Only 31 of Glavine's assists were on non bunt ground balls to him. Retrosheet also credits him with 62 total assists. I don't know why there is a difference.

Handedness of pitcher is going to have an effect on the percentage of hits that are triples.

IBB's are an available stat without PBP data for 2001 so the actual numbers should be used.

I think it is wrong to be adjusting for pitcher assists. You have no way of knowing whether the balls hit to a pitcher were easy grounders or hard smashes, just as you have no way of knowing whether the outfield PO's are easy fly balls or line drives. The ability of the pitcher to induce "easy" BIP outs is always going to be an unknown factor for the data you have available to you.

If Glavine were league average, pitching in front of this brilliant team defense, he'd record (.289*692 BIP) 200 in-play hits. He allowed 11 fewer than that.

You don't know this because you haven't calculated this. You have calculated the team defense rate of BABIP (.289) not for a league average pitcher, but for Atlanta's pitching minus Glavine.

Let's take some of these one at a time.

I agree a factor needs to exist for DPs and Outs on base...I need to find a way in the pre-PBP era to estimate DPs based on the team GIDP rate and outs on base based on a typical rate that runners advance into outs. Remember...I didn't have any of this PBP data to work with originally, so it's still a work-in-progress, which is why I posted the method here to get feedback.

As for comparing pitchers to "zero"...that center point is just generated by the truth of linear weights...they always sum to zero for an average situation. Converting pitching skill to wins created though, I would not be using average...I would be converting RAAA into component RA and finding the margin and comparing to the margin.

I have no idea why retrosheet gives Glavine 62 A while baseball-reference only gives him 40...that's highly disturbing. Rest assured though, with PBP data in the database now, I would use the (more likely to be accurate) PBP counts of assists (and only assists that were not sac bunts, since I have sac bunt events, and only assists that were not pickoffs since I have pickoff data and only assists that did not start double plays since I have double play data and will be accounting for that separately...I can query for pure assists...not a problem).

Handedness of a pitcher isn't really an issue here...even if it's true that lefties give up fewer triples or whatever the truth is about the typical lefty, we care only about how Glavine does relative to his peers in this regard. Even if his lefthandedness aids him, that gives him real value over other pitchers and is a measurable part of his skill set.

Where are you getting IBB data for pitchers (outside of the PBP universe)...it's not in the baseball-databank as far as I know unless it was just added in the 2005/12/31 update since I haven't studied that database in as much detail yet.

I don't agree with your assessment of pitcher assists. It's true that pitchers get some asists that are hit hard, however I noticed when I went through the last iteration of pitcher evaluation that pitchers who rated as great fielders also tended to be pitchers who were control artists and who tended to show some ksill at preventing hits on balls in play. As a rule, great fielding pitchers seem to induce bad contact, and I strongly believe this can be captured by looking at pitcher assists.

And yes, I'm aware of what specifically the team-relative stats are actually calculating. There is the possibility for some error using team-relative ball in play results, but it's a hell of a lot closer IMHO than using some really ugly rough estimate of how much a pitcher controls BIP events (I see a lot of people just happily using 20% as though it were true of every pitcher). Yeah...The Braves had a group of pitchers who tended to help with the BIP events in the 90s, but this is rarely true and as I've already shown in other places here at the Fever, the differences are fairly small in a single season if you start pulling out other pitchers. The Braves' pitchers minus Glavine give you something close enough to an average sampling of pitchers in the NL that I'm happier doing that than using some rough estimate.

[SABR Matt]

Right, you would need to have those state-to-state transition rates. In fact, that's how I come up with Tables 8 and 9 in The Book. I keep the state-to-state transition rates constant for each event. I simply modify the frequency of the events.

It's not totally accurate, since if you are in a low run scoring environment, you will take more chances on taking the extra base, as opposed to playing it safe if you are in a lineup filled with boppers. So, if you are crazy, you could modify the state-to-state transition rates based on the expected run environment (which would put you in a circular logic).

I'll have to take a look at doing that...the only thing is...in order to do it, I'd need to write code to go through every pitcher...I did my league markov calcualtions in excel...one league at a time. 96 calculations were painful. 20,000 calculations requires a C++ program that gets data using MySQL code interface, does the matrix math, and spits out an answer for every pitcher/season in the PBP database...

I have not one idea how to do that.

[misterdirt]

If you are using the Markov as Tango suggests you wouldn't you still have to calculate expected hit types for each BaseOut state instead of using a pitcher's actual singles, doubles, and triples to calculate the state to state transition rates? Otherwise you wouldn't be creating a defense independent pitcher rating.

[misterdirt]

Where are you getting IBB data for pitchers (outside of the PBP universe)...it's not in the baseball-databank as far as I know unless it was just added in the 2005/12/31 update since I haven't studied that database in as much detail yet.

Mlb.com stats give IBB's for pitchers in 2001.

The Braves' pitchers minus Glavine give you something close enough to an average sampling of pitchers in the NL that I'm happier doing that than using some rough estimate.

There is no logical reason to believe that the Braves pitching - Glavine is any closer to the league average than the Braves pitching with Glavine.

I have no idea why retrosheet gives Glavine 62 A while baseball-reference only gives him 40...that's highly disturbing.

Retrosheet lists assists for any player involved in the play. There may be some rule limiting what is considered an official assist.

[SABR Matt]

If you are using the Markov as Tango suggests you wouldn't you still have to calculate expected hit types for each BaseOut state instead of using a pitcher's actual singles, doubles, and triples to calculate the state to state transition rates? Otherwise you wouldn't be creating a defense independent pitcher rating.

Yes...by "actual stats" I meant the stat line I come to after all of my defense-independent calculations.

[SABR Matt]

Where are you getting IBB data for pitchers (outside of the PBP universe)...it's not in the baseball-databank as far as I know unless it was just added in the 2005/12/31 update since I haven't studied that database in as much detail yet.

Mlb.com stats give IBB's for pitchers in 2001.

The Braves' pitchers minus Glavine give you something close enough to an average sampling of pitchers in the NL that I'm happier doing that than using some rough estimate.

There is no logical reason to believe that the Braves pitching - Glavine is any closer to the league average than the Braves pitching with Glavine.

I have no idea why retrosheet gives Glavine 62 A while baseball-reference only gives him 40...that's highly disturbing.

Retrosheet lists assists for any player involved in the play. There may be some rule limiting what is considered an official assist.

The logic in my belief that braves - glavine = something close to league average for BIP results comes from the entire logical foundation of DIPS. That foundation being that pitchers have far less impact on the events on balls in play than the defense behind them. It is natural to conclude if you accept that fundamental logical basis (and there is a mountain of data that supports it) that the Braves' team defense is going to come through much more strongly than any minor effects other Braves pitchers might have on the BIP results.

Hmm...retrosheet gives an assist to every touch of the ball on a play? That's going to create all kinds of problems if there is in fact a rule preventing some of those touches to count for official assists.

[misterdirt]

The logic in my belief that braves - glavine = something close to league average for BIP results comes from the entire logical foundation of DIPS.

I understand that Matt. What my post was saying was that there is no logical reason for subtracting Glavine from the Braves pitching. Subtracting his stats doesn't make the results more accurate than leaving him in.

[misterdirt]

Also, some parks are going to generate higher BABIP than others and some are going to skew those results due to the handedness of the pitcher so to be accurate you should adjust for that.

[SABR Matt]

Also, some parks are going to generate higher BABIP than others and some are going to skew those results due to the handedness of the pitcher so to be accurate you should adjust for that.

Another good point.

Safeco Field for example has a rather large downward pull on BABIP especially on flyballs.

Although, if I'm using team-relative stats, isn't that sort of already accounted for (other than handedness of course)? If the park has a net downward pull on BABIP, all of the pitchers will have that benefit and therefore the team performance on balls in play goes away in the adjustment process. The important thing about adjusting for BABIP park effects will come when you start analyzing fielders...THEN you need to adjust for the park.

Now as for handedness, that's gointg to be a little tough to pinpoint. Park Effects are, in my experience notoriously inaccurate and inconsistent from year to year when done the traditional way. I would rather not introduce more of those traditional park effect calculations than I need to. What I would LIKE to do at some point is extend the FSIA to account for right and left handed splits at least in the PBP era, but without left and right handedness, the matrix is already huge and difficult to process (you need a lot of computing power to make it work) and the code that makes it work was written by someone with way more programming ability than I have who now has no time for sabermetric research...so that's unlikely to move forward at the moment.

BTW re: subtracting Glavine...I guess I understand your point. The original logic was that the pitcher himself has a pull on the BABIP, albeit a small one, but there's uncertainty there.

[baseballPAP]

Not a lot to add here Matt....just amazed that I'm actually following the whole thing

Your work shows me one thing without any doubt though....you are striving for complete understanding, and many others around here would be wise to follow your lead.

[SABR Matt]

Nice to see you dropping by here PAP.

striving is the key word there...as this thread illustrates I've put a lot of though into the proper methodology and I've still got so much work to do. :\

[SABR Matt]

BTW, I know LWTS aren't perfect for this task...but until I can find some way toi implement a massive markov calculator program that accepts queries from MySQL and spits out answer matrices for every pitcher/season in the PBP database, I can't use the markov approach, and I don't think BaseRuns is going to be more accurate. I am close to completing calculation of DNRA for every pitcher/season in the PBP database using the method outlined above except with the more accurate PBP-based data and encorporating all of the proposed changes listed here (with the exception of markov analysis yet).

The improved method will include XBH analysis, DP rate analysis, baserunning impacts (pickoffs, CS, SB, PB (which is often the pitcher's fault) and baserunner outs.

[SABR Matt]

Little update relevant to this example of Glavine's 2001.

According to my calculations Tom Glavine allowed 2.3 singles more than you'd expect given his team's 1B/BIP (non-pitcher-assisted), 4.5 2Bs fewer than expected and 2.5 fewer triples. The Braves had an insanely low XBH rate that year...really helped Glavine out but he does rate as having a few fewer XBHs.

haven't done the DP an baserunning analyses yet...one query at a time.