The two Orioles-Red Sox games recently (particularly the performance of relievers in them) have gotten me thinking about just how hard it is to evaluate the impact of a reliever to his team.

For example, which reliever had a better game?

Code:
IP   H  R ER BB SO 
1.1  1  1  1  0  1 
1.1  1  1  1  2  0
If you had to guess, you'd probably say the first one. The first one is Hideki Okajima yesterday, the second one is Jim Johnson Tuesday. If you didn't watch either game, here's a quick summary of what happened. Johnson came in with the bases loaded, nobody out, and a great hitter, Manny Ramirez, at the plate. Okajima came in also with the bases loaded, but with two outs, and a poor hitter, Jay Payton, at the plate. Johnson got Ramirez to ground right back to him for the double play, then got Mike Lowell to fly out, escaping the jam with no runs scoring and the O's 5-3 lead intact. Okajima allowed a grand slam to Payton, giving the O's a 6-3 lead. Johnson then allowed a couple walks in the 8th, and George Sherrill allowed the run to score. Okajima shut the O's out in his second inning, making their ERAs equal. But anyone who watched could tell you that Johnson was far more valuable to the O's than Okajima was to the Sox. And ERA doesn't reflect that at all.

One way to correct that is by using inherited runners scored. But that's not really exact, since it's hard to blame pitchers for coming into a bad situation. One I like to use that measures a similar skill is Win Probability Added over Leverage Index. It takes into account both runners inherited as well as new runners put on by the reliever himself.

I'm running out of time, so that's all for now.