Jump to content

Providing independent coverage of the Minnesota Twins.

The same great Twins Daily coverage, now for the Vikings.

The Store


Photo

Article: Catch Framing Predictability

  • Please log in to reply
23 replies to this topic

#1 John Bonnes

John Bonnes

    Owner

  • Administrators
  • 4,964 posts

Posted 22 February 2014 - 06:25 PM

You can view the page at http://www.twinsdail...-Predictability

#2 Seth Stohs

Seth Stohs

    Owner

  • Administrators
  • 7,560 posts

Posted 22 February 2014 - 06:34 PM

[COLOR=#3E3E3E]Very interesting, and yet not too surprising. If you're getting more strikes called, a pitcher will throw less pitches...[/COLOR]

[COLOR=#3E3E3E]My concerns with pitch framing...[/COLOR]

[COLOR=#3E3E3E]1.) I think it would be interesting to see how the numbers would look if you removed 3-0 and 0-2 counts. 3-0, if it's pretty close, it's a strike. If it's 0-2, that same pitch in the same location is often called a ball.[/COLOR]
[COLOR=#3E3E3E]2.) As much as we want to think that all players have the same strike zone, does it not make sense that someone catching Sam Deduno, Kyle Gibson, Ryan Pressly, other rookies and unproven pitchers will have different strike zones than proven pitchers? It's human nature. I bet Greg Maddux's catchers in his prime had pretty good pitch framing numbers if they had had such things then.[/COLOR]
[COLOR=#3E3E3E]3.) Game situations shouldn't, but likely do, play into it.[/COLOR]

#3 Thrylos

Thrylos

    Yes

  • Members
  • 4,330 posts
  • LocationLehigh Valley, PA, USA
  • Twitter: thrylos98

Posted 22 February 2014 - 06:38 PM

I have lots of concerns about pitch framing as a metric as well:

- it has to be normalized per umpire (it is not.) And we know that a 5'7" 300lbs umpire's strike zone is different than a 6'7" 200 lbs umpire's.

- it has to be normalized per pitcher (it is not). I bet if Greg Maddux was throwing exclusively to Ryan Doumit, Doumit would have been the best pitch framing catcher in the game...

Until they find a way to normalize their data in at least those 2 dimensions, I am not buying it...
-----
Blogging Twins since 2007 at The Tenth Inning Stretch
http://tenthinningst...h.blogspot.com/
twitter: @thrylos98

#4 DocBauer

DocBauer

    Senior Member

  • Members
  • 654 posts

Posted 22 February 2014 - 07:12 PM

I can't add much to what Seth and Thrylos already stated.

This stat is attempting to quantify human nature, discrepancy and error and provide a measurable and quantifiable statistic.

You have two leagues, 4 man umpiring crews who rotate who is behind the plate, and they almost ALL call a game differently. Factor in different pitchers, not only power vs control, but vet vs rookie, LH vs RH, and I just think you have way too many mitigating factors to come up with a manageable statistic.

#5 h2oface

h2oface

    Senior Member

  • Members
  • 396 posts

Posted 22 February 2014 - 08:21 PM

I think it is sad that when we have the technology for instant correct calls, instead, we have to embrace a talent the cheats the rule of the game, and continue to have the game impacted by getting calls that are not correct.

#6 jorgenswest

jorgenswest

    Senior Member

  • Members
  • 1,613 posts

Posted 22 February 2014 - 08:27 PM

Good work.

The correlation suggests it is a skill. Once teams are willing to acknowledge it is a skill and trust the metric, the skill can be improved. Analysis can point out zones and pitch types of strength and weakness in a catcher. Video study can show technique that can be improved with those zones and pitches. Work can be done throughout the minors.

There is no reason an athletic catcher can't improve in this skill. Particularly those young and in the minors. The first step is for the Twins to acknowledge and trust the metric. I don't think they are there yet.

#7 John Bonnes

John Bonnes

    Owner

  • Administrators
  • 4,964 posts

Posted 22 February 2014 - 10:53 PM

Re: the concerns

With umpires - you would think if the umpires were that important in this study, the results of the correlation would be more random, right? It's not like a catcher's umpires that he happens to catch in front of are consistent from year to year.

Re: pitching staffs - I just reran the same study, only this time, I only chose catchers who also changed teams from one year to the next. That would seem to negate the impact of pitchers. It came back with only 46 examples (as opposed to 189 from the previous study) so there is a higher deviation, but the results were still ridiculously strong: .68 for the season and .75 for pitches per game.

This is a skill, folks.

#8 nicksaviking

nicksaviking

    Senior Member

  • Members
  • 3,734 posts

Posted 22 February 2014 - 11:11 PM

Good write up John, but I'm still skeptical. As has been mentioned, I think it has more to do with the pitchers. Umpires already have preconceived ideas and biases about the pitchers. If the umpire already has it in his mind that the pitcher is wild or young, I doubt catcher manipulation will work well. On the flip side, well I'm guessing Greg Maddox's personal catchers through the years would be ranked among the elite at framing if the data was available.

#9 biggentleben

biggentleben

    Senior Member

  • Members
  • 1,086 posts

Posted 23 February 2014 - 12:19 AM

I'll echo something that has been posted already in a slightly different way. When I was pitching growing up, I threw against a garage that had a board about the size of an old catcher's pud. Every time I was working on a new grip on a pitch, if I couldn't consistently hit that board with it, I wasn't going to use it in a game. Once I started pitching in games, coaches knew I had a strong arm from watching me at third base, but they were astounded at my command and ability to hit the catching mitt every time. I wasn't anywhere near the skill of a major league pitcher, but we've all seen plenty who don't hit their catchers in the mitt at all. I'd say the ratio is very, very high that if your catcher's glove doesn't move, you get a strike called for you, even if it's a bit outside or inside. That's one thing Eddie Perez did masterfully for the Braves for years. He would set up just a tick off the black outside or inside, and Maddux would nail his glove without moving it. Even if you're an inch inside with the pitch, if you hit that spot perfectly, you'll get the call.

I'd be curious to go through the data and find the difference in fastball vs. changeup vs. breaking pitches and framing of them. From the eye test, one thing that always astounded me with Suzuki was how many breaking pitches he caught from guys in Oakland and how well he made sure they never got by him. A young guy throwing breaking balls will make his catcher's framing look poor, but Suzuki spent years catching entire staffs of young guys living on breaking stuff in Oakland, so it's understandable his numbers were off some. One interesting correlation that Baseball Prospectus mentioned in a discussion on the Suzuki signing is that catchers who have high framing ratings tend to also have more passed balls, and the opposite has also been true, to the coefficient of roughly 0.5, so not amazing, but certainly notable, so perhaps there's something in that holding your position to the very end gets you more calls, but you end up having more balls get by you or the opposite, you never let a ball get by you, but you take your body out of position to frame the pitch best for an umpire by doing that.

Just some thoughts...I like the idea of rating framing, but I feel like it's where things like UZR and Range Factor were 10-15 years ago. As more and more factors are figured out to get the best measurement, the numbers will balance with the eye test, and right now, there are some catchers who get rated low on catch framing that seem to do a very good job protecting their pitchers, so perhaps it's not always the best indication of success behind the plate.

#10 Wookiee of the Year

Wookiee of the Year

    Senior Member

  • Members
  • 135 posts

Posted 23 February 2014 - 07:15 AM

Re: the concerns

With umpires - you would think if the umpires were that important in this study, the results of the correlation would be more random, right? It's not like a catcher's umpires that he happens to catch in front of are consistent from year to year.

Re: pitching staffs - I just reran the same study, only this time, I only chose catchers who also changed teams from one year to the next. That would seem to negate the impact of pitchers. It came back with only 46 examples (as opposed to 189 from the previous study) so there is a higher deviation, but the results were still ridiculously strong: .68 for the season and .75 for pitches per game.

This is a skill, folks.

Yeah, that's the rub--it's easy to come up with reasons why catch framing might not be all it's cracked up to be, but when you're staring at a year-to-year correlation of over .65, you're arguing against reality. The data makes a strong case this is a skill.

The hard part is translating the impact of that skill into runs and wins.

I also wonder: We've heard so much about how pitchers love working with Suzuki. I wonder how big a factor pitch selection is vs. pitch framing? We really can't evaluate the latter, but just because you can't (yet?) quantify it doesn't mean it can't have a very large impact.

#11 cmathewson

cmathewson

    Senior Member

  • Members
  • 2,273 posts

Posted 23 February 2014 - 07:54 AM

Yes, it is a skill. But I don't think the metric as currently used is a good one for development. Why? Because there are too many uncontrolled variables. If there was a way of breaking down the metric to that which a catcher can control, it would be worth tracking those factors over time.

Perhaps an anology is needed. Say a player wants to improve his batting average, but fails to do so mostly because of luck. Perhaps his LD percentage goes up but his BABIP goes way down. That is, he hits a lot of liners right at people. Because he used average as his KPI, he is a failure. If he had used LD percentage instead, he might have a better gauge of his improvement.

I just don't know how to break down the metric into similar smaller bits. And therein lies the problem. It is OK as a gross measure of one aspect of catching, given sufficient sample sizes. But it is a poor measure of ability or improvement. Also, as big an impact as it might have, pitch selection is bigger. It just goes to show how underrated the catching position is. We can only focus on what we can measure, which isn't the half of it and which is subject to the perils of small samples.

#12 jorgenswest

jorgenswest

    Senior Member

  • Members
  • 1,613 posts

Posted 23 February 2014 - 09:39 AM

Thought on umpires...

If you don't believe in the data because of umpires, than you certainly can't believe in pitcher strike outs and walks where fewer umpires would have a greater influence on the smaller sample.

Thought on developing catchers...

Ben Lindbergh did several interviews last year with catchers and coaches from teams that seemed to be using the data. Jose Molina credits his development and growth to Tony Pena and Joe Girardi. Others speak about how much they work on the skill. The data set is large enough so that catchers can look at it by zone or by pitch. They can be aware of where they are losing strikes or which pitches they lose strikes on. They can use video study to see how their technique differs from other catchers who are getting those strikes. This skill can absolutely be developed.

If the Twins don't believe in the data, they are wise not to invest the time. They are wise not to pay for the skill when acquiring catchers. Let other teams waste the time and pay for the skill they don't trust. I hope the Twins are on the right side of this argument.

#13 Nate Haseman

Nate Haseman

    Junior Member

  • Members
  • 13 posts

Posted 23 February 2014 - 09:42 AM

Nice work! I completely agree that it is a skill but I'm with a lot of the other readers in that I think it's a tough metric to calculate. If Mauer was catching for the Dodgers or Cardinals last year I bet he'd wind up being well above average. Keep up the great work in our quest for the truth about pitch framing!

#14 Seth Stohs

Seth Stohs

    Owner

  • Administrators
  • 7,560 posts

Posted 23 February 2014 - 12:24 PM

No question it's a skill. I don't think anyone will question that. It was something I practiced when I did do some catching. And there's no question some are better at it than others, like any skill.

#15 Oxtung

Oxtung

    I don't skinny dip. I chunky dunk.

  • Members
  • 1,522 posts

Posted 23 February 2014 - 12:46 PM

I don't think we need to worry about how any given umpire affects a catchers pitch calling stats. While every umpire is going to call the game slightly differently those differences won't matter just because of the sheer number of games the pitch framing metric covers. In the end they'll have umps that call it close and umps that call it loose and they should balance out.

Here's another way of looking at it. Since 08, which is what John's data covers, Joe Mauer has caught 561 games. How many games would an umpire have to call before he has any discernible impact on Joe's pitch framing skill? An umpire calls ~30 games a year behind the plate.

Does anyone know how a crew is assigned to a series? Random or by region or ...?

#16 thetank

thetank

    Senior Member

  • Members
  • 204 posts

Posted 23 February 2014 - 12:56 PM

I think it is sad that when we have the technology for instant correct calls, instead, we have to embrace a talent the cheats the rule of the game, and continue to have the game impacted by getting calls that are not correct.

It's suppose to rain on both sides. I assume they would use sensors on the both sides of the plate, but let the umpire determine the top to bottom strike zone?

#17 cmathewson

cmathewson

    Senior Member

  • Members
  • 2,273 posts

Posted 23 February 2014 - 01:24 PM

It's suppose to rain on both sides. I assume they would use sensors on the both sides of the plate, but let the umpire determine the top to bottom strike zone?


Pitch F/X is accurate--more accurate than umpires. What we have seen, especially with the taller Twins catchers, is umpires reluctant to call the low strike. The other side of the coin is that both Mauer and Doumit got more high strikes called. Considering that the staff has consisted of mostly sinkerball guys, that is not a good thing. Hopefully both Suzuki and Pinto can bring the strike zone down a bit. I'm more interested in that trend than the overall percentage of balls brought in or strikes lost.
"If you'da been thinkin' you wouldn't 'a thought that.."

#18 Jim H

Jim H

    Senior Member

  • Members
  • 439 posts

Posted 23 February 2014 - 05:41 PM

Pitch F/X is accurate--more accurate than umpires. What we have seen, especially with the taller Twins catchers, is umpires reluctant to call the low strike. The other side of the coin is that both Mauer and Doumit got more high strikes called. Considering that the staff has consisted of mostly sinkerball guys, that is not a good thing. Hopefully both Suzuki and Pinto can bring the strike zone down a bit. I'm more interested in that trend than the overall percentage of balls brought in or strikes lost.


I don't know that the first statement is accurate. There are 3 things about pitchF/X that make that statement doubtful. First, PitchF/X is 2 dimensional while the strike zone is 3 dimentional. The 16 inches of depth of the plate allows moving pitches to drop in or move in from the side. Since PitchF/X is set up at the front of the plate many pitches "called" balls by PitchF/X are often strikes according to the rule book.

Second, Pitch F/X has a "standard" one size fits all strike zone. Major league ball players range in size from about 5' 5" to about 6' 9". According to the rule book there is no standard strike zone. Pitch F/X has to miss pitches both high and low depending on the height of the player.

Finally, I think that all of us realize that as wonderful as modern technology is, it is far from perfect. Something as complex as Pitch F/X is going to need some very expert calibrating to actually have that window set up perfectly over the plate and being actually the exact same size in every ballpark.

While I have no illusions about the perfection of the average big league umpire, there is no way given the limitations of Pitch F/X, that there will not be many, many cases where the umpire is right and Pitch F/X is wrong.

#19 Oxtung

Oxtung

    I don't skinny dip. I chunky dunk.

  • Members
  • 1,522 posts

Posted 24 February 2014 - 09:51 PM

I don't know that the first statement is accurate. There are 3 things about pitchF/X that make that statement doubtful. First, PitchF/X is 2 dimensional while the strike zone is 3 dimentional. The 16 inches of depth of the plate allows moving pitches to drop in or move in from the side. Since PitchF/X is set up at the front of the plate many pitches "called" balls by PitchF/X are often strikes according to the rule book.

Second, Pitch F/X has a "standard" one size fits all strike zone. Major league ball players range in size from about 5' 5" to about 6' 9". According to the rule book there is no standard strike zone. Pitch F/X has to miss pitches both high and low depending on the height of the player.

Finally, I think that all of us realize that as wonderful as modern technology is, it is far from perfect. Something as complex as Pitch F/X is going to need some very expert calibrating to actually have that window set up perfectly over the plate and being actually the exact same size in every ballpark.

While I have no illusions about the perfection of the average big league umpire, there is no way given the limitations of Pitch F/X, that there will not be many, many cases where the umpire is right and Pitch F/X is wrong.


No way to know but I would place money on the machine being correct more often than the human. The eye-brain unit is absolutely terrible at precision and even worse when it comes to high speed precision.

#20 old nurse

old nurse

    Member

  • Members
  • 1,750 posts

Posted 25 February 2014 - 01:45 AM

No way to know but I would place money on the machine being correct more often than the human. The eye-brain unit is absolutely terrible at precision and even worse when it comes to high speed precision.

Are you outright dismissing the limitations of pitch f/x ? Correct more often is hardly a solution

#21 Oxtung

Oxtung

    I don't skinny dip. I chunky dunk.

  • Members
  • 1,522 posts

Posted 25 February 2014 - 05:00 PM

Are you outright dismissing the limitations of pitch f/x ? Correct more often is hardly a solution


Where did I say I was dismissing the limitations? It's not a perfect solution. I was saying that I believe, imperfections and all, it is still better than the human eye/brain combo.

If "correct more often is hardly a solution" what does that say about our system of umpires today?

#22 Oxtung

Oxtung

    I don't skinny dip. I chunky dunk.

  • Members
  • 1,522 posts

Posted 25 February 2014 - 05:10 PM

I will say this, there seems to be a lot of assumptions being made about what a pitch f/x umpiring system would look like. Keep in mind that the box often associated with Pitch f/x isn't actually a part of the system but rather an add on that has been superficially imposed.

Any system that was actually used in games clearly wouldn't use an "average batter size" for determining the strike zone but rather the actual definitions. That software exists today. You see it all the time on facebook or your camera when it puts a box around a persons face. A computer certainly could determine where the knees, belt and shoulders of the batter are and thereby compute the vertical limitations. The horizontal is even easier.

Would some money and time have to be spent creating and perfecting it? Yes. Would there have to be testing? Yes. Would the system have to be calibrated? Yes. However, all the components of the system are already in society today.

#23 Jim H

Jim H

    Senior Member

  • Members
  • 439 posts

Posted 25 February 2014 - 06:15 PM

The problem I have with Pitch F/X has less to do with whether it is better than the umpires in determining balls and strikes, but the often unspoken assumption that it is perfect. Clearly people are using it to informally evaluate umpires, assuming I guess, that when they disagree on a call that Pitch F/X is right and the umpire is wrong. Much the same thing is happening with the pitch framing thing. If Pitch F/X says ball, the umpire says strike, it must be that the catcher "stole" a strike thru pitch framing.

Since I believe Pitch F/X has limitations, and as a previous poster suggested, it is currently impossible to realisticly tell how many pitches Pitch F/X gets wrong, it certainly could be misleading to use it to evaluate umpires. Or for that matter, catchers either, though that is what the pitch framing metric is trying to do.

#24 Oxtung

Oxtung

    I don't skinny dip. I chunky dunk.

  • Members
  • 1,522 posts

Posted 26 February 2014 - 06:16 AM

The problem I have with Pitch F/X has less to do with whether it is better than the umpires in determining balls and strikes, but the often unspoken assumption that it is perfect. Clearly people are using it to informally evaluate umpires, assuming I guess, that when they disagree on a call that Pitch F/X is right and the umpire is wrong. Much the same thing is happening with the pitch framing thing. If Pitch F/X says ball, the umpire says strike, it must be that the catcher "stole" a strike thru pitch framing.

Since I believe Pitch F/X has limitations, and as a previous poster suggested, it is currently impossible to realisticly tell how many pitches Pitch F/X gets wrong, it certainly could be misleading to use it to evaluate umpires. Or for that matter, catchers either, though that is what the pitch framing metric is trying to do.


Pitch f/x has gone through testing to ensure that it is indeed accurate. It's accuracy has been mathematically verified to be within 1 in. of the actual location of the pitch as of '08. I have also seen a video where they tested the data physically. They placed a giant sheet of paper across the plate. Then they compared the holes created by the ball to the data their system was creating.

The strike zone graphic is created in different manners depending on the application of the data. In a broadcast that zone is created by a Sportsvision employee physically marking the top and bottom of the strike zone. This display, I would guess, is not particularly controversial. If you trust a human to umpire balls and strikes then it stands to reason you would trust a human to create a strike zone.

The second way this graphic is created is by using the PITCH f/x data to determine the average players strike zone and then uses that to create a graphical overlay. That overlay is then used to determine if a pitch was a ball or a strike. This is the approach taken by FanGraphs pitch framing team. This approach would lead to a system where any individual strike/ball decision is indeterminable but because the number of pitches analyzed is in the thousands the system as a whole would be theoretically accurate.

I think the biggest piece of supporting evidence to the accuracy of PITCH f/x data is the fact that MLB uses it to evaluate their umpires.

I would just like to say that I do not blame the umpires for getting a ball/strike call incorrect. The human brain and eyes just aren't made to track objects moving at those velocities accurately. Not to mention the psychological effects of thousands of fans on Umpire decision making. It's not the umpires fault. It's just the way humans have developed.

Here is an awesome website I found that curates many relevant pieces of literature on the PITCH f/x system.