Reviewing PECOTA's Pre-Season Twins Projections

[Matt Braun]

Just how accurate was our robot overlord?

Image courtesy of © Bruce Kluckhohn-USA TODAY Sports

If you can recall the two posts I made almost exactly eight months ago, PECOTA—the flagship projection system from Baseball Prospectus—had some thoughts regarding the Twins. Well, it had thoughts regarding every player, but we only looked at those set to don Minnesota jerseys. Enough beating around the bush: here’s how well the computer did.

(After-season numbers are taken from Baseball Prospectus’ leaderboard found here for pitching, and here for hitting.)

Perhaps most notable at the time was PECOTA’s optimism surrounding Pablo López, who joined the Twins as something of an unknown, possessing immense strikeout potential without the full season of unquestioned dominance. Turns out, the system was actually a pessimist: López crushed it in 2023, turning in 4.8 WARP, good for 3rd in MLB. PECOTA was also too low on Sonny Gray, Joe Ryan, and Bailey Ober; all three starters bested their projections, with Gray doubling his assumed WARP.

Louie Varland can claim underrated status as well; he wasn’t even in the original post and ended up as the eighth-most-valuable pitcher on the team at the end of the year.                                                                                                                                                   

Also, the computer was absolutely correct in regards to Emilio Pagán, whose ERA (2.99) and FIP (3.26) were freakily close to his projections. Perhaps this is a lesson in patience, or—rather—that giving up a lot of homers isn’t necessarily innate in a pitcher’s DNA; this is a weird and frankly unfair game we’re fans of, and Pagán proved that the difference between a hero and a villain is often just a few feet. 

Finally, the Jovani Moran train may have hit a cartoonish boulder, crashed, and exploded in a fiery rage, but he actually came within tickling distance of his projection thanks to a whiff rate amongst the best in MLB. He appears a good bet to rebound next season if healthy. 

Now, let’s move onto the batters:

It, uh, didn’t do great here! Let’s start with the positives: PECOTA nailed Max Kepler’s bounceback season, actually underselling him by a few points of DRC+, but otherwise prophesizing his best season since COVID hit. It also warned people not to be too down on Royce Lewis; we all know how that went.

But… yeah, this one is a mess. Minnesota’s 2nd and 3rd most valuable position players ended up being Willi Castro and Matt Wallner, not Jorge Polanco and Carlos Correa. Byron Buxton ended up behind Christian Vázquez. Jose Miranda is lost somewhere in the Joey Gallo void. Gallo himself… it’s best to keep his name locked up in a box, lest uttering it releases curses unto humanity. I'm a little humored that Trevor Larnach couldn't escape his fate, essentially nailing his pedestrian prediction.

 It's clear this was a season dominated by the unpredictable; be it the rookie onslaught or Castro's elevation, the exact shape of Minnesota's offensive production was atypical, but eventually effective. 

----------------------------------

Overall, I’m impressed with how accurate PECOTA was in regards to the pitching staff. Some hurlers blew past their projections, but the order was mostly in line with how the season played out. Calling on Pagán to exceed wasn’t something perhaps any Twins fan could do. Hitting was a big miss—anyone who predicted Willi Castro being Correa’s equal in DRC+ would have been hanged as a witch. Projections are helpful, but there’s a reason they play the games, and strange and unusual things happen when competitors at the highest level face off against each other.

View full article

[Rod Carews Birthday]

While PECOTA (and other projection systems) takes a great deal of data into account and works very hard to come up the best projections they can, there are innumerable human factors at play that throw everything into flux -- injuries, unexplainable ineffectiveness, crazy unexplainable breakouts, etc.  They are just another tool to try to predict what will happen.  Sometimes, they hit on a few things, sometimes they miss by a mile.  It's always interesting.  I commend them for their efforts.  I wouldn't want to try to do it better. 

However, the fact that the predictions aren't more accurate is why we play the games, and frankly, what makes it more fun!

[jmlease1]

I'll admit to not understanding how Jeffers still finished below Vazquez in BP's WARP.

 

[mnfireman]

1 hour ago, jmlease1 said:

I'll admit to not understanding how Jeffers still finished below Vazquez in BP's WARP.

 

He didn't, the chart is pre-season projections. Jeffers. actual bWAR and fWAR were considerably higher than Vazquez.

[Krasnoludki]

Yeah - it does look like Vasquez finished higher than Jeffers. How is it possible? 

[Shobae]

29 minutes ago, Krasnoludki said:

Yeah - it does look like Vasquez finished higher than Jeffers. How is it possible? 

I'm by no means an expert on WARP (or any BP stats for that matter) but I do have a surface level understanding of how it works. The first thing to note is that 0.3 WARP doesn't really mean anything statistically, it's really within the uncertainty of any war type stat.

But actually now that I go and look at the breakdown of the WARP components I am confused as well. Because Jeffers has -2 batting runs vs -12.2 for Vazquez. But Vazquez has 7.5 defensive runs prevented vs -1.1 for Jeffers. But if you do the nampkin math with the other components Jeffers still comes out on top for runs? So they could be around the same at best for Vazquez? like I said don't really understand WARP well enough to have the answer here. Though if I had to posit a guess here maybe it's something to do with their model or some other adjustments they do? Well I tried, but seems I can't explain it either, even looking at their defensive and offensive components. 😹

[terrydactyls]

I love it when someone starts explaining analytic statistics by saying "the first thing to note is that [enter any analytic statistic here] doesn't mean anything statistically".  That is certainly my feeling.

[jmlease1]

16 hours ago, mnfireman said:

He didn't, the chart is pre-season projections. Jeffers. actual bWAR and fWAR were considerably higher than Vazquez.

As noted above, for the 2023 hitters leaderboard at BP, they have Vazquez 8th for the Twins with a 0.8 WARP and Jeffers 12th with a 0.5. Which is relevant in talking about where PECOTA landed for their predictions on Twins players; if their final WARP calculations are so out of whack from where fWAR and bWAR landed those two players, it makes me question their metrics period.

Maybe the punishment for Jeffers taking some ABs at DH is more significant than I thought? But Vazquez had a staggeringly bad year at the plate and Jeffers was excellent. I haven't seen any defensive metrics that put Jeffers so much further below Vazquez as to make up that difference, let alone exceed it.

[tony&rodney]

Analytics have provided a ton of jobs for people and also drawn interest from multiple directions. The economy is good.

[RpR]

55 minutes ago, tony&rodney said:

Analytics have provided a ton of jobs for people and also drawn interest from multiple directions. The economy is good.

I wonder if there is any where one can bet money on PECOTA?