Steamer Projections evaluates the forecasting systems. The bottom line, as far as hitters go, seems to be CHONE & ZiPS for batter-quality projections and Fantistics for playing-time projections.
2009 Forecast Evaluations
Dash Davidson, Peter Rosenbloom and Jared Cross
In creating Steamer Projections we analyzed how best to use historical statistics to predict future ones. We broke batting and pitching performance into a series of components and used regression analysis to find the most effective way to combine previous years’ performances and the league average (regression to the mean) to predict future performance.
Now that the season is over and we have concrete data on how our system has performed, we need to find out in what areas our projections were accurate, and in what areas we need to improve. We intend to add park factors and age effects to our system for the 2010 season but are there other improvements we need to make?
The best way to figure this out is to compare and contrast the results of our system to those of other, “rival”, systems. One method of comparison was already done for us. For our analysis, we examined key rate statistics for both hitters and pitchers: OPS, ERA; and also key counting statistics for both of them: Wins, Ks, IPs, Hrs, RBIs, PAs. We also decided to analyze our data from a purely fantasy baseball oriented standpoint, choosing the most prudent fantasy-oriented stat, SGPs (Standings Gain Points) and seeing which of the eight systems offered the best projections for fantasy domination, and how and why.
The forecasting systems:
Marcel – The monkey. Uses three years worth of data, weighing recent years more heavily, adjusting for age and regressing to the mean. Marcel uses the same weights and age adjustments for each component. Marcel projected rate stats and used community forecasts for playing time.
Steamer – Our system. Steamer forecasts used the last 3 years worth of data for hitters and pitchers. Our 2009 projections did not utilize park factors or minor league statistics. Steamer is similar to marcel except a) although we always weigh recent seasons more heavily we have different weights for each component and regress some components more heavily than others, and, b) we did not take aging into account. Also, Steamer only projected rate stats (this includes things like “stolen base attempts per times on base” and RBI/AB), we used Pecota’s depth chart projected playing time.
PECOTA – Pecota not only creates a projection for a player X, it retrospectively creates projections for the most X-like players in history and adjusts X’s projection based on how all of the X-like players performed relative to their would be projections. Fancy. The projections we used were from PECOTA ‘s weighted mean spreadsheet not their fantasy baseball depth charts.
CHONE – CHONE uses 4 years of data for hitters and 3 years for pitchers. It utilizes batted ball data (the numbers of line drives, pop ups etc.), minor league statistics, batter’s weight and adjusts for league, park and age.
ZiPS – Like CHONE, ZiPS uses 3 years of data for pitchers and 4 years of data for most hitters (3 years for players under 24 and over 38). It also uses minor league statistics and park factors but has different aging curves for different player types. Uses GB/FB and handedness to project pitcher’s BABIP.
Sporting News – Sporting News publishes a widely used fantasy baseball guide each year. Although we don’t know this to be true, we suspect that their projections are created by an expert rather than a formula. This is the un-Marcel.
Fantistics– We analyzed Fantistics on the advice of Eric Mulkowsky who said that this system was particularly good in projecting playing time.
If we run a similar analysis in future years we would include OLIVER, CAIRO, Baseball Info Solutions (Bill James) and possibly Baseball HQ and other projection systems in the comparison.
Missing data – 475 hitters had 50 or more PA in 2009. 465 of these hitters had projections from each of the big 3 (chone, pecota and zips). 438 were projected by Marcel. We looked at these 438 hitters. Projection systems that projected fewer players (Steamer Projections and Sporting News were the main guilty parties) were given the Marcel projection for that player. This allowed for a comparison of all 438 hitters across systems. Sporting News and Steamer only projected about 270 players each. Systems could beat the monkey so long as the projetions they actually made were better than Marcel.
… [T]he difference between Chone and Zips might not be significant given our sample size. Chone and Zips are the top 2 systems for both subsets and beat Pecota by a healthy margin for both pools of players. I would feel reasonably confident in saying that Chone and ZiPS are ahead of the pack right now but not at all confident in saying that Chone is better than ZiPS.
Fantistics does really excel at projecting playing time. One advantage they present is that they update their projections throughout the offseason and these playing time projections are from immediately prior to the start of the season….
It’s hard to know exactly what to take from this. Chone and Zips seem to stand out in projecting hitter quality and they have somewhat similar methodologies which gives some hints about how to make good forecasts. Fantistics succeeds in projecting SGP’s best based largely on its success in projecting playing time which suggests, perhaps, that other systems haven’t put enough thought into how best to project playing time.
It’s worth noting, also, that for the fantasy player, not all playing time is created equal. If Jose Reyes and a #6 hitter are both projected for the same number of plate appearances and the same number of SGP’s, you’re probably better off taking Reyes. When he’s playing, he’s getting more plate appearances and when he’s not, he’s on the DL and you can play someone who, although they might project to zero SGP’s over replacement, is better than an empty slot.
Up next: Pitchers