What We Don’t Know

I meant to link to this a while ago – via David Pinto, you can now access Bill James’ pot-stirring SABR article criticizing the common sabermetric methodology – used many times by James himself – of comparing season-to-season results to determine whether things like clutch hitting and platoon differentials are persistent skills rather than transient statistical flukes caused by luck. James’ main point: several of the items he lists are just unknowable by this method because the year-to-year sample sizes are so small they can’t be useful for any study.
Of course, in some cases, like the Voros McCracken hypothesis about balls in play, there are alternative ways to attack the problem by looking at the extent to which performance over time tends to level off. It’s not that the questions James revisits can’t be answered, but he’s probably right (as usual) that the answers we have so far are unsatisfactory to the extent that they are based on this methodology.
(Of course, for selfish reasons I’m glad to see a movement away from a type of study that was always beyond my computational skills; I don’t do regression analyses).