How To Test Baseball Game Prediction System?

[davidr]

I am working on a MLB game prediction system. The system gives a percentage chance that Team A will beat Team B (with pitchers, ballpark, and other factors hopefully taken into account).

My question is how do I determine if my prediction system is accurate? If the Indians are predicted to beat the Tigers 65% of the time when they play on a certain date the actual result will either be an Indians victory or Tiger victory. I would like to know if my 65% was a good estimate.

Let's assume I'm doing this over an entire season where I have percentage chance of victory for each game during that season along with the actual results of those games. How can I test this?

Thanks for you help and if my explanation is unclear ask away and I'll be happy to clarify.

Ultimately, I want to use this system to bet games.

-Dave

[MyTurn2Raise]

go back to a time period that you didn't use to dervie the system and backtest the system over that data

goldsheet.com has some data as well as a few other places

[davidr]

Is there anyway for me to prove that my system predicts games accurately without going sgainst bookmaker results or is that the only way to prove that my system has predictive power?

[ImStillBen]

If you're struggling with this, terminate this exercise before you hurt yourself.

[davidr]

I appreciate your words of caution along with the sarcasm.

I now know how to do this using the moneyline from baseball games thanks to a poster at sharpsportsbetting.com. My question still stands. I guess what I am asking is:

1. How would I verify that my system has predictive power if the moneylines weren't available?

2. How do I know if my results are statistically significant?

Thanks,
-Dave

[rjp]

You need to use some sort of loss function. For example, if you were to classify all games in which you predicted to win 50%-53% of the time, what was the actual frequency of wins for these games? The closer to your estimate the better, although you then need to check a binomial distribution to see how likely you were to see the results.

[dudeimstoked]

what you should do is track it yourself for at least 200 games next season and see how you do. i personally would like a larger sample size to see if you are +EV and go from there. fyi it will probably take a lot of fine tuning and adjustments to your model to "perfect" it.

or you could generate your own lines and compare it to the closing lines and see how close you are. for the majority of games you should have a <5% difference.