If you flip a coin N times, you expect to get heads N/2 times. Obviously, random chance can come up with a number that's a little higher or a little lower than that. How high or how low a number of heads should make you suspicious about the coin, though?
The "standard deviation" is the answer. Most of the time, the number of heads you flip will be within one standard deviation of the number of heads you expect to flip; almost all the time, the number of heads you flip will be within three standard deviations of the number of heads you expect to flip. If you're outside three standard deviations, there's an excellent chance the coin is bogus.
For relatively simple problems like this that are basically just counting things, like number of heads flipped, the standard deviation is the square root of the number of things counted. So if you flip a coin N times, you expect to see N/2 heads, with a standard deviation of sqrt(N/2).
OK, let's make the problem more complicated. Suppose you're rolling a die. There are six equal possibilities, 1, 2, 3, 4, 5, 6. If you roll it N times, you should expect to roll N/6 1's, with a standard deviation of sqrt(N/6). If you roll it 600 times, you expect to get 100 1's and you won't be upset if the number of 1's you actually roll is somewhere between 70 and 130 (three standard deviations, which is sqrt(100), i.e. 10).
Ah, but now let's pose a different problem: Suppose what you're interested in isn't what number you roll, but whether it's odd or even? Now the problem looks like the coin flip problem, right? You've got two equal possibilities, you expect to see N/2 odd numbers. But what happens to the standard deviation?
(I raise this whole line of inquiry due to a discussion between me and blurker at a game a couple of weeks ago, in which I was rolling about 25% on a 50% chance and we were arguing whether that was reasonable or not.)
Assume that the number of 1's rolled is A, the number of 2's rolled is B, and so on. I'm going to start by making an approximation that may torpedo the entire result, I don't know, but what I'm going to do is assume that B is independent of A. There's a weak dependence (A + B < N), which is, I'm going to argue, negligible for large N. (There's also a single strong constraint, viz., A+B+C+D+E+F=N, which means that there are no more than five independent variables in the problem, but I'm only going to be working with three at a time, so I think I'm OK not to worry about that. Plus I'm totally conflating standard deviation and expected error, which are marginally different things.)
OK, so the number of odds rolled is A+C+E. The expected value for each is N/6, with a standard deviation of sqrt(N/6). As I've argued, A, C, and E are independent. When you have independent variables, the way you combine them is that the standard deviations add in quadrature (square 'em, add 'em, take the square root).
Accordingly, the standard deviation (really expected error) of A+C+E is sqrt (N/6+N/6+n/6), or sqrt(N/2), which is exactly the same standard deviation we obtained for the coin-flipping case. Therefore, it shouldn't matter whether you flip a coin to determine a 50% chance or roll a d20 and count 1-10 or 11-20; you should still get the same deviation around the expected value.
If someone in the audience knows more stats that I do, I'd be delighted to learn more.
Posted by Greg at June 9, 2005 9:44 AM
I think any results you get after that heinous assumption of independence of your variables are pretty worthless.
I'm not sure what you were aiming for here. If what you want to know is whether the die is fair, you should form a hypothesis ("P(rolling 1) >= 0.2", for instance) and then work out from your trials what are the odds that the hypothesis is true, given the number of each roll.
Or, if you were rolling too many odds, then just work with "P(rolling odd) >= 0.6" or something. Let's say the chance of rolling odd is "p", and you do N trials and get M odds. The likelihood of this outcome is then something like
f(p,N,M) = p^M (1-p)^(N-M) N!/((M!)(N-M)!)
Now, unless I am remembering wrong, we have to make some more asssumptions before we can compute anything. I like Bayes' Rule, so let us go there. Let's assume that p is some constant value that's taken from an even distribution between 0 and 1 (in reality you probably have more respect from your die manufacturer than this, and would want to assume a gaussian distribution of p that is fairly tight). Let "A" be that p >= 0.6 and "B" be that we get M odds on N rolls. Then Pr(A) = 0.4. Pr(B) = the chance of M odds on N rolls integrated over all values of p weighted by their probability, which I expect is something like f(0.5,N,M), although it might not be because the lower-than-0.5 p values and the higher-than-0.5 p values may not exactly cancel out when M != N / 2, so I will just write it as the integral from 0 to 1 over p of f(p).
Bayes' rule says that
Pr(A|B) = Pr(B|A) Pr(A) / Pr(B)
Now I need to do some more hand-waving, let A for the moment be that p = 0.6 (or anyway that p = p') and let us integrate p' from 0.6 to 1.
Pr(B|A) = f(p,N,M)
Pr(A) = a constant infinitesimal independent of p'
Pr(B) = Int_0^1 dp' f(p',N,M)
And so
Pr(p>=0.6) = 0.4 Int_0.6^1 dp' f(p',N,M) / Int_0^1 dp' f(p',N,M)
I'm not sure what you were aiming for here.
If I flip a coin, I should get heads half the time. However, if I perform the test N times, I expect to get M heads. The probability of getting M heads is obvious; graphing probability against M should give a gaussian distribution centered on N/2. What I'm interested in is the standard deviation of that gaussian distribution. My instinct is that it should be a simple function of N.
Next, expand the problem to a die. The probability of getting M 1's in N trials forms a gaussian distribution centered on N/6. Again, my interest is the standard deviation of that distribution and my instinct is that it should also be a simple function of N.
Now, combine the problem so that you're not counting 1's, you're counting odd numbers. Like the coin, the probability should form a gaussian distribution centered at N/2. The ultimate question I want answered is: Is the standard deviation of that gaussian distribution different from the standard deviation in the coin problem?
If it is, then it matters whether you flip a coin or roll a d20 to determine a 50% outcome. If it isn't, then it doesn't.
See the next post; I've harnessed the power of analogy to argue that the answer is that there's no difference.