Decision Calculator

This is just a rough prototype meant to be easy to play with inside a post. READ the help and guidebook! Suggestions for new canned examples welcome—the hard part is deriving historical performance data.

Rules

  1. Read the Decision Calculator guidebook below!
  2. Fill in the Performance Table, or click on one of the predefined examples.
  3. Fill in the Cost Comparison Table, or click on one of the predefined examples. You do not need to calculate the total: that’s done automatically.
  4. Click Calculate (or Reset between examples).
  5. Accuracy comparison rates are given between the Expert System and the Naive Guess.
  6. Cost results are found in the Expected Cost Comparison Table.
  7. Finally, a solution saying which option you should choose is given. Skill should be > 0!
  8. Important: Use this software at your own risk. No warranties of any kind are given or implied. Always consult a competent medical professional. .

Preset examples:

See Below HELP HELP HELP

 

1. Fill in the Historical Performance Table.

Present Absent
Test +
Test –

 

2. Fill in the Cost Comparison Table.

False Positive Costs Score False Negative Costs Score
Total: Total:

 

3. Click calculate (or Reset between examples).


 

4. The Optimal Naive Guess is to:

 

5. Accuracy (%) Comparison Table

Test Accuracy
Expert System
Naive Guess

 

6. Expected Costs Comparison Table

Test Expected False Positive Cost Expected False Negative Cost Total
Expert System
Naive Guess

 

7. The skill score is:


It should be greater than 0 for a skillful test!

 

8. The solution:

 

GUIDEBOOK

This article provides you with an introduction and a step-by-step guide of how to make good decisions in particular situations. These techniques are invaluable whether you are an individual or a business.

These results hold for all manner of examples—from deciding whether to have a PSA test or mammography, to get a vaccine, to finding a good stock broker or movie reviewer, to situations that require intense statistical modeling, to financial forecasts, to lie detector usefulness. Any situation that has a dichotomous outcome can use these techniques.

Many people opt for precautionary medical tests—frequently because a television commercial or magazine article scares them into it. What people don’t realize is that these tests have hidden costs. These costs are there because tests are never 100% accurate. So how can you tell when you should take a test?

When is worth it?

Under what circumstances is it best for you to receive a medical test? When you “Just want to be safe”? When you feel, “Why not? What’s the harm?”

In fact, these are not good reasons to undergo a medical test. You should only take a test if you know that it’s going to give you useful information. You want to know the test performs well and that it makes few mistakes, mistakes which could end up costing you emotionally, financially, and even physically.

Let’s illustrate this by taking the example of a healthy woman deciding whether or not to have a mammogram to screen for breast cancer. She read that all women over 40 should have this test “Just to be sure.” She has heard lots of horror stories about breast cancer. Testing almost seems like a duty. She doesn’t have any symptoms of breast cancer and is in good health. What should she do?

What can happen when she takes this (or any) medical test? One of four things:

  1. The test could correctly indicate that no cancer is present. This is good. The patient is assured.
  2. The test could correctly indicate that a true cancer is present. This is good in the sense that treatment options can be investigated immediately.
  3. The test could falsely indicate no cancer is present when it truly is. This error is called a false negative. This is bad because it could lead to false hope and could cause the patient to ignore symptoms because, “The test said I was fine.”
  4. The test could falsely indicate that cancer is present when it truly is not. This error is called a false positive. This is bad because it is distressing and could lead to unnecessary and even harmful treatment. The test itself, because it uses radiation, even increases the risk of true cancer because of the unnecessary exposure to x-rays.

This table shows all the possibilities in a test for the presence of absence of a thing (like breast cancer, prostate cancer, a lie, AIDS, and so on). For mammograms, “Present” means that cancer is actually there, and “Absent” means that no cancer is there. For a PSA test, “Present” means a prostate cancer is actually there, and “Absent” means that it is not. For a Movie Reviewer, “Present” means you liked a movie, and “Absent” means you did not.

Test Table
Present Absent
Test + Good: True Positive Bad: False Positive
Test – Bad: False Negative Good: True Negative

“Test +” says that the test indicates the test said the thing (cancer) is present. “Test -” says that the test indicates the absence of the thing. For the Movie Reviewer example, “Test +” means the reviewer recommended a film.

There are two cells in this graph that are labeled “Good,” meaning the test has performed correctly. The other two cells are labeled “Bad,” meaning the test has erred. Study this table to be sure you understand how to read it because it will be used throughout this article.

 

Error everywhere

The main point is this: all tests and all measurements have some error. There is no such thing as a perfect test or perfect measurement! Mistakes always happen. This is an immutable law of the universe. Some tests are better than others, and tables like this are necessary to understand how to rate how well a particular test performs.

 

Posted inCulture

Can You Read My Mind?

Ghosts, ESP, telekinesis, astrology, and other assorted oddities are back in view. One of the "SyFy" channel's most popular series is a show about hunting apparitions. The movie Men Who…
Posted inCulture Statistics

Disparity is inevitable: a counter argument to filing discrimination lawsuits

Introduction

Know a lawyer who is involved in a discrimination lawsuit? Particularly one in which the plaintiff alleges discrimination because actual disparities are found in company hiring practices? Were you aware that, just by chance, a company can be absolutely innocent of discrimination even though they actually are found to have under-hired a particular group? No? Then read on to find out how.

What are diversity and disparity?

We discussed earlier that there are (at least) two definitions of diversity: one meaning a display of dissimilar and widely varying behaviors, a philosophical position that is untenable and even ridiculous (but strangely widely desired). The second meaning is our topic today.

Diversity of the second type means parity in the following sense. Suppose men and women apply in equal numbers and have identical abilities to perform a certain job. Then suppose that a company institutes a hiring policy that results in 70% women and 30% men. It can be claimed that that company does not properly express diversity, or we might say a disparity in hiring exists. Diversity thus sometimes means obtaining parity.

Disparity is an extraordinarily popular academic topic, incidentally: scores of professors scour data to find disparities and bring them to light. Others—lawyers—notice them and, with EEOC regulations in hand that call such disparities illegal, sue.

And it’s natural, is it not, to get your dudgeon up when you see a statistic like “70% women and 30% men hired”? That has to be the result of discrimination!

Of course, it was in the past routinely true that some companies unfairly discriminated against individuals in matters that had nothing to do with their ability. Race and sex were certainly, and stupidly, among these unnecessarily examined characteristics. Again, it’s true that some companies still exhibit these irrational biases. For example, Hollywood apparently won’t hire anybody over the age of 35 to write screenplays, nor will they employ actors with IQs greater than average.

Sue ’em!

It’s lawsuits that interest us. How unusual is a statistic like “70% women and 30% men hired”? Should a man denied employment at that company sue claiming he was unfairly discriminated against? Would we expect that all companies that do not discriminate would have exactly 50% women and 50% men? This is a topic that starts out easy but gets complicated fast, so let’s take our time. We won’t be able to investigate this topic fully given that it would run to a monograph-length document. But we will be able to sketch an outline of how the problem can be attacked.

Parity depends on several things: the number of categories (men vs. women, black vs. white, black men vs. black women vs. white men vs. white women, etc.; the more subdivisions that are represented, the more categories we have to track), the proportion those categories exist in the applicant population (roughly 51% men, 49% women at job ages in the USA; we only care about the characteristics of those who apply to a job and not their rates in the population), the exact definition of parity, the number of employees the company has, and the number of companies hiring. That last one is the one everybody forgets and is the one that makes disparities inevitable. Let’s see why.

Men vs. Women

Throughout all examples we assume that companies hire blindly, that they have no idea of the category of its applicants, that all applicants and eventual hires are equally skilled; that is, that there is no discrimination in place whatsoever, but also that there is no quota system in place either. All hires are found randomly. Thus, any eventual ratio of observed categories in a company is the result of chance only, and not due to discrimination of any kind (except on ability). This is crucial to remember.

First suppose that there are in our population of applicants 51% men and 49% women.

Now suppose a company hires just one employee. What is the probability that that company will attain parity? Zero. There is (I hope this is obvious) no way the company can hire equal numbers of men and women, even with a quota system in place. Company size, then, strongly determines whether parity is possible.

To see this, suppose the company can hire two employees. What is the probability of parity? Well, what can happen: a man is hired first followed by another man, a man then a woman, a woman then a man, or a woman followed by another woman. The first and last cases represent disparity, so we need to calculate the probability of them occurring by chance. It’s just slightly over 50%.

(Incidentally, we do need to consider cases where men are discriminated against: in the past, we could just focus on cases where women were, but in the modern age of rampant tort lawyers, we have to consider all kinds of disparity lawsuits. For example, the New York Post of 12 May 2009, p. 21, writes of a a self-identified “white, African, American” student from Mozambique who is suing a New Jersey medical school for discrimination.)

Now, if a woman saw that there were two men hired, she might be inclined to sue the company for discrimination, but it’s unlikely. Why? Because most understand that with only two employees, the chance for seeming, or false discrimination is high; that is, disparity resulting by chance is pretty likely (in fact, 50%).

So let’s increase the size of our company to 1000 employees. Exact parity would give us 510 men and 490 women, right? But the probability of exact parity—given random hiring—is only 2.5%! And the larger the company the less it is likely exact parity can be reached.

Posted inFun

I’m thinking of turning to crime

He is a man of good birth and excellent education, endowed by nature with a phenomenal mathematical faculty. At the age of twenty-one he wrote A Treatise on the Binomial…
Posted inPhilosophy

Stats 101: Chapter 8

Here is the link.

This is where it starts to get complicated, this is where old school statistics and new school start diverging. And I don’t even start the new new school.

Parameters are defined and then heavily deemphasized. Nearly all of old and new school statistics entire purpose is devoted to unobservable parameters. This is very unfortunate, because people go away from a parameter analysis far, far too certain about what is of real interest. Which is to say, observable data. New new school statistics acknowledges this, but not until Chap 9.

Confidence intervals are introduced and fully disparaged. Few people can remember that a confidence interval has no meaning; which is a polite way of saying they are meaningless. In finite samples of data, that is, which are the only samples I know about. The key bit of fun is summarized. You can only make one statement about your confidence interval, i.e. the interval you created using your observed data, and it is this: this interval either contains the true value of the parameter or it does not. Isn’t that exciting?

Some, or all, of the Greek letter below might not show up on your screen. Sorry about that. I haven’t the time to make the blog posting look as pretty as the PDF file. Consider this, as always, a teaser.

For more fun, read the chapter: Here is the link.

CHAPTER 8

Estimating

1. Background

Let?s go back to the petanque example, where we wanted to quantify our uncertainty in the distance x the boule landed from the cochonette. We approximated this using a normal distribution with parameters m = 0 cm and s = 10 cm. With these parameters in hand, we could easily quantify uncertainty in questions like X = “The boule will land at least 17 cm away” with the formula Pr(X|m = 0 cm, s = 10 cm, EN ) = Pr(x > 17 cm|m = 0 cm, s = 10 cm, EN ). R even gave us the number with 1-pnorm(17,0,10) (about 4.5%). But where did the values of m = 0 cm and s = 10 cm come from?

I made them up.

It was easy to compute the probability of statements like X when we knew the probability distribution quantifying its uncertainty and the value of that distribution?s parameters. In the petanque example, this meant knowing that EN was true and also knowing the values of m and s. Here, knowing means just what it says: knowing for certain. But most of the time we do not know EN is true, nor do we know the values of m and s. In this Chapter, we will assume we do in fact know EN is true. We won?t question that assumption until a few Chapters down the road. But, even given EN is true, we still have to discern the values of its parameters somehow.

So how do we learn what these values are? There are some situations where are able to deduce either some or all of the parameter’s values, but these situations are shockingly few in number. Nearly all the time, we are forced to guess. Now, if we do guess?and there is nothing wrong with guessing when you do not know?it should be clear that we will not be certain that the values we guessed are the correct ones. That is to say, we will be uncertain, and when we are uncertain what do we do? We quantify our uncertainty using probability.

At least, that is what we do nowadays. But then-a-days, people did not quantify their uncertainty in the guesses they made. They just made the guesses, said some odd things, and then stopped. We will not stop. We will quantify our uncertainty in the parameters and then go back to what is of main interest, questions like what is the probability that X is true? X is called an observable, in the sense that it is a statement about an observable number x, in this case an actual, measurable distance. We do not care about the parameter values per se. We need to make a guess at them, yes, otherwise we could not get the probability of X. But the fact that a parameter has a particular value is usually not of great interest.

It isn’t of tremendous interest nowadays, but again, then-a-days, it was the only interest. Like I said, people developed a method to guess the parameter values, made the guess, then stopped. This has led people to be far too certain of themselves, because it?s easy to get confused about the values of the parameters and the values of the observables. And when I tell you that then-a-days was only as far away as yesterday, you might start to be concerned.

Nearly all of classical statistics, and most of Bayesian statistics is concerned with parameters. The advantage the latter method has over the former, is that Bayesian statistics acknowledges the uncertainty in the parameters guesses and quantifies that uncertainty using probability. Classical statistics?still the dominate method in use by non-statisticians1?makes some bizarre statements in order to avoid directly mentioning uncertainty. Since classical statistics is ubiquitous, you will have to learn these methods so you can understand the claims people (attempt to) make.

So we start with making guesses about parameters in both the old and new ways. After we finish with that, we will return to reality and talk about observables.

2. Parameters and Observables

Here is the situation: you have never heard of petanque before and do not know a boule from a bowl from a hole in the ground. You know that you have to quantify x, which is some kind of distance. You are assuming that EN is true, and so you know you have to specify m and s before you can make a guess about any value of x.

Before we get too far, let?s set up the problem. When we know the values of the parameters, like we have so far, we write them in Latin letters, like m and s for the Normal, or p for the binomial. We always write unknown and unobservable parameters as Greek letters, usually ? and ? for the normal and ? for the binomial. Here is the normal distribution (density function) written with unknown parameters:

(see the book)

where ? is the central parameter, and ? 2 is the variance parameter, and where the equation is written as a function of the two unknowns, N(?, ?). This emphasizes that we have a different uncertainty in x for every possible value of ? and ? (it makes no difference if we talk of ? or ? 2 , one is just the square root of the other).

You may have wondered what was meant by that phrase “unobservable parameters” last paragraph (if not, you should have wondered). Here is a key fact that you must always remember: not you, not me, not anybody, can ever measure the value of a parameter (of a probability distribution). They simply cannot be seen. We cannot even see the parameters when we know their values. Parameters do not exist in nature as physical, measurable entities. If you like, you can think of them as guides for helping us understand the uncertainty of observables. We can, for example, observe the distance the boule lands from the cochonette. We cannot, however, observe the m even if we know its value, and we cannot observe ? either. Observables, the reason for creating the probability distributions in the first place, must always be of primary interest for this reason.

So how do we learn about the parameters if we cannot observe them? Usually, we have some past data, past values of x, that we can use to tell us something about that distribution?s parameters. The information we gather about the parameters then tell us something about data we have not yet seen, which is usually future data. For example, suppose we have gathered the results of hundreds, say 200, of past throws of boules. What can we say about this past data? We can calculate the arithmetic mean of it, the median, the various quantiles and so on. We can say this many throws were greater than 20 cm, this many less. We can calculate any function of the observed data we want (means and medians etc. are just functions of the data), and we can make all these calculations never knowing, or even needing to know, what the parameter values are. Let me be clear: we can make just about any statement we want about the past observed data and we never need to know the parameter values! What possible good are they if all we wanted to know was about the past data?

There is only one reason to learn anything about the parameters. This is to make statements about future data (or to make statements about data that we have not yet seen, though that data may be old; we just haven?t seen it yet; say archaeological data; all that matters is that the data is unknown to you; and what does “unknown” mean?). That is it. Take your time to understand this. We have, in hand, a collection of data xold , and we know we can compute any function (mean etc.) we want of it, but we know we will, at some time, see new data xnew (data we have not yet seen), and we want to now say something about this xnew . We want to quantify our uncertainty in xnew , and to do that we need a probability distribution, and a probability distribution needs parameters.

The main point again: we use old data to make statements about data we have not yet seen.