Thinking About Things

su3su2u1:

Remember that optional stopping thing perversesheaf brought up?  

I was responding to a work email about some model performance issues, and I realized the email overlapped a lot with the optional stopping problem:

Frequentist confidence intervals are, by definition, what the LW crowd would call calibrated.  If you estimate hundreds or thousands of parameters over your career with 95% confidence intervals, then you’ll find that you missed your mark 5% of the time.  Frequentist statistics could really be thought of as rigorous way to make sure estimates are “calibrated.”

Bayesian credible intervals have no such calibration guarantees.   A Bayesian has no reason to believe his estimates are “calibrated” and studies show that generally Bayesian estimates only overlap with the frequentist confidence intervals asymptotically.  

The optional stopping problem is a special case of this where the Bayesian credibly interval is particularly poorly calibrated (0 overlap with the true parameter), but in general Bayesians should not expect to be calibrated. 

You know how else you can achieve perfect calibration? Guessing. If you guess randomly over the solution space, you’re perfectly calibrated. If you want to make intervals, just choose the entire interval space 95% of the time, and a blank interval 5% of the time.

Wow, my new epistemic theory is even simpler to calculate than frequentism! And produces just as good results!

raginrayguns:

wrapscallion:

Maybe one of the reasons that people may refuse to argue with me is not because they are afraid of their idea being destroyed, but because they are afraid that I will dismiss it for weak reasons? No argument and no resolution can possibly be deemed a matter that is still open for debate, rather than one that is closed and settled by dismissal.

my mom dislikes arguing with me because she thinks I apply unfair standards of evidence for the purpose of dismissing what she says. I say, “how do you know this is true,” and because I ask she assumes that I’m just trying to get her to admit there was no double-blind study so I can dismiss it.

So, she knows she’s right, and she knows that she has sufficient evidence to be right. But she doesn’t know that when I think of “evidence” I include non-conclusive stuff, that the evidence that’s good enough for her might be good enough for me too.

So she doesn’t want to argue because she doesn’t think I’m reasonable enough to be convinced by arguing. It’s like she’s saying, “don’t try to look at my evidence because you’ll draw the wrong conclusion, just trust me because then you’ll draw the right conclusion.”

This is written sot of from her perspective…. I think a lot of the time her evidence really as not any good

this is how it used to be anyway… we had a conversation in college where she told me this. But we don’t fight as often now

That almost exactly mirrors a conversation I had with my mother a bit ago (or at least my memory of my internal narrative at the time).

How did you resolve that, or does she still think of you as unreasonable? 

wenevergotusedtoegypt:

sootonthecarpet:

it’s weird thinking about it, but progressive Xians often blame homophobia and homophobic laws on Jews & now conservative Xians are blaming marriage equality on Jews too

What else is new? We’re also responsible for both capitalism and communism, are simultaneously filthy rich and poor leeches on society, are racist for wanting to marry other Jews but at the same time are promoting race mixing/trying to destroy the white race via intermarriage, and are simultaneously clever enough to be running the world but also inferior human beings.

peterhurford:

The Supreme Court of the US is basically an ensemble of nine different highly trained neural net models that run a binary classification of whether a problem is “constitutional” or “not constitutional”.  Each of these models was selected based on how well they had performed on an in sample data set of previous decisions when they were run on lower-level decisions.

Additions to the Supreme Court ensemble model are selected from a wider space of possible models by a single neural net model that is retrained every four years by an ensemble of over 300 million models that are, on average, poorly trained, but hopefully together in a sufficent ensemble large enough.  Many of these 300 million neural nets do not cast votes.  Also, these votes are not done in majority rule, but rather according to a different weighting method based on the geographic location of the server in which the model currently runs.

For additional verification, additions to the Supreme Court ensemble are also checked via an ensemble of 100 other highly trained neural net models, each of which was selected by a smaller subset of the 300 million ensemble mentioned earlier.

Uh, the classification is definitely not binary; in fact, many cases don’t involve constitutional issues at all.

Also, you spelled sufficent wrong.

su3su2u1:

thinkingornot:

su3su2u1:

Sometimes when I read something ridiculous on the internet that doesn’t jive with my sense of reality at all, I like to imagine there is some portal to another planet that the internet has some how breached.  

Last time I mentioned the Mandela Effect you got all upset.

I think imagine is different then claim to be the truth of things. 

I got the impression that many of the people talking about it didn’t actually believe it was true.

From http://woodbetweenworlds.blogspot.com/2014/06/commenting-on-berenstein-bears.html

9) I don’t really believe we switched universes.

Obviously, with my rational mind, I understand that the most reasonable explanation is that I misremembered.  Occam’s Razor and all that.

…

That said, I don’t really believe that we switched universes.  While my theory was certainly intriguing to me (and apparently no one else), I don’t actually believe it.  I don’t know if my theory describes the universe we live in.  Even if it did, I’d doubt we shifted.  The blog post was me having fun in a way only physics PhDs can, which is rationalizing totally irrational behavior with hyper-rational mathematical analysis.

I feel like there’s some good joke to be made about physics PhDs here, but can’t quite think of one.

su3su2u1:

Sometimes when I read something ridiculous on the internet that doesn’t jive with my sense of reality at all, I like to imagine there is some portal to another planet that the internet has some how breached.  

Last time I mentioned the Mandela Effect you got all upset.