Good scientists being careful

[DrRocket]

This is why experiments are checked and rechecked. Good work.

 

http://www.sciencenews.org/view/generic/id/331050/title/No_new_particle_from_second_detector

[mississippichem]

This is why experiments are checked and rechecked. Good work.

 

http://www.sciencenews.org/view/generic/id/331050/title/No_new_particle_from_second_detector

 

Shows how scientific theories are in fact constantly being diced/cut/edited/scratched and updated. Many who are skeptical of the scientific process would benefit from reading more about modern controversies in science. This is a prime example of how the peer-review and experimental repetition process tends to sort out problems over time. It is a sort of "evolution of information" to grossly abuse a metaphor. Useful ideas that reflect reality are "selected for" and tend to reproduce spin-off ideas, whereas not so solid ideas tend to die out.

 

Nice link. Many here need to read it.

[swansont]

Also a good reminder of why citing a single study can be insufficient to buttress a claim, especially when it runs counter to experiments that have already happened.

[timo]

This is why experiments are checked and rechecked. Good work.

I'm not really sure if you're serious or sarcastic. I found the whole story somewhat embarrassing, since for several reasons it was very likely that the bump is just a random bump in the first place. Think about it: the likeliness of the bump just being random fluctuations was given with something like 1:4000. So, ... imagine an experiment with a few thousand physicists working on it and analyzing the data as extensively as possible: how surprising is it that you find a deviation from the average that has a likelihood of only 1:4000? Exactly.

I'm not sure how widely known the bump in the data became world-wide. In Germany it was picked up by Spiegel Online, the largest German online newspaper, and sold as "physicists find possible evidence for a fifth fundamental force of nature" and "possibly the most important finding in physics since decades" (to the defense of the Tevatron people, this is really not what they claimed in their publication).

[CharonY]

So, it sounds like a multiple-hypothesis testing type of problem?

[DrRocket]

So, it sounds like a multiple-hypothesis testing type of problem?

 

No, it's a "make sure the data is real and the experiment is reproducible" problem.

[timo]

I'm not sure that I understood your question, CharonY. The hypothesis that has been tested was "the standard model predicts this curve correctly". This hypothesis is routinely performed on collider data, on a variety of curves. And in one these tests the deviation of the data from standard model predictions (more precisely: predictions of computer simulations of the standard model) and the experimental data was large (compared to the estimated error bars, not compared to the signal being tested). That is certainly worth of mentioning, and even publishing, since it is the particle physicist's job to look into such deviations in detail. Still, if you perform several thousand tests and one of them fail with "if it's a random deviation, then it's chance of appearance would be only 1:3700", then it is not very likely to be a scientific breakthrough (but "possibly", especially if you want to sell newspapers).

Edited June 12, 2011 by timo

[Cap'n Refsmmat]

Still, if you perform several thousand tests and one of them fail with "if it's a random deviation, then it's chance of appearance would be only 1:3700", then it is not very likely to be a scientific breakthrough (but "possibly", especially if you want to sell newspapers).

That's the very definition of the multiple-hypothesis problem. If you test thousands of hypotheses (thousands of different tests) and get a hit on one of them, it's not very likely to be a breakthrough.

 

Suppose you're going for p < 0.05 and you test 1000 hypotheses through 1000 different tests on your data. None of the hypotheses are true. By definition, you'll get around 50 p < 0.05 results, despite none of the hypotheses being true.

[CharonY]

What Cap'n said. Basically a variation of the low number/rare event fishing game.

[FarmForest Research]

As a biologist I never understood why P is at 0.05 what makes the 5 percent percentile so good or for 95% percent for the purists. I like fuzzy statistics..where 50% is good enough for me. I observe nature and I am usually correct.

 

like looking for neutrinos in a salt mine for proof is like looking at a dragonflies wings.

[Cap'n Refsmmat]

p = 0.50 does not mean you have a 50% chance of being correct. It means that for any case where you're wrong, you have a 50% chance of claiming you're right.

 

Combine that with a statistical power (odds of finding a relationship that does, in fact, exist) of 50% and you have a recipe for disaster.

 

Consider a test of 1000 drugs to see which ones have an effect on a certain tumor. Suppose 100 of the drugs do. I do a study to find out which.

 

Of the 900 that don't, I get a p < 0.50 result for 450. Of the 100 that do have an effect, I get a p < 0.50 result for 50. Hence I get 500 total "hits", of which only 10% are real.

 

Those aren't good odds.

[swansont]

p = 0.50 does not mean you have a 50% chance of being correct. It means that for any case where you're wrong, you have a 50% chance of claiming you're right.

 

Combine that with a statistical power (odds of finding a relationship that does, in fact, exist) of 50% and you have a recipe for disaster.

 

Consider a test of 1000 drugs to see which ones have an effect on a certain tumor. Suppose 100 of the drugs do. I do a study to find out which.

 

Of the 900 that don't, I get a p < 0.50 result for 450. Of the 100 that do have an effect, I get a p < 0.50 result for 50. Hence I get 500 total "hits", of which only 10% are real.

 

Those aren't good odds.

 

Plus you've missed half of the drugs that work. False positives and false negatives can both cause problems.

[CharonY]

In biology you need good statistics. Especially with high-dimensional data low statistical power and high false positive detections pose immense problems. Using Capn's example (and 10% "good drugs" is really very generous, the odds in reality are much lower) the likelihood of any given find to be really one of the good ones is around 10%. I.e. you are not better off than by randomly selecting a drug. Saves money on the study, though.

[SCIENCEROCS]

I think that it is very important to have the figures or even experiment checked and rechecked. I can't even tell you how many times I have been in the lab and see someone do something that could possibly alter the results of the experiment. I wonder how many times the results on really important experiments have been changed due to carelessness.

[Marat]

Although experimental accuracy is of course important, the major problem is that the conceptual units the researchers factor the phenomenal continuum into at the outset determines whether the experiments have the possibility to be worth much, and this initial decision is a matter of having the right hypothesis, which is a theoretical issue. An example will illustrate the problem.

 

In allopathic medicine, the foundational ontology over which induction operates are entities artificially constructed by the theory called 'diseases.' If persons A, B, C, and D are all very different in their somatotype and physiology, but they all have symptoms of essential hypertension, then they are all treated as being 'the same' since the disease entity, not the person, is the primary unit of analysis.

 

Thus if allopathic medicine tests a new drug for its effectiveness in treating essential hypertension and it proves effective in only 5% of those taking it, the drug will be dismissed as ineffective and the 5% positive cases will be explained away as placebo effects, statistical irregularities to be eliminated when the curve is smoothed out, or failure of the double-blind process, etc. Because the focus is on the disease entity and not the individual person, if the drug has not replied to the challenge of the disease entity, it is worthless.

 

But homeopathic medicine operates with a different foundational ontology focused on persons rather than diseases as its essential factors. If persons A, B, C. and D each have distinctly different somatotypes and physiologies, then they will each need a different treatment if they are feeling unwell for any reason. Whether they have any of allopathic medicine's ontological entities, that is, traditional 'disease things' such as essential hypertension or not is not the key issue in treating the patients successfully, since it is the person and not the disease who is treated.

 

So if homeopaths tested the same new anti-hypertension drug described above and found that it worked only in 5% of patients, they would regard it as a complete success for the special type of individual who constituted the 5% of patients in whom the drug stopped the symptoms of essential hypertension. Instead of rejecting the drug as useless as would the allopath focused on an ontology of disease entities, he would accept the drug as good for the 5% of freckled, pale-complexioned, neurasthenic ectomorphs in whom it worked, since that was the appropriate remedy for that type of person.

 

Now I'm not posing this example to say that homeopathic medicine is right: far from it. I'm just using it to show how the initial ontology into which data is unitized preliminary to the inductive process can totally change the significance of the experimental or investigational work done. So since theory always precedes induction, science is always going to be threatened with the uncertainties of theory-formation, and it can't just hide safely in induction.

[CharonY]

It is getting somewhat off-topic but anyway:

 

Personalized medicine is something on which whole study sections are working on. The challenge is to find indicators that are predictive with any kind of certainty for a given physiological status and even more challenging to bring it into a medical and finally pharmacological relevant context. There are some examples for genetic markers that are indicators whether a certain drug may or may not work, but overall our biological knowledge is so limited that most markers are based on statistical inference, with all the problems associated with it.

 

Homeopaths, naturally, have not solved the problem, either. The described study is in its design basically worthless as it provides no context.

At the very least a control group would have to be treated with a placebo. If the success rate is also 5% it simply means that we can treat people successfully with sugar (or water, the hoemeopath's standard treatment).

 

To see whether an individualized treatement with using homeopathic diagnostics is feasible one would have a large population tested with this methodology, assign them a treatment based on this methodology. Then half of the group would be treated with according to the suggestions, the other half receives placebo treatments. Then one can see whether they really have a method for individualized diagnostics (I faintly recall that such a kind of study has actually been conducted, but forgot about details).

 

Few are arguing that personalized diagnostics is, in theory, superior to bulk diagnostics. However, we lack the knowledge to do so.

[Marat]

Perhaps the example was misleading, but I'm arguing that since our perceptual dispositions, depth grammar, intellectual wiring, and language predispose us to recognize certain similarity spaces as 'natural,' our science begins its investigations from the implicit 'natural kinds' we operate with, which may not be the natural kinds that actually exist in nature, or may disguise the most informative patterns out there. But since our inductive methods usually begin by operating over what we think are natural kinds, the strictest empiricism is already covertly informed by a certain ideology which may lead to misleading results. Similarly, an overlay of theoretical doctrine may also confuse attempts to base science strictly in empirical evidence.

 

Thus what if the most scientifically informative patters in a forest for understanding how forests grow and develop is not most rationally based on the type of units we are predisposed to have our inductive processes operate over, such as trees, grass, leaves, etc., but over novel unitizations of nature which seem unnatural to us, such as the colors or the smells? What if the best laws to describe the universe should have been based on how each planet tastes to a codfish, rather than on how the planets move, and what we should be doing is putting the mouth of a live (Mercury, etc., are out!) codfish up to each planet and using electrodes to record what it senses?

 

Theoretical overlays are more commonly noted as blocks to inductive discovery of nature's reality, but they just represent a more potentized version of the perceptual/linguistic natural kinds screen between us and nature. Thus one of the most commonly demonstrated experiments in 18th and early 19th century physics involved showing students that electricity and magnetism are not related, but only Oersted's famous 'accident' showed that the reality was being disguised by the contemporary theory of how Newtonian forces had to operate.

[CharonY]

Perhaps the example was misleading, but I'm arguing that since our perceptual dispositions, depth grammar, intellectual wiring, and language predispose us to recognize certain similarity spaces as 'natural,' our science begins its investigations from the implicit 'natural kinds' we operate with, which may not be the natural kinds that actually exist in nature, or may disguise the most informative patterns out there. But since our inductive methods usually begin by operating over what we think are natural kinds, the strictest empiricism is already covertly informed by a certain ideology which may lead to misleading results. Similarly, an overlay of theoretical doctrine may also confuse attempts to base science strictly in empirical evidence.

 

 

Every scientist worth his salt is aware that the categorizations we do are oversimplifications and often done in order to create the context for testable hypotheses. A part of scientific endeavors is squarely aimed at developing methodologies that e.g. create more accurate physiological models.

It is both, much more and much less of an issue than you make of it. Less, because it is pretty much well-known. It is more of an issue because it highlights our limitations of biological systems and complexity.

Therefore the use of mathematical models are powerful, as they provide a quantitative context in which we interpret how the system works. Instead of normal and diseased we would be able to work on the natural continuum. However, we are unable to obtain this goal yet.

To summarize, biology is bloody complicated and we need much more basic knowledge (IMO).