A large number of people think that controlled testing of options is wrong

[Strange]

This result is just staggering and (to me) completely incomprehensible. And potentially really dangerous.

Almost 50% of people think it is wrong to run an "A/B test" (eg. a randomised controlled trial of two medical treatments) but few people think it is inappropriate to just impose either A or B, without knowing which is best. I just cannot understand the psychology behind this.

Quote

We find evidence across 16 studies of 5,873 participants from three populations spanning nine domains—from healthcare to autonomous vehicle design to policies to address global poverty—that people frequently rate field experiments designed to establish comparative effectiveness of two policies as inappropriate even when the policies those experiments compare are widely seen as appropriate. 

https://www.pnas.org/content/early/2019/05/08/1820701116

I just hope it is the other (sane) 50% who get to make the decisions!

[Dagl1]

While I of course can't talk for those people, I believe this occurs, in part, because people want to be in control of what they get and/or may deem 1 to be better than the other (even just based on browsing the internet) and thus may not want the 50% chance that they get the treatment that is in their opinion worse.

Dagl

[Strange]

11 minutes ago, Dagl1 said:

While I of course can't talk for those people, I believe this occurs, in part, because people want to be in control of what they get and/or may deem 1 to be better than the other (even just based on browsing the internet) and thus may not want the 50% chance that they get the treatment that is in their opinion worse.

Dagl

This is more general than the opinion of the person being offered treatment though (if I have understood your suggestion correctly). This is more about people's attitudes to the principal of doing the tests (on other people) or not.

They do attempt to identify the reasons for these attitudes (by a combination of questionnaires and further experiments) but there doesn't seem to be any one factor behind it.

[Endy0816]

Frequently read about Social media platforms doing this. Even if a member of the control group, leaves a bad taste in the mouth when you end up a test subject without any knowledge, consent or compensation.

Concerned about this practice for software testing in general too. Suppose it is something more serious like your car?

Edited May 10, 2019 by Endy0816

[Strange]

3 hours ago, Endy0816 said:

Frequently read about Social media platforms doing this. Even if a member of the control group, leaves a bad taste in the mouth when you end up a test subject without any knowledge, consent or compensation.

That is one of the examples they use. And consent is one of the reasons that people give for their misgivings. And in this case, it seems fair. Social media platforms haven't asked (or even told) people about these trials. 

The odd thing is, people still use "consent" as a reason even when consent is given (eg. medical trials).

Quote

Concerned about this practice for software testing in general too. Suppose it is something more serious like your car?

But they used examples where there is no reason to think either A or B is worse.

And if you (people generally) worry that you might get the "bad" one in a blind trial, why wouldn't you have exactly the same worry if the manufacturer just arbitrarily picks one? It seems that people think "well, they wouldn't have picked the one that isn't any good". But we can't know that until we do a comparative trial!

I can sort of see the emotional reaction, but it doesn't really make sense to me.

[Dagl1]

4 hours ago, Strange said:

This is more general than the opinion of the person being offered treatment though (if I have understood your suggestion correctly). This is more about people's attitudes to the principal of doing the tests (on other people) or not.

They do attempt to identify the reasons for these attitudes (by a combination of questionnaires and further experiments) but there doesn't seem to be any one factor behind it.

Oh yes I meant the people DOING the trials; Yes I agree the idea that this is general bad thing to do, even if YOU yourself don't join such a trial is bizarre....

[Intrigued]

It occurs to me that the explanation is as follows:

An almost universal tendency of humans is to lie.

Lying tends to increase in proportion to the severity of the consequences of not lying.

Lying is related to concealing information.

These tests, necessarily, conceal information.

Therefore the scenario is one in which there is concealed information and a perceived incentive to  conceal findings possibly prejudicial to the individual in the test.

Therefore, "Somethings going on here and I don't like it".

Your error Strange is that you are expecting people to act logically, when most will simply adopt the faux logic outlined above.

[swansont]

31 minutes ago, Intrigued said:

It occurs to me that the explanation is as follows:

An almost universal tendency of humans is to lie.

Lying tends to increase in proportion to the severity of the consequences of not lying.

Lying is related to concealing information.

These tests, necessarily, conceal information.

Therefore the scenario is one in which there is concealed information and a perceived incentive to  conceal findings possibly prejudicial to the individual in the test.

Therefore, "Somethings going on here and I don't like it".

Your error Strange is that you are expecting people to act logically, when most will simply adopt the faux logic outlined above.

I agree it's a logical error. Lying may be related to concealing information, but that does not make concealing information lying.

I have a secret, therefore I am concealing information. But that does not mean I am lying.

——

I think the conclusion of this report is that people will always find something to be outraged about.

[Intrigued]

40 minutes ago, swansont said:

I have a secret, therefore I am concealing information. But that does not mean I am lying.

Of course not, but you are thinking logically. Evidence and observation suggest you are, in the population at large, in a minority.

[swansont]

1 minute ago, Intrigued said:

Of course not, but you are thinking logically. Evidence and observation suggest you are, in the population at large, in a minority.

Perfectly consistent with my conclusion.

[Intrigued]

4 hours ago, swansont said:

Perfectly consistent with my conclusion.

Yes. I was agreeing with you and giving you a compliment.

[Endy0816]

On 5/10/2019 at 10:21 AM, Strange said:

That is one of the examples they use. And consent is one of the reasons that people give for their misgivings. And in this case, it seems fair. Social media platforms haven't asked (or even told) people about these trials. 

The odd thing is, people still use "consent" as a reason even when consent is given (eg. medical trials).

But they used examples where there is no reason to think either A or B is worse.

And if you (people generally) worry that you might get the "bad" one in a blind trial, why wouldn't you have exactly the same worry if the manufacturer just arbitrarily picks one? It seems that people think "well, they wouldn't have picked the one that isn't any good". But we can't know that until we do a comparative trial!

I can sort of see the emotional reaction, but it doesn't really make sense to me.

There is normally some rational basis for the options.

Could be another case where genes behind us seeking equal treatment for ourselves(and the group) are at fault too though.

Seen other case studies where people act against their own best interests in favor of maintaining parity.

[CharonY]

On 5/10/2019 at 3:52 AM, Dagl1 said:

While I of course can't talk for those people, I believe this occurs, in part, because people want to be in control of what they get and/or may deem 1 to be better than the other (even just based on browsing the internet) and thus may not want the 50% chance that they get the treatment that is in their opinion worse.

Dagl

I doubt that it is about control per se. Folks are alright is only presented on of those options, for example (i.e. when they have no control whatsoever). Moreover, when presented separately folks were fine with either choice in such a randomized trial.

I suspect that it is the thought of getting something worse than someone else is something that is at least one  of the reasons. In various psychological (and economic) tests (such as the ultimatum game, where on person decides the share of a given monetary amount and the second either rejects or accepts the deal) folks have been shown that folks would rather penalize unfair shares rather than accepting any amount of money (which would be the rational choice). Some similar thinking may play a role here.

The corollary is that theoretically people may have less misgivings if they were specifically recruited to either group separately. However, that would make blind trials and accounting for placebo impossible as well as raise the issue of non-randomized selection.

Edit:

A bit off-topic though it may be a similar mechanism: It is interesting to speculate what other impact these things have in other personal and political decision making. As recent studies have shown, economic loss is, for example, surprisingly not a strong predictor for radicalization (and goes against traditional wisdom, which assumes a rational decision-process). Instead, fear of unfair treatment and status loss or that someone could take ones share, are stronger motivators. I.e. the perception of potential loss seems to be stronger than the actual loss, which is somewhat mind-blowing bit also implies that many standard political assumptions are wrong.