paulsutton Posted September 16, 2022 Share Posted September 16, 2022 I spotted this on Twitter and decided to share as it is science related 19th to 23rd September is peer review week https://scholarlykitchen.sspnet.org/2022/09/15/ask-the-chefs-is-research-integrity-possible-without-peer-review/ I am not associated with this, just sharing the information. The question "Is Research Integrity Possible without Peer Review?" could make an interesting discussion at some point. Paul Link to comment Share on other sites More sharing options...
exchemist Posted September 16, 2022 Share Posted September 16, 2022 1 hour ago, paulsutton said: I spotted this on Twitter and decided to share as it is science related 19th to 23rd September is peer review week https://scholarlykitchen.sspnet.org/2022/09/15/ask-the-chefs-is-research-integrity-possible-without-peer-review/ I am not associated with this, just sharing the information. The question "Is Research Integrity Possible without Peer Review?" could make an interesting discussion at some point. Paul It's obviously possible, but how would the community have confidence in it? Furthermore the issue is not so much integrity, but quality control. There is plenty of poor science done in good faith, i.e. with integrity, but also little or no value. There are occasional scandals of bad faith or fraudulent science but the main job of peer review is to maintain quality, isn't it? Link to comment Share on other sites More sharing options...
swansont Posted September 16, 2022 Share Posted September 16, 2022 2 hours ago, exchemist said: It's obviously possible, but how would the community have confidence in it? Furthermore the issue is not so much integrity, but quality control. There is plenty of poor science done in good faith, i.e. with integrity, but also little or no value. There are occasional scandals of bad faith or fraudulent science but the main job of peer review is to maintain quality, isn't it? Agree. Peer review would not catch a reasonably well-executed fraud, because the description of the method would be acceptable, and the data would support the conclusion. It would look like good science. Peer review might not catch plagiarism, either. Link to comment Share on other sites More sharing options...
paulsutton Posted September 16, 2022 Author Share Posted September 16, 2022 3 hours ago, swansont said: Agree. Peer review would not catch a reasonably well-executed fraud, because the description of the method would be acceptable, and the data would support the conclusion. It would look like good science. Peer review might not catch plagiarism, either. You raise a good point here, Sense about science have some resources on peer review https://senseaboutscience.org/activities/peer-review/ How important is it, that the general population who are not from a science background understand these processes so that they can trust science. Surely there is little point in peer review if people don't understand / know what this is. If any science is also funded through taxation, then surely those conducting any research etc are accountable to tax payers. In turn should expect the highest standards to be upheld. So what can run along side peer review to help ensure the processes are robust and trustworthy? Paul Link to comment Share on other sites More sharing options...
swansont Posted September 16, 2022 Share Posted September 16, 2022 54 minutes ago, paulsutton said: Surely there is little point in peer review if people don't understand / know what this is. I'm pretty sure most scientists and science journalists know what peer review is, and that's the target audience. What's the impact of not knowing what peer review is to a layperson? They don't read the journals. Link to comment Share on other sites More sharing options...
CharonY Posted September 16, 2022 Share Posted September 16, 2022 20 minutes ago, swansont said: What's the impact of not knowing what peer review is to a layperson? They don't read the journals. I think most interested folks will come across the term a fair bit, as newspapers and popular science magazines do mention whether papers are peer reviewed or not. This has also become more common during this pandemic. So I would think many folks would have at least a vague idea what it means. Folks might see as a high bar rather than the minimum requirement it really is, though. 1 hour ago, paulsutton said: If any science is also funded through taxation, then surely those conducting any research etc are accountable to tax payers. In turn should expect the highest standards to be upheld. I think in recent times folks increasingly believe things to be true based on their beliefs rather than evidence (probably also depending on country). Link to comment Share on other sites More sharing options...
swansont Posted September 16, 2022 Share Posted September 16, 2022 1 hour ago, CharonY said: I think most interested folks will come across the term a fair bit, as newspapers and popular science magazines do mention whether papers are peer reviewed or not. This has also become more common during this pandemic. So I would think many folks would have at least a vague idea what it means. Folks might see as a high bar rather than the minimum requirement it really is, though. I think if they don't know the term there's no real damage, though, since the gatekeeper is probably the journalist writing an article, so the quality of information depends on whether they are doing their due diligence, not the consumer of the article. I agree that some people don't realize that it's the much lower hurdle of not being obviously wrong or having shoddy methodology. It can still be wrong. The next step is checking the "repeatability" box. Not just one experiment, but several of them, with different approaches, to ensure there's no hidden issues, or that you didn't just get lucky that your p value was below some arbitrary value, or some other statistical happenstance. Link to comment Share on other sites More sharing options...
paulsutton Posted September 16, 2022 Author Share Posted September 16, 2022 55 minutes ago, swansont said: I think if they don't know the term there's no real damage, though, since the gatekeeper is probably the journalist writing an article, so the quality of information depends on whether they are doing their due diligence, not the consumer of the article. I agree that some people don't realize that it's the much lower hurdle of not being obviously wrong or having shoddy methodology. It can still be wrong. The next step is checking the "repeatability" box. Not just one experiment, but several of them, with different approaches, to ensure there's no hidden issues, or that you didn't just get lucky that your p value was below some arbitrary value, or some other statistical happenstance. In the UK, part of digital literacy is being able to find information online, however it expands to finding legitimate / accurate and credible information based on evidence. It is part of a push to ensure people can do this, given the amount of information that is available. I think this is a move up from simply being able to looking things up, So understanding terms such as peer review is going to be essential to help people with this, it also helps them form arguments. https://www.bbc.co.uk/bitesize/guides/zxs2xsg/revision/1 An example I came across is trying to find a a recent how to for how to carry out a task on the Linux operating system, in many cases the first links that come up are also the links that have been read or followed the most, as they have been around longer, so you end up with a how_to for say Ubuntu 12.04 which is less than helpful. Paul 3 hours ago, CharonY said: I think in recent times folks increasingly believe things to be true based on their beliefs rather than evidence (probably also depending on country). Or what their friends tell them, as they see people as very knowledgable, even if they are not. So what they say on a topic goes. I think this article perhaps illustrates what we are up against and why it is important to learn how to find credible information TikTok Search Results Riddled With Misinformation: Report https://www.huffpost.com/entry/bc-us-tiktok-misinformation_n_6321c9b0e4b027aa40614814 Paul Link to comment Share on other sites More sharing options...
Recommended Posts