COVID-19

[nomadscientist]

Is it customary during the inchoate period of an epidemic/natural disaster/man made disaster or any event with far reaching health consequences to see this type of scientific output i.e. all the Dr. Lemmings running into the scientific sea?

It happens even with far weaker stimuli such as a topic becoming known to be increase one's likelihood of being awarded an invited talk at a domain-specific conference. After all "scientists" dont make the system and they gotta pay their mortgages.

[Lemon]

@Jacob

A fair bit of what is produced won't even be 'data' of any quality. This is a beyond fashionable topic to be publishing on and given how much is currently not even going through peer review being quick and first is being prioritised...This is how treatments etc. get overhyped before being found to be duds months/years later.

[tonyedgecombe]

@fiby41 From what I can tell Modi has handled the situation with remarkable courage and success, notwithstanding the enormous scale of the challenge. I would be interested to hear your perspective.

Kerala seems to be coping particularly well.

[Seppia]

https://www.reuters.com/article/us-heal ... 370OQ?il=0

This is pretty big.
After dropping the ball mightily at the very beginning, my compatriots have been extremely careful and always have erred on the side of caution.
So this is not some MAGA dude saying it's a hoax or something.
San Raffaele is the most important hospital in Milan, and what's stated echoes what I've been hearing through my dad's sources (he's a doctor in Lombardy).

I'm cautiously hopeful

[chenda]

@tonyedgecombe - Interesting, I believe Kerala is amongst the most the most developed part of India.

@seppia - Hopefully good news, though I'm predicting the Mediterranean is going to try and salvage a remnant of the tourist season and open up to flights from northern europe, which could lead to a second wave. I' booked to visit Ibiza in October but even if possible I'm not sure I should even if I can...

[Lemur]

Here in the United States, I expect COVID to cause many deaths. I guess herd immunity is only option basically...

https://www.youtube.com/watch?v=rRA1LTsyZqA

[theanimal]

Alaska had it's single largest absolute increase in positive cases today, 27 (460 total). 2 days after getting it's highest of 13. There were 3 days with 0 cases in the past week and business are fully open with no restrictions. My area has only had 2 cases in the past month, most of the cases are centering around Anchorage which is still under restrictions. The governor also lifted the travel restrictions that required all those travelling into the state to undergo mandatory 14 day quarantine. Now you can skip the quarantine if you can provide a negative test within 3 days of arrival or take one at the airport. Cruises are completely cancelled for the summer but fishing season is still on which will bring an influx of people from out of state. I'm not sure what this means going forward but it is not trending well..

[jennypenny]

NYT: There Has Been an Increase in Other Causes of Deaths, Not Just Coronavirus

New York and New Jersey have had more than 44,000 deaths above normal from mid-March to May, according to a New York Times analysis of data from the C.D.C. While Covid-19 is the leading cause of these excess deaths, more people have also died from other causes like heart disease, diabetes and Alzheimer’s disease in recent weeks than for the same period in previous years.

[bostonimproper]

@jennypenny Interesting piece. Seems like a combination of caretaker penalties (i.e. too busy with COVID-19 response to deal with patients that need a lot of help day to day) and increased heart disease. Based on COVID-19 being somewhat of a blood-clotting / inflammation disease, I'd be willing to assume some of that is misattribution and caused by the virus.

In other news, Governments and WHO changed Covid-19 policy based on suspect data from tiny US company (The Guardian).

Surgisphere, whose employees appear to include a sci-fi writer and adult content model, provided database behind Lancet and New England Journal of Medicine hydroxychloroquine studies

This is wracking the scientific community, which is up in arms that the elite institutions (premier journals, labs engaging in research, etc.) allowed these studies to get published with suspect data.

[Tyler9000]

In other news, Governments and WHO changed Covid-19 policy based on suspect data from tiny US company (The Guardian).

This is wracking the scientific community, which is up in arms that the elite institutions (premier journals, labs engaging in research, etc.) allowed these studies to get published with suspect data.

Holy crap. The data company is shady enough. But even more disturbing is the fact that major scientific institutions promoted the resulting research with no scrutiny. How on earth did this paper get published as-is and used as a predicate to shut down other HCQ studies? If it's truly as bad as it looks on the surface, heads have to roll.

[nomadscientist]

@jennypenny Interesting piece. Seems like a combination of caretaker penalties (i.e. too busy with COVID-19 response to deal with patients that need a lot of help day to day) and increased heart disease. Based on COVID-19 being somewhat of a blood-clotting / inflammation disease, I'd be willing to assume some of that is misattribution and caused by the virus.

In other news, Governments and WHO changed Covid-19 policy based on suspect data from tiny US company (The Guardian).



This is wracking the scientific community, which is up in arms that the elite institutions (premier journals, labs engaging in research, etc.) allowed these studies to get published with suspect data.

Ahahaha.

Everyone in the scientific community knows the data and codes are typically like this.

The non-elite ones are even worse.

[jacob]

This goes back to the earlier "discussion" about how the public largely doesn't know how scientific research works or what is really meant by science and instead substitutes in some ideal curated view that "science is a collection of facts" ... which [the process] indeed converges on eventually.

Normally, a research project comprises some data, some instruments, some code, and some analysis, that hopefully leads to some insight ("a result"). The lead author writes a paper that summarizes the pertinent result along with a description of how the result was arrived at in practice. This gets put on a preprint server so that others may access it. At the same time, it's submitted to a journal where it gets peer-reviewed. Now, a peer review is technically supposed to replicate and verify the entire experiment. This very rarely happens because it's simply impossible. The obvious constraint is that verifying it would take about as long as doing the experiment in the first place. Second, the paper might not publish all the codes. It might show summaries of the data rather than all the data. Insofar a suffering grad student spent three months calibrating an instrument, not all of this experience is or can be communicated leaving the verifying team to replicate the entire list of trial and error (null results or all the stuff that didn't work are seldomly published).

What happens instead is that other research groups consider whether this "result" is "interesting" or makes sense and they try to build on that. Insofar this effort leads anywhere, they will cite back to the original paper which will slowly attain the status of "verified". After all, while not having been subject to deliberate verification&validation like one would do in engineering, the method and results have been considered by a lot of people. Insofar they were not valid, some "shit" would have appeared as theory meets up with reality. If shit appeared, the paper just gets ignored or retracted. Consequently, after a few years, you can tell how "useful" a paper was by the number of citations it has accumulated.

IOW scientific confidence is not established just because something has been peer-reviewed. A scientist would find this notion laughable. It's slowly built up as results and methods are tied into the overall body of knowledge that's called science. Daylen covered some of that earlier on. Science is not "a bunch of facts" (like the multiplication table is apparently thought of as a collection of scientific facts---don't get me started...) in books or journals. Rather, it's an institutional and intellectual framework for increasing our knowledge about objective reality. The papers are just one part of this like the [unpublished] experience of the scientists is another or the codes on the computers are a third.

What happened with COVID was that "reviews" were fast tracked. Normally peer review takes a lot of time. The longest paper I've had in the grinder took almost two years because the @#$@#$ ref wanted me to run a !@#! parameter study on top of what I had already done. Normal from preprint submission to journal publication [in astrophysics] is more like 3-6 months for a standard paper. Obviously for a pandemic this waiting time is intolerable, so a fast-track mechanism (essentially a preprint server) was invented for more "experimental" treatments.

I think the public largely underestimates how decisions have to be made under uncertainty. Whatever happened made sense at the time. Hindsight is 20/20 and all that. In particular, the fact that this was eventually caught shows that the overall (the normal slow process) was working.

I would expect engineering to focus much more on verification and validation when building stuff out of known theory, but science is not engineering. Science is not verified and validated on a project by project basis. The goal of science is discovery, not construction. I don't know where medicine fits into this since treating a novel disease obviously contains aspects of both discovering treatments that might work and verifying that they actually do work.

[Lemon]

That is particularly bad but this is the issue of fashionable time critical topics.

Too many studies with too few patients meaning we don’t get a good idea of what actually works

See https://www.bmj.com/content/369/bmj.m1847

And https://www.bmj.com/content/bmj/369/bmj.m2045.full.pdf

Both from pre this latest issue

[bostonimproper]

RE all the scientists here laughing at my naivete: I would expect in most fields there are well-known data vendors using some sort of consortium standards that would do acquisition / normalization for these cross-institutional clinical datasets. AFAIK, that's how data and standards sharing usually works, at least in industry. That some mysterious new data provider nobody has ever heard of before was used was, I don't know, a subject of interest to anyone doing peer review is kind of shocking to me. How do you even do an analysis like this if you don't have access to the providers to ask about known issues, e.g. around collection methods, changes, etc.? This is basic stuff.

Also, obviously policymakers aren't going to be relying on every single study by its lonesome in decision making. But in this pandemic, it's hard to get clinical data on interventions from any single institution at a large enough scale to be statistically meaningful. So governments kind of have to rely more heavily on things like multi-registry analyses and literature reviews from domain experts, etc. to figure out as well as they can what's going on, and these things are going to be more heavily weighted. So I'm way madder about this than I would be about an anomalous single-institution study coming out with fraudulent results. /rant

[nomadscientist]

Now, a peer review is technically supposed to replicate and verify the entire experiment. This very rarely happens because it's simply impossible.

It's true that it never happens, but that isn't why it never happens. Let's take this seriously for a moment.

The obvious constraint is that verifying it would take about as long as doing the experiment in the first place.

That is proof that it is possible, not proof that it isn't. I can name problems that you cannot solve in the age of the universe. If someone already did the calculation, it's proof you can do it too.

Second, the paper might not publish all the codes. It might show summaries of the data rather than all the data.

Then failure to so publish would either be grounds to reject the paper, or the code would be reproduced from scratch (the only thing that would satisfy the requirement of "reproducing the experiment").

Insofar a suffering grad student spent three months calibrating an instrument, not all of this experience is or can be communicated leaving the verifying team to replicate the entire list of trial and error (null results or all the stuff that didn't work are seldomly published).

The purpose of the review is to verify that this trial and error converged on a correct answer, rather than an erroneous but presentable answer. So, it should be reproduced from scratch.

What happens instead is that other research groups consider whether this "result" is "interesting" or makes sense and they try to build on that. Insofar this effort leads anywhere, they will cite back to the original paper which will slowly attain the status of "verified". After all, while not having been subject to deliberate verification&validation like one would do in engineering, the method and results have been considered by a lot of people. Insofar they were not valid, some "shit" would have appeared as theory meets up with reality. If shit appeared, the paper just gets ignored or retracted. Consequently, after a few years, you can tell how "useful" a paper was by the number of citations it has accumulated.

Or, you publish an incorrect and influential result, cause malign impact in the world, be appointed a professor on the basis of your impact, and then build even more impact publishing papers discovering your own errors.

What happened with COVID was that "reviews" were fast tracked. Normally peer review takes a lot of time. The longest paper I've had in the grinder took almost two years because the @#$@#$ ref wanted me to run a !@#! parameter study on top of what I had already done. Normal from preprint submission to journal publication [in astrophysics] is more like 3-6 months for a standard paper. Obviously for a pandemic this waiting time is intolerable, so a fast-track mechanism (essentially a preprint server) was invented for more "experimental" treatments.

What really happened with COVID is that epidemiologists who were good at epidemiology failed in the political game against cowboys who did what needed to be done to get a job in a zero-stakes field. Then, the stakes suddenly became large.

[jacob]

@nomadscientist - Well, yes and no, depending on whether we're talking in idealistic principles or in terms of what's going on at the ground level($). It depends on whether scientific publications should be seen as "canonical results", that is, the end product of the scientific process; or whether they're seen as part of the process of how scientists make each other aware of what they're doing on par with conference talks, teaching lectures, "apprenticing" the next generation, or doing research visits.

($) I'm basing this mainly on my experience in nuclear astrophysics. I could easily imagine that the culture in e.g. chemical engineering science would be different.

The research world is effectively in the latter camp. I'm convincing that if some virus killed all scientists but left all the journal libraries intact, humanity would be SOL because science couldn't be recreated from the publications alone. It's like the difference between information and knowledge. The journals are just the way scientists communicate between each other. Operationally peer review is an attempt to make sure that people aren't talking bullshit, at least not for long. The resources to turn it into verification&validation aren't there because in practice the system eventually takes care of it.

For sure, this broke down once science-journalists started going to the same preprint servers and started communicating "the latest results" to the public w/o qualifying the veracity of said results. Ditto when this stuff made it into the non-research decision process. As you note, the short-term consequences of new science are usually minuscule (after all most new results are ignored until some time in the future). At worst some bench worker wasted a few months pursuing what turned out to be a type I error. In the long term it's pretty hard to propagate erroneous conclusions because scientists (the process itself) ultimately fix their type II errors(*) in the pursuit of building a theory that is consistent with itself and observations.

I don't know what the right way to get both accuracy and speed would have been. From my experience in financial engineering in building system responses, you can pick any two of speed, precision, and accuracy. In this case, the world wanted all three and got disappointed when it didn't get one of them.

(*) Maybe this warrants a bit of a discussion. To a researcher a type I error is not as big a deal as a type II error, because the experiment can always be done better. In much of the real world, type II errors are worse than type I errors because mistakes tend to be costlier. This difference is apparent in the salaries paid too.

[George the original one]

Expect USA total infections to officially hit 2 million next week. Brazil's official infection rate is now higher than the USA's.

[Tyler9000]

Since the coronavirus is only a few months old, my guess is the reviewers might have taken it for granted that there was a lot of mystery meat in the dataset.

I'll grant them that. But this case goes way beyond mystery in the data. The company itself appears to be a sham. Just a few examples from the Guardian article:

• A search of publicly available material suggests several of Surgisphere’s employees have little or no data or scientific background. An employee listed as a science editor appears to be a science fiction author and fantasy artist. Another employee listed as a marketing executive is an adult model and events hostess.

• Until Monday, the “get in touch” link on Surgisphere’s homepage redirected to a WordPress template for a cryptocurrency website, raising questions about how hospitals could easily contact the company to join its database.

• In 2008, Desai launched a crowdfunding campaign on the website Indiegogo promoting a wearable “next generation human augmentation device that can help you achieve what you never thought was possible”. The device never came to fruition.

BTW, here's another excellent article on the topic. Basically, there's an extremely high probability that the Lancet published a paper from a scam artist and the WHO used it as justification to shut down other legitimate research.

[fiby41]

HCQ:
ICMR studies show the chance of infection for high exposure healthcare workers falls by 80% after 4 doses (1 month.)
Sample size= 378 positives
Control group= 373 symptomatic negatives (58% male)
Could this be due to antibodies produced by then?
ICMR had approved use and gone ahead with their study despite WHO advisory.
Could the haste to cling to anything discrediting it be because Trump endorsed it?
114.5 million additional 400 mg tablets have been ordered from 2 firms by the central government.
200 countries/territories/governments had already received their shipments.

[nomadscientist]

@nomadscientist - Well, yes and no, depending on whether we're talking in idealistic principles or in terms of what's going on at the ground level($). It depends on whether scientific publications should be seen as "canonical results", that is, the end product of the scientific process; or whether they're seen as part of the process of how scientists make each other aware of what they're doing on par with conference talks, teaching lectures, "apprenticing" the next generation, or doing research visits.

Peer review purports to make a result canonical but in fact it is a means of enforcing influence networks and says very little about the correctness of a result. It usually means a glaring error isn't stated in the text, but not that a glaring error isn't present in the chain of reasoning used to arrive at that result.

The claim I replied to was that real reproduction was impossible when it is merely not requested or paid for. If the state (or whoever else is funding the research) really wanted to know the answer to a question that couldn't be immediately verified with an application they would fund siloed A and B teams by default.

($) I'm basing this mainly on my experience in nuclear astrophysics. I could easily imagine that the culture in e.g. chemical engineering science would be different.

In more engineering-y fields there is more of real scientific discipline. This isn't coming from the peer review process but from the fact that someone may try to actually use your results and the greater consequences of doing that and something going wrong. So my point made a few times in this thread: in the course of the career of the typical epidemiologist (not today, because they weren't appointed based on success in today's conditions) how much skin did a typical epidemiologist have in the game? Probably about as much as a typical nuclear astrophysicist.

The research world is effectively in the latter camp. I'm convincing that if some virus killed all scientists but left all the journal libraries intact, humanity would be SOL because science couldn't be recreated from the publications alone. It's like the difference between information and knowledge. The journals are just the way scientists communicate between each other. Operationally peer review is an attempt to make sure that people aren't talking bullshit, at least not for long. The resources to turn it into verification&validation aren't there because in practice the system eventually takes care of it.

Scientists communicate with each other using email, site visits, and beer. Functionally journal articles are scrip that is swapped for jobs and being an editor gives you power to pay or decline to pay people in your field (within some bounds).

For sure, this broke down once science-journalists started going to the same preprint servers and started communicating "the latest results" to the public w/o qualifying the veracity of said results. Ditto when this stuff made it into the non-research decision process. As you note, the short-term consequences of new science are usually minuscule (after all most new results are ignored until some time in the future). At worst some bench worker wasted a few months pursuing what turned out to be a type I error. In the long term it's pretty hard to propagate erroneous conclusions because scientists (the process itself) ultimately fix their type II errors(*) in the pursuit of building a theory that is consistent with itself and observations.

Erroneous conclusions are propagated on the timescale of real review (reproduction) and attempted application. Sometimes it's short, other times it's long. It's possible to get a job publishing erroneous but impactful stuff and if you then later "discover" your own errors you are very unlikely to be fired. Instead your increased paper count and citation count make you look like an even more productive researcher. The research system pays for "evidence of work" rather than correct results.

I don't know what the right way to get both accuracy and speed would have been. From my experience in financial engineering in building system responses, you can pick any two of speed, precision, and accuracy. In this case, the world wanted all three and got disappointed when it didn't get one of them.

Throwing the peer review process overboard is a no-brainer because it's essentially political and contributes little to the trustworthiness of the result. When pushed even the activists who claim peer review makes a result canonical won't object to this. The problems are:

1. what's the quality of your people? this is a result of processes already played through and unalterable on the pandemic timescale. if you selected people who are good at "gaming the system" producing "evidence of work" then your accuracy and speed at getting real answers will be lower

2. do you have A and B teams ready to go or is the whole field essentially a single influence network however spread across countries and universities? in most fields the answer is "no" because the networkers successfully review and publish one another's papers while outsiders like Newton do not publish, do not win grants, and are disemployed