Gaming the peer review system in India

Pharmacies in India might be great, but one has to be careful with research coming out of India, a sad worldwide trend:

Let us keep in mind, that research misconduct happens in every country, and at every institution - including in labs run by Nobel Prize recipients.

As always, just because something is published doesn’t mean we should immediately trust it.

5 Likes

There is an amazing chart in the article showing paper retractions by country. China has more retractions than the next ten combined, just an epic amount!

Of course the interesting stat is retractions per 1000 published, but China is a champ in every category - the tsunami of fraud coming out of that country is impressive in its own way.

5 Likes

And I was surprised to see the IIT Rookie on the list. My general impression of the IITs in India is positive (we have many engineers from the IITs in the Silicon Valley).

The source for some of the information:

3 Likes

Thanks, RapAdmin, I don’t know to imded images here. I guess, the key takeaway for me is that while this kind of fraud/flimflam/gaming has always been around, it has really exploded just in the very recent times, so if one has had a decent opinion of research coming from some countries/institutions/researchers, we need to very rapidly readjust our expectations. Right now my sense of scepticism is very, very high for almost all papers, and the more “interesting” the result, the more scrutiny it should face, “extraordinary claims require extraordinary proof” and all that… but at this point I’m ready to say “rock solid proof at all times”. The presumption of innocence has taken a major hit, guilty until proven innocent, lol!

Just a reminder that those retractions in the chart are the total number. The percent of fraudulent papers is much different. For example, the USA has the second largest number of retractions, but the percentage as a whole is only around 10%. Japan has the lowest percentage. China and Russia have the highest percentages of fraudulent papers.

Right, that’s why I said the key metric is retractions per 1000 papers, or any metric which measures the proportion of legit vs retracted papers.

Of course, there is the issue of whether a paper is truly contributory, or just ‘salami slice science’, where you examine some very narrow and trivial issue. The usual measure that attempts to quantify this, is “citation number”, journal impact where the paper appears and such comparatives. Unfortunately, as with all attempts at measurement, they can be gamed. Same as it ever was. The old publish or perish imperative is a perfect example of a legit measurement attempt of productivity, and the result of gaming that measure, which in turn resulted in salami slice science. Of course, in recent times, many have abandoned such “innocent” tricks and simply plowed into full on fraud.

Until there are real consequences to such fraud, it will continue to proliferate virtuallu unchecked. The number of offenders so outpaces the “checkers” and investigators, that it’s a losing battle.

The worst is that the institutions are frequently complicit in this fraud. They have a “star researcher” who attracts grant money, endowments, attracts students, raises prestige and so forth, well, if the star transpires to be a fraud, the institution administrators are not eager to discipline, fire, present consequences, instead, they cover up. Everyone is in on it, from the admins to the other authors and reasearchers at the lab all of whom ride along on the coattails. Reviewers scratch each other’s backs. The loop closes. The big loser is science, and trust in science, and downstream from that (in medicine) drugs that don’t work or injure patients. And sadly, again, nowhere is immune, including top institutions like Stanford, Harvard and so on. By the time you get to some obscure institution in China, India, Russia etc., you are really looking at lottery type odds - like in economics, bad money pushes out good money. When the fraud is up and down the hierarchy, any honest researcher is just pushed out.

It’s hard not to become cynical, but the answer is always the same: follow the science, replicate and verify. And good ones still exist (ITP!).

I think it’s inevitable that something has to be done. Proven deliberate fraud should result in a permanent ban for those responsible. Fraud, perhaps should even have legal consequences, same as if you lie on a loan application or receive money (research grant), but deliver fraud (don’t fulfill the terms) - pay restitution and go to prison. Negative, null results and studies that simply attempt to replicate results should all be elevated in publishing and valued on a CV.

2 Likes

I was discussing some important research performed in Portugal with a professor recently. The professor said that the important research would probably not get in an journal because it did not fit with the conventional wisdom.

That is maybe an even bigger problem than lots of fraudulent research getting through.

I think pre-publication peer review’s day has gone. There is now a role for post publication review and perhaps some summaries of papers which are thought significant by enough people qualified in the area.

2 Likes