A Cautionary Tale from Dr. Melton's Lab At Harvard

The soundtrack for this blog posting is the blues: Ball Peen Hammer sung by Joe Bonamassa:
http://grooveshark.com/#!/s/Ball+Peen+Hammer/2pulWL

This is a cautionary tale of research from Dr. Doug Melton's lab at Harvard.  I considered carefully if I should blog on this subject.  It is certainly a lot more fun to blog about successes, and new opportunities, than it is to blog about failures and mistakes.  However, I think it is critical to include both good news and bad news.  At the very least, so that people following type-1 research understand that mistakes get made, and that one of the strengths of the scientific method is the ability to recover from mistakes.

This posting is about Betatrophin, which I did not cover in my blog when it was first announced. The research was all done on animals, and I focus on human trials.  However, now that this research has gone full circle, I think it makes a useful scientific morality tale.  Obviously, lots of scientific research doesn't pan out for one reason or other.  Usually it dies a quiet and obscure death.  But this research was a little higher profile, and therefore it's death was a little higher profile as well, so more about it's life and death is known, than about your average research dead-end.

History

In early 2013 Dr. Melton's lab at Harvard released a "big news" research paper. They had identified a natural human hormone which caused beta cells to naturally regrow.  This one hormone, which they named "Betatrophin" had a huge impact in beta cell regrowth.  The research had been done in mice, and was published in Cell (a prestigious scientific journal).  Two big name pharmaceutical companies (Evotec and  Janssen Pharmaceuticals, a subsidiary of Johnson and Johnson) paid millions of dollars for the rights to Betatrophin.  The lead author of the paper (Dr. Peng Yi) was hired by an important diabetes research center (Joslin) to do research, including future clinical trials focused on Betatrophin.

But about 18 months later, it all collapsed.  A research group working for Regeneron Pharmaceuticals found that Betatrophin did not cause beta cells to regrow, and submitted a paper to that effect.  When the paper was sent to Dr. Melton for peer review, he endorsed it, and wrote a "Perspective" stating that his own lab had been unable to reproduce it's own previous findings.  It now appears that the conclusion in the first paper was incorrect.  I want to stress that no one has suggested that there was any fraud or mistakes in the research; it is just that the conclusion turned out to be wrong.

Discussion of Fallout

Betatrophin was the big news from this lab in 2013, but what about the big news in 2014?  In 2014 Dr. Melton's lab released a "big news" research paper showing that they could grow large amounts of functional beta cells from embryonic stem cells.  (Again, I did not blog on this, because it was not yet being used in a clinical trial.  However I was asked about it specifically on CWD, and posted some comments.)  How does the failure of the 2013 news, affect how we view the 2014 news?  This is the question that should be in the front of the mind of everyone following research aimed at curing type-1 diabetes.  Does the collapse of Betatrophin suggest that the beta cell breakthrough might collapse?

There is no way for me to have any insight into that question.  A pessimist would say that a lab that is wrong with one thing, could easily be wrong about the next thing;  that whatever caused the first incorrect conclusion could still be there for the second one.  An optimist would say that testing Betatrophin is a very different technology than transforming embryonic stem cells into beta cells, and the fact that the lab was wrong about one does not mean it is wrong about the other.

Of course, this brings up this question: if a world class scientist, working at one of the most highly regarded universities in the world, publishing in one of the most prestigious journals in the world, can still be wrong, how are we -- everyday people affected by a disease, but without extensive scientific knowledge -- to know what is correct and what is wrong?  How will we ever know?  (I discuss this in the conclusion section.)

Discussion of Peer Review

One of the interesting "side issues" that this brings up is conflict of interest in peer review.  When the scientific journal Cell got the paper showing that the research done in Dr. Melton's lab was wrong, they sent it to Dr. Melton as part of peer review.  I was a little shocked by that; but it makes sense in a pure-science sort of way.  Dr. Melton is a world expert on Betatrophin (by virtue of being "senior author" of the paper discovering it's function) so it makes perfect sense to ask him to peer review this paper.  However, from a human point of view, it seems nuts to have a person peer review a paper that directly undermines his own paper.  Even if there is not a monetary conflict of interest, there certainly is an intellectual one!  We were all well served by Dr. Melton's ethical actions after he was asked to be a peer reviewer.  But the opposite can happen as well.

This policy, of having papers which contradict previously published papers (in the same journal) reviewed by the authors of the previous paper, appears to be common in scientific journals.  At least it used to be.  The big name journal Nature did this in the 1990s.  More shocking, it then did not publish the second paper, based on a bad review from the author of the first paper.  Two other reviewers gave the second paper good reviews. You can read the sorry tale here:
http://retractionwatch.com/2013/06/19/why-i-retracted-my-nature-paper-a-guest-post-from-david-vaux-about-correcting-the-scientific-record/

Corporate vs. University Research

There is the growing trend to ignore corporate research in favor of academic research, often under the guise of "conflict of interest".  Obviously, some of this is a well earned reaction to various corporate attempts to manipulate scientific research (nuclear, tobacco, and pharmaceutical industries, just to name the ones who have gotten caught at it).

However, it is important to not take that attitude to extremes.  There are pressures to deliver in academia and the non-profit world as well.  In this case the wrong results were from university research and the right results came from industry.  I think it is important to remember that, in the face of a growing "corporation = evil" narrative.

The Original Paper Has Not Been Retracted

An interesting question (at least it's interesting to me) is should this paper be retracted?  It hasn't been, and it doesn't look like it will be.  It is certainly wrong in it's conclusions, but is that enough to retract it?   There are two schools of thought here.   One says that publications should only be retracted if they are "in error", meaning there was an error in design, data collection or analysis, or if there was fraud or ethics problems.  The other school of thought says that being wrong is enough.  If the authors / editors / publishers are sure it is wrong, then it should be retracted.

This sounds like a good topic of debate in a college level ethics class, with a scientific bent.
But one of the existing complications, is that there is no global standard.  Each publication is free to make their own decisions about retractions, and even in one publication, they don't have to be internally consistent, if they don't want to be.

My Summary

First, the press:

I have a very low opinion of how "the press" (ie. mass market news web sites) covers science and medicine.  Among their other sins, I think they over hype certain medical news, based on buzz words in press releases.  Although these "over hype triggering buzzwords" change slightly over time, a good PR firm or savvy researcher keeps up to date, so they can use the trendy buzz words to manipulate the press coverage.to get more than their research deserves.

For the last decade, I think "stem cell" is one of those triggering buzz words, and I think "Harvard" is one, too, and reports from big name universities are generally over hyped, in relation to universities with less name recognition.

Second, how to determine that science is correct:

This incident reinforces my belief that the only way to be sure a scientific paper is correct, is to follow it's research for a period of years after the paper is published, to see what happens.  There is no way to look at a freshly printed paper and know that it is correct.  I know that a lot of people try to take short cuts, and they say (or think) things like "it is a peer reviewed paper [so it must be right]", "the researcher is one of the most famous people in the field [so it must be right]", "the researcher is at an ivy league calibre institution [so it must be right]", "it was published in the leading journal in it's field [so it must be right]", "the researcher's family is affected by the disease so he's totally committed to the research [so it must be right]", "he is so personable, so articulate, and the description of what is happening makes perfect sense [so it must be right]", "the researcher has a great history and a great reputation [so it must be right]", and so on.

For Betatrophin, every one of those statements was true, and yet the research was flat out wrong.

This case is a clear example where the only way to see if research was correct, was to wait and see what happened to the research over time, as people tried to capitalize on it, to build on it, and to productize it.  The take home point is simple: there is no short cut.  There is no way to know quickly if research is correct.  Only the passing of years will tell us with certainty.

Now, I certainly don't claim perfection in this regard. I'm sure I've fallen into this trap myself. But it still is a trap, and something to be avoided.  I do stress peer reviewed results in this blog, almost to the exclusion of non-peer reviewed results, but that is different than implying that peer review means the research is good.  I consider peer review to be a necessary component of good research, but not sufficient to prove good research.  Put another way: if research is not peer reviewed, then it is not worthy of serious consideration.  But even if it is peer reviewed, it still might be wrong.

Some people, optimists, might look on this example as a fluke, but I think that's a mistake.  Most research does not pan out.  Most phase-I clinical trials do not lead to a marketed treatment. Therefore, the failures described here are quite common.  Maybe even normal.  I'm posting about this particular case because it made a bigger than usual news splash.  I get emails from people asking "remember research project X from years ago?  Whatever happened to that?  Did the money making, type-1 conspiracy suppress it?".  Nine times out of ten, the answer is that the research just didn't pan out at the next level.  Like Betatrohpin, the first publication was in error, and nothing could be built on it.

I'm always very nervous about the number of people who assume because they have not heard of it, that means it was really successful, and was therefore suppressed.  It creates a mindset where failure is taken as evidence of conspiracy.  In a world of research, where most new ideas do not pan out, this is spectacularly dangerous logic.

To join the two parts of this conclusion, notice that news organizations have an impossible conundrum: they are judged on how quickly they get (incredibly superficial) articles about scientific research out on the web, yet the only way to know if those articles are accurate is to wait, often months or years!   Yet as long as we reward news sources for speed, and ignore their accuracy (the current situation) this is the news we are asking for.

More Reading

Betatropin news at the time:
http://news.harvard.edu/gazette/story/2013/04/potential-diabetes-breakthrough/
http://jdrf.org/blog/2013/discovery-of-new-hormone-may-impact-t1d/

The papers:
Harvard paper: http://www.cell.com/abstract/S0092-8674(13)00449-2
Regeneron Pharmaceuticals: http://www.sciencedirect.com/science/article/pii/S0092867414011763
The Perspective: http://www.cell.com/cell/abstract/S0092-8674(14)01177-5

Discussion and fallout:
Retraction Watch: http://retractionwatch.com/2014/11/10/i-kind-of-like-that-about-science-harvard-diabetes-breakthrough-muddied-by-two-new-papers/
Blog: http://www.ipscell.com/2014/10/the-betatrophin-blues/
Pubpeer: https://pubpeer.com/publications/8EAC9ED889CC6E498199B11B1BC135 and also https://pubpeer.com/publications/0F7B9DF45743305575A36FDBA7FC9A
Money issues: http://seekingalpha.com/article/1730632-companies-race-to-develop-the-next-novel-blockbuster-diabetes-drug-part-2

A note on titles: In the past I have not been consistent about it, but in the future, in this blog, I will attempt to refer to anyone with a doctorate (PhD, PharmD, DVM, etc.) or a medical degree (MD) as Dr. X Y the first time they are named, and Dr. Y thereafter.   (Sorry lawyers, on this blog, you're not doctors, no matter what your degree says. :-)

Joshua Levy 
http://cureresearch4type1diabetes.blogspot.com 
publicjoshualevy at gmail dot com
All the views expressed here are those of Joshua Levy, and nothing here is official JDRF or JDCA news, views, policies or opinions. My daughter has type-1 diabetes and participates in clinical trials, which might be discussed here. My blog contains a more complete non-conflict of interest statement. Thanks to everyone who helps with the blog.