Friday, July 20, 2018

Dr. Faustman Publishes Follow On BCG Data From Phase-I Trial

Dr. Faustman published a paper [r2] and a poster [r1] that contained results from an extension to her Phase-I trial [r17].  These results have generated a lot of buzz, and the study was complex, so this is going to be a long post with four sections:  First, a quick summary.  Second, a more detailed discussion of her results.  Third, a discussion of the buzz surrounding those results.  And fourth, a discussion of where her research goes from here.  The [dN] marks mean that there is more discussion about this point at the end of the blog post, and the [rN] marks are references which are also at the end of the posting.

I've written a total of 18 blog entries on this line of research over the last 10+ years, and you can read them all here:

The short history is this: Dr. Faustman is trying to cure type-1 diabetes by using BCG, a widely used tuberculosis vaccine.  Dr. Faustman published phase-I clinical trial data in 2012, and is now publishing data from an extension to that trial. 

Quick Summary of Results and Importance

All of these points are discussed in a lot more detail below, especially including the data they are based on, and how I got from the data to these summary points:
  1. C-peptide is the FDA, researcher, and industry standard for evaluating cures for type-1 diabetes, and the C-peptide data reported here shows the BCG did not cause a rise in C-peptide levels.  These results mean that this extended phase-I trial has worse results/is farther away from a cure, than the initial phase-I study reported years ago.   The first phase-I showed tiny increases in C-peptide, but here, no clinically significant increases are seen.
  2. A1c data is generally used to evaluate treatments for type-1 (not cures).  The A1c data is the best data reported here (an improvement of about 0.8), and is similar to several other treatments already available or in later phases of clinical trials.  All the data presented here is based on a very small number of people who actually got BCG (3 people in some cases 12 in others).  Furthermore, it is inconsistent.  At some points in time the BCG group did worse than the control groups, while at other times they did better. 
  3. Dr. Faustman had a theory as to why BCG could cure type-1 diabetes.  The paper is clear that the A1c results seen here are not caused by this theory.  The paper presents a new theory to explain the cause of these results.  
  4. There is a phase-II trial underway (with results expected in 2023), so we will have more data then.
  5. And finally, the primary end point for the phase-I trial was autoreactive t-cells, and this extension to the phase-I trial did not include data on autoreactive t-cells, which means the clinical trial was unsuccessful.
Results and Discussion

As with all science, the important information is the results of the study, so let's take a look at the data presented (and the data that was not presented):

A1c Data
Graph is from Dr. Faustman's paper, and is presented
 for educational purposes only.

The best data in the paper was A1c data.  The graph at the right (taken from the paper) shows the A1c numbers for the people treated with BCG (in red) vs. two types of control groups (in black). [r2]

You can see that the people treated with BCG had worse A1c numbers for the first two years after the treatment.  They then improved noticeably for the next two years, and then gradually increased for the next four years.  Overall the patients had worse A1c numbers for the first two years, and better for the next six years.

Importance of the A1c Data

For me, this data does not support the idea that BCG is a cure for type-1 diabetes.  It doesn't even support the idea that BCG is a treatment for type-1 diabetes.

First and most importantly, A1c data is typically used to measure treatments not cures [d1].  This is for a very good reason: many things, unrelated to a cure, impact A1c numbers.  Being more aggressive about insulin dosing,  going on a low carb diet, using a CGM, or taking an SGLT or GLP-1 drug can all impact A1c numbers as seen here [d2], but none of them are a path to a cure.

On the other hand, the only thing that impacts C-peptide numbers is the body generating its own insulin [d3].  That is why researchers commonly use (and the FDA expects) C-peptide as the end point for clinical trials aimed at curing type-1 diabetes [d5].   Generating your own insulin and maintaining that production is what cures type-1 diabetes, and it is exactly what C-peptides measure.  This is an important point and discussed in detail in [r9] (the conclusions of the D-Cure workshop).

Second, the data above in inconsistent: worse for two years and better for six.  If you are going to argue that the good numbers (3-5 years out) are really caused by BCG, then you have to assume that the bad numbers (0-2 years out) are also caused by the BCG.  After all those numbers are closer to the BCG dose.  It seems much more reasonable to me to assume that neither the bad numbers nor the good numbers had much to do with the BCG dose.

Third, existing treatments have already shown better and more consistent improvements in A1c than are seen here.  These are described in more detail in [d2] and [r11-13].  If I wanted to get excited about new treatments for type-1 diabetes, BCG would get in line with the many treatments which have stronger evidence in larger clinical trials [d10].

Fourth, the eight year data is based on 3 people, and the five year data on 12 people, and there are two problems with these numbers.  The big, obvious problem is that they are tiny, especially the 8 year data [d6]. The second issue is that more people were added after the end of the phase-I trial.  This is unusual.  Normally an "extension" or a "follow on" trial simply follows the same group of people (or a subset) for a longer period of time.  It's quite unusual to see new patients added after the end of the trial as described by the clinical trial registry.   

The A1c results also have a serious problem with "results switching" described [d7] and [r16].

As a side note, even as a measure of treatment success, A1c is falling out of favor as compared to "time spent in range" and quality of life measures as proposed by the "Beyond A1c" movement:

C-Peptide Data

As stated previously, C-peptide numbers are the best measure of progress towards a cure [d4], so these are the numbers we should pay the most attention to.  Here are a few quotes from the paper: 
"The BCG-treated type 1 diabetic subjects at year 4 after glucagon challenge had a negligible to no return of clinically significant C-peptide. "
"The human pancreas after BCG even at four years after repeat vaccinations did not secrete significant insulin as clinically measured by C-peptide."
"Therefore we concluded that BCG vaccinations did not induce a clinically meaningful return of C-peptide levels in the pancreas by regeneration" [r2]
The paper reported that at 4 years (the point of highest A1c effect) the C-peptide numbers for the treated patients were "in the range of 2–3 pmol/L."  Table 1c in the paper included C-peptide numbers, but the numeric data was not included in the paper or the supplemental materials.  My eyeballing of the data is that the control group started off just below 2 pmol/L and the treated group started off just above 2 pmol/L.

Importance of the C-Peptide Data

In terms of measuring progress towards a cure, C-peptide data is the most important data.  When the FDA, EMA, or other researchers evaluate this study, it is the key data they will look at [d11].  It shows no progress towards a cure.  That is bad news for BCG-as-a-cure research.

Several Additional Metabolites

In addition to A1c and C-Peptide data, the paper also reported on a variety of metabolites.  These are various chemical markers of what is happening inside the body.  The purpose of these measurements is to try to figure out what was causing the changes to A1c seen in the study.  If you care about these details, then I urge you to read the paper [r2].

The statistically significant differences between people with type-1 who were given and not given BCG are summarized as follows:
In the purine pathway, adenine, N6-carbamoylthreonyladenosine, 7-methylguanine and N2,N2-dimethylguanosine all statistically showed significant increases in BCG-treated T1Ds compared to untreated T1Ds [r2]
Autoreactive T-Cell Data

This study reports on an extension to the phase-I trial, but it does not report on the primary outcome of that study (autoreactive t-cells) [r6].  In the world of clinical trials, this means this extension to the phase-I trial was unsuccessful.

A clinical trial is considered successful if there are good, statistically significant results for the primary outcome, using standard data analysis.  You can read a lot more about this definition here:
The key point is that not reporting on a primary end point, means the trial has failed.

The autoreactive t-cells results were involved in the "results switching" described [d7] and [r16].

The Change of Theory

Until this publication, Dr. Faustman believed that BCG worked by causing the body to generate more TNF, and this TNF caused the body to generate fewer autoreactive T-cells [r17].  Fewer of these bad T-cells resulted in a cure [d9].  She has published a few papers and edited a book on this theory [r10].

However, in this paper she makes it clear that this theory is not causing the A1c changes.  To quote her paper:
"The mechanism for lowered HbA1c values was not equivalent to the NOD [non-obese diabetic] mouse pancreas regeneration after BCG treatment" [r2]
And the paper describes a replacement theory:
"[BCG causes] a cellular switch from primarily oxidative phosphorylation, a low glucose utilization state, to augmented early aerobic glycolysis, a high glucose utilization state associated with high purine metabolism" [r2]
BCG lowers A1c by changing the way the body uses glucose, so that it burns more, which lowers blood glucose levels and therefore A1c numbers.  The new theory and the old theory are completely different.  Among other things, the old theory was based on immunology, while the new theory is based on glucose metabolism.  The new theory could replace the old theory, or both could be happening in parallel.

The Importance of The Change In Theory

Initially, I didn't think this change mattered much.  I'm much more focused on the question of effectiveness than mechanism.   (Put another way: I want a cure for my daughter, and I don't care exactly why it works, so long as it does work.)  But then I realized the implications of this change in theory.

BCG has finished a phase-I clinical trial.  At this point, most drugs would have two reasons to think they might be successful: the results of their phase-I trial and the results of the previous animal experiments.   That means that even if the phase-I trial was unsuccessful, the researcher could still rely on the animal trials for motivation, and try another human trial to capitalize on whatever good results were seen in the animal studies.   This is particularly important for BCG because the phase-I trial did not lead to successful C-peptide numbers.

However, Dr. Faustman is now saying that the TNF theory did cause the good results in mice, but is not causing the good results in people.  So therefore, it is hard to go back to her animal research to get support for her current human research.  And the human research itself is not yielding good results [r21].

Why The Hype?

A big part of the reason this study is important, is because of the buzz it has generated.  Therefore, understanding where that buzz comes from is important.  In my opinion, the results from the paper don't merit much excitement.  The hype comes from the news coverage of the press release, and I think it is always a mistake to react to hype in press releases when the underlying paper does not generate the same level of excitement.  This is a general problem in medical research and I'd recommend reading the articles listed here [r14].  Those articles cover the problem from several different points of view.

 The press release starts out with this sentence:
"Long-term follow-up of participants in clinical trials of a generic vaccine to reverse advanced type 1 diabetes finds significant clinical benefits, including restoration of near-normal blood sugar levels." [r3]
Consider the word "reversal", which is often interpreted to mean "cure".  (Compare "drug X reverses disease Y to "drug X cures disease Y".  Same meaning.)  Reversal is also in that first sentence to refer to results in people.   However, in the body of the paper, different forms of reversal are used 5 times (3 times for mice, 1 time to say the results did not include reversal, and 1 time for speculation about reversal).  Never in the paper was the word "reversal" used to describe the results in people, yet it was used in exactly that way in the first sentence of the press release.

Also, the press release uses the term "near-normal blood sugar levels" repeatedly.  Many newspapers interpreted this to be near-cure, and wrote their headlines accordingly.  But let me ask you a simple question:  If someone has type-1 diabetes and uses a lot of technology and generally works hard at treating their type-1, and has an A1c in the mid or low 6s, would you describe that as "near-normal blood sugar levels"?  Maybe.  But that says nothing about if they are close to a cure for type-1 diabetes.  Saying "near-normal" generates a lot of hype, but a cure is based on not needing to constantly treat your type-1 diabetes.  And the study is clear:  No one treated their type-1 diabetes any less because of the BCG: not fewer blood checks, not less insulin.  Everyone continued their standard care: dosing for what they ate, counting carbs, and anything else that we would associate with type-1 diabetes [d8].

The subtitle of the press release is:
"Mass. General study finds novel mechanism underlying stable, durable blood sugar control" [r3]
Now take a look at the previous graph of A1c numbers.  Does that look stable to you?  Does it look durable?  Not to me.  Quite the opposite, the good results are completely dependent on when you look at the data.  Two years after treatment the numbers are bad.  Between four and six years they are good.  At the end of the study, they are heading back towards where they started.  This is neither stable or durable.

The press release gives specific A1c data for 3 years and 4 years, and the average for the four year period from 3-7 years.  The 4 year numbers are the best found in the study, the 3 year second best, and the 3-7 year time frame the "good years" of the study.  However, the 1st and 2nd years (when results are bad) are not mentioned, and the average presented in the press release specifically excludes those years. It's like calculating a child's GPA but excluding their worst grades.  Of course it looks good, but it doesn't represent their real level of accomplishment.

Where BCG Research Goes From Here

One answer to this question is simple: a phase-II study is already underway, so we just wait until 2023 for those results to be published.  For BCG to be successful as a cure, it needs a specific kind of good news from the phase-II study: C-peptide data which is both statistically significant and clinically significant, and which comes from a large group of people with a good control group.  As a treatment (something taken in addition to insulin) then A1c data is enough.  It would still need to be statistically significant and clinically significant, and come from a large group of people with a good control group.  But all of that is possible from the phase-II trial.

Another answer is this: At this point, both publications from the phase-I trial were unsuccessful.  While an unsuccessful phase-I trial usually ends the line of research, this is not always true.  My guess is that about 20% of the current current phase-II trials are occurring after an unsuccessful phase-I result.  So there is always some hope.

But the real question is, how optimistic should we be about this line of research?  In my opinion, not very optimistic.  Above, I've described why the C-peptide and A1c data in this specific paper don't give me much hope for success in the future.  However, when I look at the (roughly) 15 year history of BCG trials in people, and put this paper into the context of the previous BCG research, I see a couple of additional red flags:

First is the lack of forward progress, given 15+ years / 34 million dollars [r20].  Fifteen years is enough time to get from the start of a phase-I to end of phase-III trials, and $34 million is more money than most academic researchers can spend on one line of research.  But for all that, I don't see any forward progress.  In 2003, we had no data on BCG's curative effect on people.  Now, we still don't have any positive C-peptide data to answer that question.  The hope is that the phase-II trial will answer it in 2023 or so.

Second, is the changing target of the research.  Successful research tends to have one target ("primary end point"), and gathers more evidence and stronger evidence on that target over time.  That is the progress that researchers expect.  However this line of research has changed its target repeatedly.  When the phase-I study started, the primary end point was autoreactive t-cells [r6].  When the phase-I study ended, the headline data was C-peptides [r17], and now this extension headlines A1c data [r2].

Even worse, this paper conflicts with the previous paper, even though they are both based on data from the phase-I trial. The initial phase-I paper showed: small, good results for C-peptide, no good results for A1c data, and support for the TNF hypothesis.  This paper shows no good results for C-peptide, mild, good results for A1c data, and support for a sugar metabolism hypothesis, but not a TNF hypothesis.  Good science builds on itself: the first results might be small, but the next results are stronger.  But here, the next results are not bigger, they are different.  That is not the normal course of scientific progress.

Personal Note

Many people, with wildly different viewpoints, reviewed this blog posting.  I want to thank everyone who spent time on it.  It needed a lot of work, and benefited from every reviewer's feedback.   All mistakes are my own.

Extra Discussion [d-Numbered] Footnotes

[d1] Consider a simple example: injecting insulin.  If you inject more insulin your A1c will go down, but your C-peptide numbers will not change.  A researcher who is treating A1c as progress toward a cure will see injecting more insulin as progress towards a cure.  That is why researchers measure C-peptide to evaluate progress to a cure.

Or consider this: "in the 1990s, the FDA began to approve drugs for the treatment of diabetes based upon hemoglobin A1c (HbA1c) as the outcome. The prevailing belief was that risk reduction could be achieved by a clinical focus on reaching target values of HbA1c" [r18]

[d2]  For instance this slide [r20] presents data from four different groups treated with different type-2 diabetes medicines (two medicines at two doses).  All four of these groups dropped the same or more as is seen here.  And [r21] shows that simply using CGMs can lower A1c numbers in pregnant women about the same as seen here.  And [r22] shows that a new class of drugs (approved in type-2s and being tested in type-1s) called SGLTs lower A1cs about as much as seen here.  And the list goes on.

The bottom line is that the average improvements seen in this study are similar to the average improvements seen in many other treatments, which are much closer to FDA approval (or already have EMA approval) for type-1 diabetes. 

When we look at A1c for people who got BCG over the life of the study, it averages about 6.6 which is about 0.8 below the 7.4 where it started.  (The result is a little worse (about 0.5) if we compare it to the control group, which started at about 7.1).

For comparison, all of these treatments have gotten results similar to the 0.8 improvement seen here in either type-1 diabetes, type-2 diabetes, or both:
    Semaglutide: A1c improvement of 1.5 [r20]
    Delaglutide: A1c improvement of 1.2 [r20]
    CGM use during pregnacy: A1c improvement of 0.6 [r21]
    SGLT2 inhibitors: A1c improvements of 0.5 to 0.8 [r22]

[d3] As an example, When was the last time your doctor said "If you do X, Y, or Z you will have better A1c numbers next time?  We get it all the time.  But when was the last time your doctor said "If you do X, Y, or Z your c-peptide number will be better"?  Never.  This shows both that A1c is a measure of treatment, and why it is not a good measure of a cure.

[d4]  For example the following quote is from the D-Cure workshop of international experts held in Barcelona in April 2007 [r9]:
"It is now an accepted approach to evaluate endogenous insulin secretion by measuring C-peptide levels (with highly sensitive and normalized measurement methods) in response to a physiologic stimulus (liquid mixed-meal) under standardized conditions."
This report goes on to specifically to consider and reject A1c as a measure of a cure:
"differences in HbA1c between treatment and placebo groups are minimal and thus cannot serve as robust measures of efficacy"
[d5] I reviewed all phase-III clinical trials aimed at curing type-1 diabetes in the last 15 years.  These are in the final stage of clinical trials, which lead (if successful) to FDA approval.  There have been 11 by my count, testing a total of 4 different treatments.   Of these, 6 used C-peptide as their sole primary outcome.  None used A1c alone as their sole primary outcome.  One used it as part of a dual primary outcome. Four used other measures as their primary outcome.

[d6]  It was not supposed to be that small.  The phase-I trial was supposed to have 12 treated people and 12 control [r6].  However, for reasons never published, the original trial only gave 3 people BCG.

[d7] Finally, these A1c results represent what is commonly called "results switching" in clinical trials, and this is very dangerous in reporting results.  Results switching is when the researcher says they are going to report one result (and designs the trial to do this), but then ends up reporting on a different result.  One of the reasons the FDA has a public clinical trial registry [r19] is specifically so that researchers need to publicly announce what their end points are ahead of time.  This prevents them from selecting end points to create success after the data is gathered.

Obviously, it is bad when secondary results are switched, and worse when a secondary result is used to replace an unsuccessful primary result.  However, in this study we see the worse form of "results switch" where an unsuccessful primary end point (autoreactive t-cells) is not reported, and replaced with a better, but still mediocre, result (A1c) which was previously not part of the study at all!

There are several articles on results swapping listed here [r16].

[d8] Compare Dr. Faustman's use of language in the press release with Dr. Bernstein's use of language.  Dr. Bernstein uses a low carb diet and aggressive insulin dosing to achieve A1c numbers lower than those reported in Dr. Faustman's research.  His target A1c is 4.5, much lower than Dr. Faustman achieved here.  However, Dr. Bernstein never refers to curing type-1 diabetes.  He is very clear that the low A1c numbers he aims for might be the same as someone without type-1 diabetes, but that is in no way a cure.

To put it bluntly: if you think Dr. Faustman's A1c numbers in the mid-6s represents a near cure, then you would have to agree that Dr. Bernstein's A1c numbers in the mid-4s would represent an actual cure, but no one does that.

[d9] The essence of Dr. Faustman's older theory on how to cure type-1 diabetes is:
  • BCG causes the body to generate TNF
  • TNF causes fewer autoreactive T-cells
  • Fewer autoreactive T-cells results in natural beta cell regrowth and more insulin generation
  • More insulin generation is the path to curing type-1
BCG (Bacillus Calmette–Guérin) is a biologic that has been given to over a billion people (in low dose) as a tuberculosis vaccine, and is also approved (in much higher doses) as a bladder cancer treatment. It is a generic drug with a very long record of safety.

TNF ("Tumor necrosis factor" or TNF-alpha) is a naturally occurring protein that can cause cells to die. It is involved in the natural regulation of immune cells.

"Autoreactive" refers to immune cells that mistakenly attack the body's own beta cells. The destruction of these beta cells leads to type-1 diabetes. This is sometimes referred to as an "autoimmune attack" because the body's own immune system attacks the body itself.

Many more details are available here [r10].

[d10] For A1c improvements, my standard (which I think is pretty common among both researchers and the FDA) is that changes below 0.5 are not of importance, changes above 1.0 are definitely important, and numbers between these are of mild importance.  So the BCG results (if supported by larger trials) would be in the mild interest area.

[d11]  Here are three supporting quotes:
Clinical studies aiming at preservation of beta cell function should be randomized, preferably double-blind and placebo-controlled and should include patients with a documented residual beta cell function. The primary outcome should preferably consist of co-primary endpoints including not only the change from baseline in C-peptide (e.g. C-peptide AUC) or, if appropriately justified, the percentage of patients with C-peptide increases above a clinically meaningful threshold following a physiological stimulus (e.g. liquid mixed meal) under standardized conditions but also HbA1c, frequency of hypoglycaemic episodes, particularly severe events, or the percentage of patients not requiring insulin therapy or with a relevant reduction in insulin requirements. Any of these endpoints not included as co-primary endpoint should be evaluated as important secondary endpoint. [r22]
FDA and EMA stand ready to approve disease modifying therapies for T1D. and have expressed reasonable expectations for demonstrating efficacy of therapies aimed at preserving insulin secretion in new onset patients. It is unclear what minimum treatment effect on preservation of C-peptide secretion, the regulatory primary efficacy endpoint, would be considered clinically meaningful for a new onset intervention. A small effect size (10-20% at two years) might be enough if the safety profile is very benign. [r23]
And finally, compare these two quotes from [r24]:
Efficacy endpoints. Stimulated C-peptide response is accepted as the regulatory primary endpoint because it is a direct measure of reducing the hormonal deficiency state of T1DM.
Secondary endpoints and their considerations. HbA1c is the gold standard measure of glycemic control, but it is an insensitive measure of improved beta cell function resulting from an intervention. 

Reference [r-Numbered] Footnotes

[r1] The Press Release:
and the Faustman lab FAQ is here:

[r2] The Paper:  and this includes supplementary data here:

[r3] Abstract of The Poster:
This tweet contains the poster and a little discussion:

[r4] Medical Press Coverage:

[r5] ADA/JDRF Response:
and coverage:

[r6] The FDA clinical trial record for the phase-I study:

[r7] The FDA clinical trial record for the phase-II study:

[r8] More than you ever wanted to know about how the FDA evaluates clinical trial end points:

[r9] This report summarizes the conclusions of the D-Cure workshop of international experts held in Barcelona in April 2007 and the current recommendations and updates in the field:

[r10] Dr. Faustman's most recent paper describing TNF as the mechanism of a cure was published in 2017:
And she had edited an entire book on the subject in 2014:
And she had several earlier publications on the same (now defunct) theory:



[r16] These are general references for results switching:
And this organization:
This abstract is an interesting read as well:

[r17] Initial results of the BCG phase-I clinical trial:

[r18] This paper argues that A1c is the right end point to measure treatments for type-2 diabetes:

[r19] This is a link to the US FDA's Clinical Trial Registry Site, plus an article about it:

[r20] Dr. Faustman's Lab has raised about $11 million for their phase-I trial and about $23 million for their phase-II trial:

[r21] The two key quotes from her paper are:
As previously published, the elevations in tumor necrosis factor (TNF) from the BCG vaccine stimulate cytotoxic T cell death and beneficial Treg expansion [in live mice and isolated human tissue]
and then:
The mechanism for lowered HbA1c values [in people] was not equivalent to the NOD diabetic mouse pancreas regeneration after BCG treatment



Joshua Levy 
publicjoshualevy at gmail dot com 
All the views expressed here are those of Joshua Levy, and nothing here is official JDRF, JDCA, or Bigfoot Biomedical news, views, policies or opinions. In my day job, I work in software for Bigfoot Biomedical. My daughter has type-1 diabetes and participates in clinical trials, which might be discussed here. My blog contains a more complete non-conflict of interest statement. Thanks to everyone who helps with the blog.