Dr. Faustman's Phase-I Results

For over 10 years, Dr. Faustman has been researching the theory that BCG (Bacillus Calmette-Guérin, used as vaccine for tuberculosis) could be an important part of a cure for type-1 diabetes.  She published animal research to support this theory in 2001 and 2002, research in human cells in 2008, and started a human trial in 2008. In late June, as part of the ADA convention, she published two abstracts, which are her first results in human trials. 

You can read the abstracts here:
http://ww2.aievolution.com/ada1101/index.cfm?do=abs.viewAbs&abs=12326
http://ww2.aievolution.com/ada1101/index.cfm?do=abs.viewAbs&abs=12141
This posting is more about the first one.  The second one is interesting enough for it's own posting, but that will have to wait until another blog.

Here is a supplemental FAQ on the trial from Dr. Faustman's lab:
http://www.faustmanlab.org/docs/clinicalt/Phase%20I%20Trial%20FAQ.pdf

Here is the original clinical trial paperwork (for comparison) from the FDA:
http://www.clinicaltrials.gov/ct2/show/NCT00607230

Some news coverage (there is a lot more):
http://online.wsj.com/article/SB10001424052702304231204576406011708905314.html?mod=googlenews_wsj
(note the WSJ reporter describes 6 treated patients, when the abstract is clear that there were only 3)
http://articles.latimes.com/2011/jun/25/health/la-he-bcg-diabetes-20110625
 
Scott King's blog on this research. (He is more optimistic than I am):
http://www.solvingdiabetes.org/2011/07/04/highlights-of-ada-scientific-sessions-i-bgc-trial/

Everyone wants a one sentence summary of results.  Unfortunately, this research as described in the abstracts, doesn't have a good one sentence summary.  But I've provided three to choose from:

1. An optimistic summary: A low dose of BCG shows some effect on people with long established type-1 diabetes, and that is so unusual that it is a good result, worthy of follow up.
2. A neutral summary: A tiny study was done, but the information in the abstract is so limited that it can not be evaluated, so we must wait for the paper and the additional information it contains.
3. A pessimistic summary: The data that was supposed to be the primary results of the trial were not included in the abstract.  The results that were reported are so small and so vague that nothing good can be concluded from them.  More worrisome, the study reported in many ways is a step backwards from the study that was discussed at it's start. 

Here is my attempt of a one paragraph summary:

Dr. Faustman's research theory is that raised TNF levels would cure or help to cure type-1 diabetes. No data presented in the abstract supports that idea, because no TNF data was presented. Dr. Faustman's clinical theory is that giving people BCG would cure or help to cure type-1 diabetes (by raising TNF levels). The very tiny amount of data presented here is unclear because the effect was seen in 2 of 3 treated patients, but also in 1 of 3 untreated patients (who came down with "mono"), and at the tiny numbers of patients involved, it's hard to interpret those results. No data on the size or duration of the effect was presented in the abstract. The study reported now is much smaller than the study described at it's start (in FDA paperwork). An interesting reaction was seen in one patient who came down with Epstein-Barr virus infectious mononucleosis (commonly known as "mono" or EBV). That data is the focus of a separate abstract.

Please read below for lots of details and specifics.  Marks like this: [d0] refer to extra discussion at the end of the posting.

Before Discussing the Study
Before even discussing the results of this study is is necessary to discuss two very important points.  In light of these two points, I think the best thing to do, might be to just wait for the paper.   (On the other hand, the rest of this post is my attempt to analyze the abstract, so I'm not really following my own advice.  :-)

First, I only have access to the abstract, a few short conversions with various people, a few short emails with Dr. Faustman, and a "FAQ" document that she sent me.  Abstracts are required to be very short (often only a few hundred words), so can not contain all the important details.  Also, Dr. Faustman doesn't have time to reply to every question that comes up (especially since a short abstract leaves many questions unanswered).  Even if she did have time to answer every question; all those answers together would basically be the paper, and the paper is not yet published.  So the bottom line, is that there are several critical unanswered questions, and we will all just need to wait for the paper.   I expect a major update to this posting when the related paper is published.

Second, because this study only involved giving 3 patients BCG, even with the extra information from a paper, the results are likely to be almost meaningless, because of the tiny size of the study.  In my opinion, only in extremely rare cases where the results are extraordinarily obvious will 3 treated patients yield a useful result.  (Obviously, Dr. Faustman has a different opinion, and that is discussed very briefly below.)

The Study and What Was Reported

Patients were randomly assigned into two groups: 3 people were dosed with BCG, and 3 people were given a placebo.  These were all people who had type-1 diabetes for a long time (average 15 years).  The treated group were given two small doses of BCG, one month apart.  They were followed for a total of 20 weeks.  In addition, there were also 6 "clinical controls" (who did not have type-1 diabetes), and two other patient "reference" groups: 58 people who had type-1 diabetes and 17 who did not  [d1]. These two reference groups were not given anything, but were monitored.  The patients were monitored for this data: autoreactive T-cells, regulatory T-cells, GAD autoantibodies, and C-peptide.

There were two results reported:
1. Transient increases in the number of circulating dead autoreactive T cells against insulin.  (The abstract did not say if these increases were statistically significant, or which group of patients they were compared to.  For this discussion I will assume the most positive possible interpretation: that the increase was statistically significant and that it was in comparison to the placebo group.)
2. C-peptide levels rose transiently above baseline levels. (In this case the abstract says specifically that the rise was statistically significant and in email Dr. Faustman said the comparison was in the same patient: before vs. after treatment and that it was true of two patients in the BCG group and also the patient who came down with EBV in the placebo group.)

Some terminology:
"Autoreactive" refers to T-cells which attack the body's own beta cells.  In other posts, I refer to these as "bad killer T-cells", because they attack the wrong cells causing type-1 diabetes.
"C-peptide" is a chemical that your body makes only when it also makes insulin. So measuring that is really measuring your body's ability to make it's own insulin.  Also, C-peptide is an FDA approved marker for progress towards a cure of type-1 diabetes.
"Statistically significant" means the change was unlikely to be due to chance or luck.  In general, only statistically significant findings are taken seriously by the scientific community.
"Clinically significant" means different enough so that the patient would notice or gain some benefit from the difference.  Phase-I trials are often not clinically significant; that often comes later.

Discussion
There are several large areas of discussion on this study:

First, Why was the study size so small?
Except in a few very rare situations (such as complete cures in all cases), a study that only treats three people is very unlikely to provide unambiguous results. When you are looking for slight improvements, or even medium improvements, three people just isn't enough to rule out random chance. All the people who work with statistics, who gave me opinions on this research, commented on this point. The abstract acknowledges the tiny size of the study, by calling it a "proof-in-principle" study. But no matter what it is called: it is the smallest phase-I clinical trial that I have ever seen. For comparison, phase-I trials of already approved drugs (similar to BCG) include: Anakinra dosed 15, Etanercept dosed 18, AAT dosed 15, etc. So a 3 person study really stands out as being much smaller than the others.

Why were there so few people in the study?  Remember: the published plan for the study at it's start was for dosing about 12 people, not 3.  There are three common reasons a study is small: 1. money, 2. recruiting, and 3. safety.  But this study was well financed with about 10 million US dollars (and using an inexpensive, unpatented drug, too), had many volunteers lining up, and there were never any safety issues.  So why not enroll 25 people, like they planned to?  What happened?

Obviously, Dr. Faustman does believe that 3 people is enough to lead to significant results, and felt that the small number of people was offset by the large number of blood tests given to each person.  The FAQ document and Dr. Faustman herself suggest that by doing a lot of tests on a very small number of people, they can attain significant results.  The FAQ document makes several mentions of the number of blood tests done as part of this study (1012), as evidence that the study was large enough.  However, standard practice is to measure studies by number of patients involved, not number of blood tests.  Especially in this trial, where it looks like 3 patients were given BCG, but approximately 84 (58+17+3+6) were not, that suggests that over 96% of the blood tests were run on people who didn't get the drug being tested.  So the large number of blood tests does not accurately reflect the size of the study.

Second, why was a new "reference group" added to the study?  And more generally, why was the study design so complex with so many different groups of patients?
The abstract refers to reference patients. These were patients that either had type-1 or did not, but they were not treated.  This group was not mentioned in the original description of the trial.  The second reported result (the C-peptide change) compared the treated group to this reference group.  The original design compared the treated group to a placebo group, and this is a common comparison to make.  So there are obvious questions about why it was added, and why not just compare the treated patients to the placebo patients. 

More generally, the study included 5 different groups (treated, placebo, comparison, reference with type-1 diabetes and reference without type-1 diabetes). That's a lot of different groups, and it is not clear to me why they were all needed.  It is also unusual. The common thing is to compare the treated group to the placebo group, and that was the original design of this study.

Finally, there are the issues with the size of the various groups.  If you start out with a 3 person group treated with BCG, then it makes perfect sense to have 3 or maybe even 6 people in the placebo group, (they had 3) but it's not clear from the abstract what the data from those patients was used for.  On the other hand, a 58 patient "reference" group seems unusually large.  There was a 17 person group that didn't have type-1 diabetes, in addition to a separate 6 person "comparison" group that also didn't have type-1?  I am looking forward to reading the paper to learn why all this was done, rather than do the simple thing.

Third, what were the size of the effects seen?
This abstract did not contain either the size of any effect, nor their durations, so I can not comment on how big the results were or how long they lasted.   We will all need to wait for the paper for that.  The size of the effects seen (especially the TNF, C-peptide, and autoreactive T-cell levels) are critical to understanding these results. [d2]

Fourth, why were TNF levels not reported in the abstract?
The essence of Dr. Faustman theory is this:

BCG causes the body to generate TNF and TNF causes fewer autoreactive T-cells

(and fewer autoreactive T-cells results in more insulin generation)

So the most important data in her experiment was the TNF levels as compared to the autoreactive T-cell levels. This was why the levels of autoreactive T-cells were designated as the primary outcome. However, in the published abstract, the TNF levels are not mentioned. This weakens the abstract in at least two ways. First, it is impossible to see correlation between TNF levels and autoreactive T-cells, which is exactly the confirmation Dr. Faustman is looking for. Showing this correlation would be direct evidence that Dr. Faustman's theory is correct. Second, it is impossible to see if the BCG and EBV patients had similar TNF profiles and therefore should have the similar results that were seen.

Fifth, what was the result of the primary outcome measure? Why were dead T-cells reported on, but not live ones?
The first goal of a trial is to report it's primary outcome measure. For this study, the researchers clearly specified "concentration of autoreactive t-cells" as the primary outcome in their FDA paperwork. The published abstract says that this data was gathered, but does not report it. Obviously, I hope that this data will be published in the paper or poster (and is statistically significant). But right now: nothing is nothing.

If this was a commercial company, the failure to report on the primary outcome would be a clear signal that the trial had failed.  End of story; end of discussion.

Levels of dead autoreactive T-cells are listed as being statistically significant, and this is an interesting finding, but it is not a replacement for data on live autoreactive T-cells.  It is the live ones that matter; they cause type-1 diabetes.  The implication from the dead T-cell data is that the bad cells are being killed off.  If true, this is great news. But dead T-cells are an indirect measurement, while reporting a drop in live T-cells would be a direct measurement, and it was what the original design described.

Sixth, discussions of dosing.
This study apparently used the low (vaccine dose) of BCG, and Dr. Faustman is talking about using larger doses to get better results in the future.  And I agree that if small doses lead to small results, then it makes sense to try larger doses.  So we'll all need to wait for the paper what the current results are.  Remember: the abstract had no results numbers.

Seventh, the difference between a Phased Clinical Trial and a Case Report.
Dr. Faustman has two abstracts in ADA (links above), and it is important to remember the differences between them, and not mix them up or combine them. The first is a report on a Clinical Trial. Clinical trials are structured programs where people are given treatments and the results are measured. The second abstract is a Case Report, which is a report from a doctor on an unusual occurrence in a patient. In this case, it is an unusual occurrence in someone enrolled in a clinical trial. Phased trials are part of a path to find out if a treatment works and get it approved. Case reports are basic research which can start a line of research. Phased trials are at the end of a research program, case reports usually at the start. The difference is often 5 or 10 years of work.

Eight, why is this study so much different (and in many ways worse) than the study first registered and started 3 years ago?
The study (as completed and reported) is very different than the study (as registered on the FDA's clinical trials site).  In particular:

• The reported study was much smaller (6 vs. 25 people).
• The reported study included two groups of patients which were not included in the original description at all, and these new groups were roughly 12 times bigger than the placebo/treated groups described in the original filing (75 vs. 6).
• TNF was one of the secondary outcomes of the planned trial, but was not reported in the abstract. 

These are all steps backwards in trial design and implementation.  None of them had to happen, but they did.  The smaller size in particular leads to many of the uncertainties in the current results.

Ninth, what about the money?
I don't want to talk too much about money.  After all, a cure (or even much improved treatment) would be worth every penny no matter what the cost.  The funding goal for this phase-I study was US$ 10 million.  That would make this the most expensive phase-I clinical trial aimed at curing type-1 diabetes that I have ever encountered.  At the same time, with 3 dosed patients, it is also the absolute smallest phase-I trial that I have ever encountered. Even if you divide it by the 1012 blood tests, that's almost $10,000 per test.  

Rumors

The following information are rumors which I picked up from people who were at ADA in San Diego or associated with Dr. Faustman's lab. None of them are in the published abstracts, so I don't know if they are accurate or not:

1. The effects described as "temporary" in the abstracts actually lasted two weeks.
2. The effects that were described as statistically significant, were not clinically significant. 
3. A paper is in process to be published in a well known and respected scientific journal. (I have not gotten an OK to include the journal's name, so I'm not).  
4. TNF data was collected as part of the trial, even though it is not included in the abstract (and I asked specifically about the EBV patient).
5. There was a poster for the second abstract (the case study of EBV), but not for the first abstract (the more general BCG trial).  Or maybe this poster covered both abstracts, because there was plenty of BCG data on it?  (And I've tried to get a PDF of that poster.  No luck yet. If you were there, and got the thumb drive with all the posters on it, and this poster is there, please email it to me.  Thanks!)
6. Autoreactive T-cell data was on the poster.

Opinions (and some Ranting)

When I realized that this entire abstract was based on giving 3 people BCG, I was dumbfounded. In the final analysis, we -- the people who funded Dr. Faustman -- paid about 10 million US dollars, waited 3 years, and now we get results based on three people? Three people! It's completely shocking. Even more so when you realize that part of the reason BCG was chosen was because it was known to be both safe and inexpensive! 

For me, one of the most worrisome things about this clinical trial, is how much it has changed since it started years ago, and all for the worse.  It started out being a straight forward clinical trial, much like any other test of an already approved drug: about 12 people would get the drug, about 12 would not, results could be compared.  Somewhere along the way the study included 5 different groups.  And the size of the treatment group and the placebo group actually shrank!  (And not just a little bit: to one forth their original size).  The resulting trial has some size oddities that I've never seen before.  Two simple examples (which I've already touched on):

• First, they followed and ran tests on a total of 84 patients, and ran a lot of tests on each one. Yet only 3 people actually got BCG. That's a huge imbalance. Normally clinical trials are either 50/50 or 2/3s 1/3.  Meaning either half the people get the treatment and half don't, or 2/3s get the treatment and 1/3 don't.  Occasionally, you get designs with multiple dosing levels, and even there, the placebo group is the same size as each dose.  This study is wildly different. 

• Second, the study involved a total of 23 people who did not have type-1 diabetes at all. That group is about 4 times larger than the total number of type-1 diabetics in the combined treated  / placebo groups.  Again, that's very unusual. [d3] 

Fewer patients means less clarity, and that's exactly what happened here.  I'm very much looking forward to the published paper, and whatever discussion it includes on the reason for the small patient count, and the difference between the sample size as the study was registered and the sample size as the study was reported. 

The abstract did answer one question, which is why did we never hear of anyone who was part of the study?  Participants of other studies are often quoted in newspapers or they post on internet forums, but I never saw anyone who was actually part of the study in any public space.  And now it is clear why: with only 3 people getting BCG (and 3 people in the placebo group) this was a truly tiny study.

Previous Work

My previous blogging on Dr. Faustman's research is here:

• http://cureresearch4type1diabetes.blogspot.com/2008/10/faustmans-research-part-1-history.html
• http://cureresearch4type1diabetes.blogspot.com/search/label/Faustman

Many people who follow Dr. Faustman's work assume she is the only person doing BCG research, or the first person.  This is completely wrong.  In the late 1990s BCG was a topic of research as a possible cure for type-1 diabetes.  Three previous placebo controlled studies which gave similar BCG doses to type-1 diabetics showed no good effects.  To the best of my knowledge, the only difference in treatment between these studies and the c-peptide part of Dr. Faustman's study, is that these guys were given one dose of BCG, while Dr. Faustman's trial used two.  However (according to Dr. King's blog) Dr. Faustman used a much more sensitive C-peptide measurement than the older studies:

• http://care.diabetesjournals.org/cgi/content/ab stract/22/10/1703
  http://care.diabetesjournals.org/cgi/reprint/22/10/1703
  Published in 1999,  this study treated honeymoon diabetics with BCG and saw no improvement, compared with an untreated group. From the abstract: "Vaccination with BCG at the time of onset of type 1 diabetes does not increase the remission rate or preserve beta-cell".
• http://care.diabetesjournals.org/cgi/content/abstract/21/10/1691
  This study was published in 1998:  "BCG vaccination in children who have been recently diagnosed with IDDM does not affect the progressive decline in C-peptide levels or alter the clinical course of the disease." 
• http://210.101.116.102/Diabetes/koreamad/JournalSearch_index.asp?year=2000&page=340&vol=24&iss=3
  ftp://210.101.116.17/kiss8/27203283.pdf (in Korean)
  Published in 2000.  Here is the key sentence from their results section: "During follow-up, there was no significant difference in fasting and postprandial 2 hour C-peptides." They then go on to list various good things they did see, in particular there were slight differences in C-peptide and insulin usages, and temporary remission in two patients, but these were not statistically significant. (And remember: this study was of honeymoon diabetics.)  Their paper is in Korean, which I cannot read, however the tables are presented in English, and each table says very clearly "The differences between the two groups were not significant." 

In addition, three earlier studies compared type-1 rates in people given BCG as a TB vaccine to those who did not.  They showed no difference in rates of diabetes:

• http://cat.inist.fr/?aModele=afficheN&cpsidt=11111587
  In English the article is entitled "THE CUMULATIVE INCIDENCE OF CHILDHOOD DIABETES MELLITUS IN SWEDEN UNAFFECTED BY BCG-VACCINATION".  Published in 1995.
• http://care.diabetesjournals.org/cgi/content/abstract/20/5/767?ijkey=de477936685fbc95cfd98bc52cb120be655acd09&keytype2=tf_ipsecsha
  Retrospective study on people who had been vaccinated with BCG.  Published in 1997.  "However, as a whole, results from these analyses fail to support a protective role of BCG vaccination against juvenile-onset IDDM."
• http://care.diabetesjournals.org/cgi/content/full/28/5/1204
  Published in 2005.  Neonatal vaccination with BCG has no good effect on type-1 diabetes rates: "The cumulative risks for developing islet autoantibodies by age 2 or 5 years were unaffected by BCG vaccination in the first 3 months of life" and "Progression to type 1 diabetes in BCG-vaccinated autoantibody-positive children was significantly faster than in nonvaccinated children" and "No evidence was found that BCG vaccination could prevent against ß-cell–damaging processes leading to type 1 diabetes in genetically at-risk children. The findings do not suggest that BCG vaccination will affect the overall incidence of type 1 diabetes"

Where are We, Where are We Going, How do We Get There?

The first thing to do is to wait for the paper to be published.  

After that, one way to move forward is to change the dose.  And Dr. Faustman is already talking about doing that.  In a sense, this is the easiest change to make.  However, right now, there is no published evidence that it would help.  In order to see if this is worthwhile, we will need to see the data comparing TNF to C-peptide data (for all three groups: BCG, EBV, and placebo).  If higher TNF leads to higher C-peptide, then it is worthwhile to try higher BCG doses (and other methods, as suggested by EBV) to try to achieve higher TNF results.  Hopefully all this will be in the paper.

Another way is to focus on the single EBV patient.  This was a very interesting occurrence.  One of the patients in the placebo group came down with mono during the trial.  That patient apparently had high levels of dead bad killer T-cells and also elevated C-peptide counts.  (Although keep in mind the caveats above, especially the lack of actual numbers.) If that one patient had high TNF levels, and high levels of dead bad killer T-cells, and high levels of C-peptide (and that is three very large "ifs"!), and it is not just random luck with one patient (a fourth huge "if"!), and then it could mean that Dr. Faustman is right about TNF helping to cure or treat type-1 diabetes, and suggests that there are things besides BCG that can get us there.  So there might be some interesting research to be done there.  Of course, with one person, it might just be just some random weirdness.  

Some Personal Notes

I have put more hours into this posting than any previous blog entry.  One thing that I learned, is that I should not write blog postings based on abstracts alone, when those abstracts don't contain any results data. 

I know this will not be a popular posting.  However, in the final analysis: if a commercial company, indeed, if any other researcher, had published results that did not include their own primary outcome measurement, everyone would immediately agree that the clinical trial was a failure.  And if any other researcher had added (after design) a "reference" group with more than 12 times as many patients as the treated/placebo groups, then again, everyone would immediately clamor for an explanation.  And finally, if any other researcher had designed such a tiny experiment, they would be called out for an explanation.  Dr. Faustman did all of these things and so I treat her the same way that I would treat any other researcher.

Also, I have spent a lot of time comparing the clinical trial as described in the abstract at the end of the trial, to the clinical trial as described in the FDA paperwork which was submitted at the start of the trial. I know many supporters hate that whole attitude. They think that what Dr. Faustman said three or more years ago, is irrelevant to the results she reports today. They don't want past promises compared to current realities.   For me, that attitude ("what is said before doesn't matter to what is given now") is not merely wrong, it is dangerous.

For me, this comparison between what is promised and what is delivered is important information. If someone promises a piece of bread, but gives you a crumb, should you thank them for the crumb, or complain about not getting the bread? Does your opinion change if you were charged for the bread in advance? If the person is, even now, asking for even more money for future loaves? Does possession of a crumb now mean that the bread will be available in the future? Does not delivering the slice of bread now, mean that future loaves will not be delivered as well? The future is always uncertain, and research is all about the future.

I'd like to thank everyone who helped with this posting, either by giving me information, reviewing it, or motivating it by asking insightful questions. All mistakes here are my own, just as all opinions are mine.

I grant permission to anyone to republish this article anywhere they wish, as long as it is not edited (except to fix spelling and grammar).  If you wish to republish parts of it, rather than the whole thing, please email me.

Extra Discussion

[d1] I don't know exactly how the 6 "simultaneously studied clinical trial controls" (who according to the abstract did not have type-1 diabetes) were different from the 17 "reference patients" (who also did not have type-1 diabetes).

[d2] Dr. King's blog talks about C-peptide changes measured in pmol/l, which is a very small change, indeed.

[d3] On a few rare occasions I've seen studies that involved people who didn't have the disease being studied, but I've never seen a study with 4x as many people who didn't have the disease, as had the disease and were in the combined treated/placebo groups.  Or any other studies that mixed people with the disease and people without it as this study does.

Joshua Levy
All the views expressed here are those of Joshua Levy, and nothing here is official JDRF or JDCA news, views, policies or opinions. My blog contains a more complete non-conflict of interest statement. 
Blog: http://cureresearch4type1diabetes.blogspot.com
To Get as Email Join here: http://groups.google.com/group/type-1-diabetes-clinical-trials-news