Friday, July 12, 2013

Should the DoD Buy Epic, or Cerner, or GE, or…?

The Department of Defense (DoD) is in the market for an EHR solution… again. After a lengthy foray into building its own EHR from scratch (AHLTA), with miserable results, and another shorter detour through the fantasy land of an open-source integrated EHR (iEHR) with the Veteran Administration (VA), during which no material progress was made, other than spending taxpayers’ money of course, the DoD announced that it will begin looking for a commercially available product to suit the DoD’s unique needs. This decision is the source of much angst for some and much excitement for others, because no matter what the DoD decides to do, many more billions will be flowing out of taxpayers coffers and into the hands of a lucky few.

On one hand, since most people served by a DoD EHR will eventually be served by a VA EHR, it makes sense that these two government agencies should use the same product and aggregate a lifelong record for their patients. On the other hand, the VA EHR (VistA), although held in high esteem by its creators and users, is very old, and the VA itself is engaged in a major refurbishing effort through a public open-source framework. Answering to a highly frustrated House Committee on Veterans’ Affairs, looking for explanations and bemoaning the death of the iEHR, the DoD’s Frank Kendall reiterated the insurmountable costs and difficulties inherent in building a brand new EHR from scratch (better late than never, I guess) and highlighted the reasons why VistA is not as obvious a choice for the DoD as it is for the VA. Since the VA has VistA already deployed in all its facilities and it already employs an army of experienced VistA developers, a salvage operation for the aging VistA may make perfect sense for the VA, but none at all for the DoD which will be conducting a complete rip and replace program in the next couple of years.

Now that the self-developed iEHR delusion is dead, the technical controversy is focused on whether the DoD should select an open-source product, either VistA or a derivative thereof, or a proprietary system, such as Epic, Cerner, GE, Siemens, etc. The main arguments against a commercial solution are that the DoD should leverage the billions of dollars already invested in VistA, and that it should adhere to this administration’s preference for transparency and open-source technology. To better understand this conundrum, perhaps some terminology clarifications are in order.

Computer software comes in several flavors, and you may recognize most of these flavors if you researched your own EHR purchase recently:
  • Proprietary – Commercially Licensed Software - Licensed commercial proprietary software is software you purchase as a black box. You can install it on your computer and use it as is for as long as you want. This is a license to use the software, and implies no ownership rights for the product. The vendor will usually charge you for upgrades. Most often, particularly for business software, you can supplement this one-time acquisition with a maintenance and support subscription fee that will provide you with upgrades and assistance from the vendor. Think Microsoft Office.
  • Proprietary – Commercially Licensed Software + Source Code - Commercial proprietary software may include the source code (for additional fees) and rights to extend the original product for private use. Sometimes instead of the entire source code, or in addition to it, selected integration points (Application Programming Interfaces - APIs) are exposed to clients so they can add/customize certain functionalities on their own. Rights to enhance and augment the original source code for commercial resale are usually corporate arrangements not really intended for mere users of software.
  • Proprietary – Subscription Based Software - A more modern variation on the above is software-as-a-service (still commercial and proprietary), where you don’t purchase a license for the product, but instead pay monthly fees for using the software, which is maintained and operated by your vendor. These fees are usually a bit higher than the maintenance and support fees for licensed software, mainly because the vendor has to bear the infrastructure costs in this scenario. This goes by the name of “cloud”. Although the vendor may expose APIs for third party developers, this type of arrangement almost never comes with source code, since the buyer is not running the software. Think Microsoft Office 365.
  • Proprietary – Free Software - Free software is software you can obtain without paying either a license fee or maintenance fees. Free software may refer to software that you can download and use for free, or more often, free services provided through software residing in the developer’s datacenters (“cloud”). The larger the provider and the more individually geared the software is, the more likely it is that the provider will expose APIs to third party developers. Think Google anything.
  • Open Source – Free Software + Source Code – This is what comes to most people’s minds when they say open-source. This type of software is usually developed and managed in a public framework by an all-volunteer squad of techies who have an irresistible urge to donate their labor to better humanity. There are mountains of open-source projects, some highly professional, others not much more than a hobbyist pastime, and there are almost none in wide consumer use, other than a handful of exceptional non-profit foundations (e.g. the Mozilla Foundation and its Firefox browser or the Apache Foundation and its development tools and modules). Open-source pieces of software are often utilized by commercial software companies to speed up their internal development process. For example, IBM’s Watson includes various open source modules, but Watson itself is neither free nor open source.
  • Open Source – Maintenance Subscription Services + Source Code - A slightly different business model is an arrangement by which the software itself is provided for free, including source code, but maintenance, improvements, installation and training are billed on an ongoing basis. Sometimes special requests for features or premium functionality are also available for purchase. The software products available through this model are either developed in house by the service vendor, or obtained from a conventional community developed open-source project and enhanced by the service vendor. This model allows clients to customize the software on their own, if they wish to do so, but it may void or complicate maintenance agreements with the vendor.
Regardless of the model employed to obtain software, extensive APIs may or may not be available to third party developers who may be actual users, but most often are not. Software products that expose a good amount of well documented APIs are said to have an open architecture, which is based on accepted and open standards. APIs come in various flavors as well. Some are just plug-ins allowing third party developers to add minimal functionality that does not affect the main product in any way, and certainly does not affect the data layer. Tighter integration that may affect data integrity in the main product is harder to come by and usually requires approval from the original vendor if the software is proprietary. If you have access to the source code, either free or purchased, you can obviously do whatever you want, if you are reasonably certain that you can maintain the software on your own going forward. People or businesses without an IT department are rarely interested in tinkering with source code, so this conversation is really between IT shops, large businesses and true believers in this or that paradigm for software development.

Before returning to the DoD situation, one more word about vendor lock-in is in order. There is no doubt that having the source code (open source or purchased source), and particularly the database schema, makes transferring your data from one product to another a lot easier, if and only if, you have proper IT resources at your disposal. It is possible though, and most often very likely, that if your old product and your new product are very different, data loss will occur no matter how good your IT guys are and no matter how open both products are. Also, it is usually not necessary to have access to every line of code in order to migrate data from one system to another. A clean open architecture based on widely accepted standards is much more important for preventing vendor lock-in and for interoperability in general. Unfortunately none of the choices available to the DoD fit this bill currently, although some may be better than others. So what are the options available to the DoD?
  1. Adopt the VA VistA – The DoD rejected this option, but it may be helpful to review it just in case they change their mind again. The DoD could of course obtain the entire VA source code for free, and could also benefit from the VA meritocracy-based framework of open-source freebies donated by selfless defense contractors and eventually by some guy in the Ukraine that is inclined to help the U.S. military save money on software development.
  2. Obtain a different open-source product – There are none. The only viable open-source EHR option for large systems, other than the VA VistA is the privately enhanced and maintained VistA offered by Medsphere, which is also contributing enhancements to the VA open source framework. The other VistA derivative is WorldVistA.
  3. Buy a commercial EHR – None of those are of course open-source, although if you buy Epic for example, you get the source code, and I would assume that the same is available from all others. A commercial EHR will most likely still need to be tweaked and enhanced for military purposes, whether by the vendor or by the DoD, but it should come out of the box with a lot more bells and whistles than VistA, some useful and others not so much, depending on which one is selected.
Basically, no matter what the DoD chooses to do, they will have access to the source code, and significant programming effort will be required to tailor the product to DoD needs. Most products that could be considered by the DoD have some useful APIs and increasing adherence to standards, but open architecture is not a term that comes to mind in this context. The DoD could hope and pray that the open-source framework established for VistA will produce the software improvements the military needs, or most likely, the DoD will have to pay for all enhancements and new features. In addition to software programming, deploying an EHR requires funds for training and implementation which can easily exceed the software costs by orders of magnitude, particularly since the military has very unique deployment locales.

We may believe that software purchased or built with taxpayers money should reside in the public domain, which almost never happens by the way, but this entire discussion is not really about open-source vs. proprietary products, which is largely irrelevant for the DoD itself, but very much about the plum Pentagon contracts to oversee this mammoth change to its medical records system. The choices are: internally developed DoD resources, a ragtag team of VistA veterans, a newer service entity like Medsphere, a commercial EHR vendor like Epic, or an entrenched usual suspect like SAIC, and most likely a combination of a defense contractor with any one of the lottery winners above. No doubt that as is always the case, the Pentagon will be making a wise, frugal decision, free of biases and bereft of the undue influence of special interests.

Correction 7/14/2013: The original text mistakenly and regrettably stated that WorldVistA is not in active development based on this WorldVistA page. According to the WorldVistA home page, the EHR is very much active and Meaningful Use certified. My sincere apologies to the WorldVistA folks.


  1. Is no surprise DoD is looking into a commercial product. That has always been their hidden agenda. And how is it not possible, Epic's owner seats at some of the government HIT committees. Talk about conflict of interest.

  2. It's easy to tell you spent a good bit of time putting this together...well done. Since this is an opinion piece, here's my two cents worth...because there are a couple of comments maybe we should all think about...
    1. The iEHR was never really an effort to work with VistA if you take the gap between form and function into was an inside attempt to get the inconvenience of a working eHR (VistA) out of the IT acquisition path. Agreed on the fantasy detour.
    2. It's interesting that EPIC is recognized as a potential answer but no one mentions it is the same old technology as EPIC...and by your apparent definition SQL tech would be old as well. From what I have seen, GE and Sieman's are also based in the same technology. I can't speak to Cerner.
    3. So your comment about a “salvage operation” exposes a fundamental poverty of information on the well as pejorative intent. When the VA exposed the source code of VistA as public domain...a large percentage of the eHR companies out there simply stood up an instance, put their name on it and started selling their “brand”. It's the American way...apparently.
    4. Accordingly, your flavors discussion is a fairly accurate description of the “American way” trivialising the skepticism these untested approaches clearly deserve. If this was an easy task it would have been solved. It is a dis-service to infer otherwise.
    5. Your seem to believe the battle ground is not about the source code...I beg to differ. I don't have space or inclination to debate this fundamental concept...except to say it is ALL about the source code. We have to accept IT departments as a fact of life...oursourcing mission critical capability is a VERY short-sighted plan.
    6. The DoD has not rejected the option of adopting VistA...this is fiction. Over the last week the House Armed Services Committee and House Veteran's Affairs Committee have jointly met to hear testimony...I refer you those meetings. DoD expects VistA participation...
    7. It is also clear you don't buy in to the open source concept...but using “hope and pray” is nothing less than mis-characterization of the concept...which is clearly targeted at the uninformed...what is your motive?
    8. WorldVistA is in active development at all times...I can't decide if you know the truth and are trying to hide it or if you don;t know what you are taking about...either way it is repugnant. There are several other VistA variants AND for that matter convergence between standard coded variants is close to your point is moot...except for the motive.
    9. Could you please explain why an intellectual property 100% built using 100% public funds should NOT be public property...this concept amounts illegal conversion of public funds...IMHO.
    Finally, my global response to this somewhat preposterous effort leads me to believe you speak in ignorance except for the pejorative tone and question of motive.

    I just think readers need to know this reads like an infomercial for an embedded interest. I have many problems with much more than I stopped to mention.

    Look elsewhere for authoritative, accurate and/or unbiased information.

    1. Hello Crunchy,
      First let me say that I have no dog in this fight, other than my taxes, which I would very much like to see put to good use. So hopefully in order, but not necessarily:

      1) All large EHR products are old technology, and VistA and Epic are probably more similar than any other two. I would say that large software is always going to be old because it takes time to grow large. So I am not too terribly disturbed by the "old" description.

      2) Salvage is not pejorative. It is what you do to keep old stuff alive and updated. The alternative is to accept that every software reaches a natural end of life, and go buy/build something new. Unfortunately, there is nothing much newer and building will take a decade.

      3) I don't see how the battle ground is or can be about the source code. Unlike the VA, the DoD is not deciding between built in house or purchased. Both alternatives are going to be purchased for the DoD. One may be significantly cheaper, but both are outsourced. Certainly, from this point onwards the DoD can bring development in house for either one (see the Epic deployment at Kaiser). I actually agree with you that IT departments should be kept in house for large organizations and had this exact conversation the other day on G+.

      4) The DoD according to Mr. Kendall, has received 3 out of 15 proposals that are VistA based, one from the VA itself, so I guess the DoD just wants to compare its options? If they wanted VistA they could have saved us all tons of wasted money going back many years....
      As to World VistA, the last release I am able to find is from 2008 (CCHIT certified). Is this considered active development?

      5) I love open source (I come from the Java world). But open-source is in my opinion for parts, not for complete products. You get some parsers, some loggers, some this and some that, and build your product. I don't know of anything worthwhile for a large enterprise that you download from sourceforge and call it a day. Maybe if the DoD was actually building an EHR in house, shopping for tried and true open source utilities would be great.

      6) That line about using public fund to build secret software was sarcastic. Of course it should be public, but is it? What publicly funded software is open to the public? PRISM? This is the Pentagon we are discussing... A certain level of realism is I think in order...

      Thank you for your thoughtful comments.

    2. Margalit,

      "As to WorldVistA, the last release I am able to find is from 2008 (CCHIT certified). Is this considered active development?"

      Goodness you didn't look in the most obvious places to find the latest version... the WorldVistA homepage and the software download page. Both pages clearly show that WorldVistA V2.0 was recently released and passed full certification for both inpatient and ambulatory settings... the only non-VA VistA version to accomplish this.

      If you were part of the VistA community you would be aware of the many active projects that WorldVistA's team and extensive network of collaborators are involved in which produce these updates. For example: moving the MOCHA drug database web service to an open source platform; porting, with the help of Jordan's EHS, the EDIS emergency department package to open source, and working to meet MU2 requirements. WorldVistA has 4 to 5 development related conference calls a week related to these efforts and is one of the most active contributors the various activities at OSEHRA. In other words WorldVistA EHR development done primarily via a distributed, collaborative community model. This is how VistA was developed at the VA and is the most robust and sustainable business ecosystem for open source in healthcare. Unfortunately this is something the DoD doesn't seem to get, yet... while other big health systems do.

      I recommend you do your homework before blogging if you want to be taken seriously. If you aren't certain about a fact email those who can answer it, otherwise, to quote Crunchy your blog reads like “infomercial for an embedded interest”.

      Joseph Dal Molin
      Chairman, WorldVistA

    3. My apologies Joseph, and thanks for the correction. The post has been updated.

      I would suggest that you guys update the "WorldVistA EHR" page on your website, which seems to be stuck at the 2008 CCHIT certified version, as well as the News page.