Open Source: How the Management Paradigm Works. Open Source: How the Management Paradigm Works.

         Contents

     Chapter 1	The New IT Management Paradigm
	   1.0.1  Looking Ten Years Forward
	1.1  IT Urgencies
	   1.1.1  Shorcomings of present software business models
	1.2  Chaos and Paralysis: Proprietary Software in Health Care
	   1.2.1  Costs and fees
	1.3  Control
	     1.3.0.1  The database, as well as the code, is proprietary
	      1.3.0.2  Inaccessible code
	   1.3.1  Communication is layered
	   1.3.2  The Sun Sets Eventually
	   1.3.3  Upgrades occur on the Vendor's timetable
	1.4  Vendor-centric "community" development
	1.5  Efficiency and control
	1.6  Exploitation
	   1.6.1  The Exploitation Model
	      1.6.1.1  Reprise: Management Challenges to which We've Grown
		       Accustomed
	1.7  The Open Source Paradigm: A Way Out

     Chapter 2	Open Source Best Practices
	2.1  Competition and Cooperation
	2.2  Open Source Paradigms Improve Quality
	2.3  Case studies of the successful use of modular techniques

     Chapter 3	A CIO's Perspective of Open Source
	3.1  The Windows Equation
	3.2  The Consultantocracy
	3.3  The Functionality Vacuum
	3.4  Bringing open source into the healthcare enterprise
	
Chapter 4 Recommendations

         Chapter 1

The New IT Management Paradigm

A lot of people have heard about the Linux operating system, and many know about something that's come to be called "open source software." Yet only its aficionados understand it well. I want to tell you something that even the aficionados don't usually realize: the most important strength of this software movement - to a manager - is not the reliable, powerful software it has produced. Its strength is that it brings to us a new paradigm for Information Technology management, one that enables the client firm to regain control of its software costs and its software functionality and to escape the vendor-lock trap.

Most firms use commercial software that's developed and owned by traditional software companies - the "closed source" or "proprietary source" model. Such software can be very good, and relieves the client firm from the complexity and burden of internal software development.

The management cost to a company is that in using standard proprietary commercial software, the client firm gives up control of mission-critical functionality to the vendor firm and opens itself to financial exploitation - "vendor lock."

The commercial model that has evolved around open source is fundamentally different: it parallels the service model of the medical profession: Knowledge is shared openly; but this knowledge must be expertly applied. Professionals compete based on their skill in applying this knowledge to the needs of individuals. Whether any particular professional is chosen by a client is based on a perception of competence, availability and affability.

Any sizable business, especially one in health care, is a complex system. The maintenance of this system is what we call "management." This task requires skill and effort in public and peer relations, personnel management, cost control, and process management.

To simplify, we can divide management responsibilities into various kinds of personal relationship management and function management. Our IT systems mainly are a challenge of function management. To manage functionality well, the manager must be able to control both process and cost.

The main problem with "vendor lock" is that the manager intended - in essence - to delegate IT systems development and maintenance to a contractor (the "vendor") in order to have predictable, contractually specified costs and to take advantage of expertise, but may end up losing control of both process and cost (the "lock"), because if it fires the vendor, the vendor extracts the client company's IT heart - the software. After all, the vendor owns the code and takes it back home when the contract terminates.

In the evolving world of open source commercial software, if the client must separate from the vendor, the client at least can continue operating, as it gets to keep the software, even though it may have to scramble to find adequate support.

But there are more significant benefits to using open source software than escaping vendor lock.

First, competitors can share research and development costs for common components, allowing them to focus their competitive efforts on the customization and service that creates loyal customers.

Second, it brings with it a model of collaborative development. In this scheme, developers and users work together, through debate and cooperation, to forge a system of better quality that could be done working apart. To do this well requires disciplined adherence to a set of procedures and standards for those procedures which together create a productive and high quality collaboration.

Collaborative development is the most urgent need of health care today: we have congruent needs for patient data, technological data, knowledge, and management information. But instead of working toward information systems that have interchangeable basic components, we struggle with an inefficient, expensive, medieval collection of software fiefdoms. This needs to stop. Health care institutions are under intense financial pressure while being expected to maintain world-standard quality. The only way we can make our IT systems more efficient is for institutions and commercial vendors to cooperate in making software tools and modules that share common components wherever there is common need.

Experience has already taught us that it is not enough to create common standards for coding and data interchange. This is the failure of HL-7: we have created an elaborate and all-encompassing system that is manifested chiefly in proprietary, unsharable code. This is a terrible waste of society's capital. Along with creating HL-7 standards, we need to be creating software modules that express those standards, which any client institution or vendor can freely include in their IT systems.

1.0.1 Looking Ten Years Forward

return to contents

If we start working on ways to incorporate open source tools, systems, and collaborative development into our health care institutions, we are embarking on a project that will see fruition in perhaps a decade. Society cannot afford our negligence of this opportunity. We in health care have the training and understanding of what is necessary to maintain and improve the quality of our systems; it is only we who can make this happen. Congress cannot! Our vendors cannot.

This is not a proposal to sunset health care software vendors! Institutions will always find it wise to purchase IT systems and support commercially - this is, after all, simply commercial collaboration. Vendors see vendor lock as giving them security, but this is false: it ensures that the majority of commercial firms will fail, as has continually occurred, and that only the strongest will survive. It thus ensures oligopoly and encourages financial exploitation; furthermore, it removes control of development from health care institutions, who understand their responsibilities, and development is instead led by technologists, with consequent natural inefficiencies and strictures.

1.1 IT Urgencies

We are used to our present software business model. We accept its frustrations and deficiencies as natural and inevitable. If you put a frog on a stove in a pan full of cool water, and turn on the burner, the frog will not notice the gradual change and sit happily until it's cooked. We, the healthcare frogs, are so used to the current paradigms of the IT marketplace that we aren't able to step back and see how terribly dysfunctional it is - and how readily repaired.

1.1.1 Shorcomings of present software business models

return to contents

Current paradigms presume, in particular, that the best and only way to manage software within commercial society is to treat it as privately owned property that must remain secret, that the only meaningful social benefit which can be provided to the copyright owner is royalties, and that the user acquires no rights to proprietary code even if the code is critical to the operation of its business.

The social problem with this model is that these standard assumptions promote financial exploitation of clients by vendors and permit them to attain managerial control of their clients. A financially stressed organization cannot long endure this.

The governance problem with this model is that the vendor that provides services to its government (or the military) is in this area in control of its government (or of the military), a reverse-power situation that will not long be endured by the government. It is this reality, that the government dog must not be wagged by the vendor tail, that is behind the move, first in Peru, and later in other countries, to require state-purchased software to be open source. If the vendor is from another country, this mandate is felt even more strongly. Thus, this movement is not specifically anti-Microsoft; it is pro-independence. That Microsoft is the chief protagonist in this debate is simply a side effect of the fact that Microsoft is aggressively exploitive of its monopoly position.

Beyond the need for management (governmental or private) to have necessary control over its own critical IT systems, it is past time for health care professionals and managers to reclaim control of development. When software only was an elaborate and useful accounting device, the control of development and standards by vendors was trivial. But now that IT systems are essential to every nuance of our enterprises, we discover that we control of the development of nothing important; our medical functionality is in the hands of non-medical technicians, whose main interest is exploiting this control financially.

Does our hospital need intensive care physiologic monitoring? We are forced to shop among vendors, each of which has emphasized priorities somewhat differently. We must accept whatever hardware platform they have chosen, whatever operating system they have developed upon. And we must, regardless of the vendor, accept that its output is not designed to be sent to any interface but the vendor's own, if any at all.

The large health care enterprise finds itself, ultimately with 250 to 400 different software products running on a collection of hardware that displays just how amazingly divergent engineering minds can bee from one another, using several operating systems. Each product requires its own maintenance. And none of these hundreds of systems is designed to interchange data with any other.

Is there something admirable about this?

Let's look in more detail at the hurt.

1.2 Chaos and Paralysis: Proprietary Software in Health Care

return to contents

I'll review the current situation from three aspects:
    I.Costs and fees
    II.Efficiency and control
    III.Exploitation (vendor lock)

1.2.1 Costs and fees. Licensing.

Commercial software carries high license fees that are not proportional to service received from the vendor. This is not an acquisition cost; it's a rental arrangement. License fees are an economic issue only to the extent that they create economic hardship for the client; they are a social issue only if they are so sizable as to represent gouging of an impoverished client by a prosperous software vendor.

Thus whether any license fee is problematic is relative to the economic situation of the vendor and client. In a strictly business sense, licensing fees have long had a rationale in providing a return on R & D costs of the vendor. Some have argued that licensing intellectual property is a monetary reward for the social good represented by the work or invention that it represents. This is a relativistic value only; any fee is determined by current business social norms and the ability of the client to pay it.

Microsoft licensing fees are especially problematic for clients because of their complexity. Managers of small enterprises often hire a full-time employee just to keep track of Microsoft installed software and to calculate license fees; those in large enterprises tell of having to hire legal consultants to help interpret the requirements and of creating small teams that only ferret out and keep track of licensure. It is said that it is impossible to maintain 100% compliance.

Microsoft led the formation of the Business Software Alliance, which is interested in enforcing compliance of intellectual property laws and license fee payments. BSA conduct is threatening to clients because it presumes all unlicensed use to be willful and criminal. The fact is that most unlicensed use is not with criminal intent, but that installation and licensure are not tightly bound at the client, and there are no comprehensive tools available by which a client can assuredly track installation and use of proprietary software. Hence incomplete tracking is inevitable, no matter how much time and effort (money) is put into compliance.

That the BSA considers the use of unlicensed software copies to be a criminal offense, and that complete compliance in a company seems impossible is deeply threatening to the managers. The risk in being caught is high because of the severe penalties involved, because the rigorous enforcement disrupts company functioning, the difficulty and complexity of mounting a defense, and the risk to the manager's own career.

1.3 Control.

return to contents

The loss of control over IT systems that is inherent in the proprietary paradigm hinders management more significantly than does the mere threat of a BSA enforcement visit. Loss of control occurs in several ways with proprietary software. Most of this loss of management control is a side effect of this paradigm, which permits the vendor to keep a dark wall of secrecy around every aspect of his application.

1.3.0.1 The database, as well as the code, is proprietary.

Although the client owns its own data, the vendor owns both the database design and the code which stores it and makes it accessible. Any business catastrophe affecting the vendor may make the client's access to its own data expensive or impossible. Catastrophe could be business failure of the vendor, but also loss of key technical personnel.

More often, the client discovers that modifications to the database are impossible or very expensive, that any change in user ergonomics, new needs for reports or exporting data is not only expensive, but are accomplished on the vendor's timetable, under the vendor's priorities - and with sometimes a whole crew of intermediaries, which practically guarantees misunderstanding.

Thus the proprietary IT management model that calls for a vendor-client relationship causes data to be inaccessible by the client except with the permission of the vendor, which leads to interoperability frustrations when data exchange is needed between software packages from different vendors, and leads to expensive interfaces. This frustration can be abated with modern middleware solutions and languages such as Perl, but this involves intercepting or emulating I/O to or from the vendor software, not access to the vendor database or code.

These restrictions are absent with open code; an intermediate solution is to purchase access to vendor code if possible, which is a solution Mayo has used.

1.3.0.2 Inaccessible code

return to contents

Inaccessible code delays repair of bugs and security holes.

The sine qua non of the proprietary software commercial model is secret code, unavailable to and withheld from the client. Thus bug fixes and security patches are completed on the vendor's timetable, within the vendor's priorities and resources.. This may work well, or it may not; the point for management is that this is not at under the client's control, and it is impossible to bring in under client control except by purchasing the vendor. (Such things have been done.)

1.3.1 Communication is layered. >All too often, there are several layers of client and vendor staff between the user who needs a change and the programmer who implements it. On the client side, the employee who discovers the need for change usually doesn't have sufficient discretionary authority to initiate a change request. There may be a committee or a supervisor and a manager who must understand the need, describe it, and specify the change formally; and as theses things cost money, the expense must be estimated and approved. The person who is actually responsible for getting the work done usually loses her place at the table in this process, so that the change request, after being edited and analyzed managerially, may not be closely aligned to the actual need.

Even if all goes smoothly in the client firm, they typically talk, not to the programmer who must actually produce the code, but to a sales representative, who has firsthand knowledge neither of the work being done nor of the coding process. No one is less qualified to understand and accurately communicate the need. But usually the programmers are kept sequestered, protected from contact with end-users, ostensibly to reduce distractions that would lower programmer productivity, but perhaps more often simply to maintain the pecking order, to justify the existence of layers of administrative types.

Open source does not prevent this sort of common stupidity, but at least in makes it easier to design organizational structures and procedures that avoid it because the client owns both the database and its design. A support catastrophe - loss of a key programmer, for example - is still a problem, but it can be addressed directly by the client firm, which is not faced with the problem of having no access to the code.

Similarly, byzantine communications channels can just as well exist within as between companies, and having open source does not mean that anyone in the client firm can actually read the code or implement change; but it does mean that the client, if necessary, can continue to hold and use the code no matter what happens to the vendor, and can seek competitive alternatives for support.

And if some support or development is done within the client firm, there is always the chance that the user and programmer may meet at the water cooler and productively shortcut layered misunderstanding.

1.3.2 The Sun Sets Eventually.

return to contents

In the proprietary world, it is the vendor who decides whether an application is worth supporting. This is not some mysterious and reprehensible commercial plot to make client firms suffer! The vendor must survive or thrive in order to exist. But its strength and prosperity are not closely correlated to the needs of any particular client. Thus if the needs of the vendor and client are dysconjugate, then the client suffers. It's the rule of commerce. However, if the client uses open source, then it is free to recruit support of software satisfactory to its needs from other sources if its current vendor discovers that to continue is not profitable. The client controls its fate.

A small hospital, for example, used an accounts-receivable package that found cost-effective and satisfactory. After a few years, its vendor was acquired by another firm, which did not intend to continue maintaining the package: its own strategy had been to acquire clients for its own package by buying the smaller company, which it intended to "upgrade" to its wholly incompatible system.

The small system was not Y2K compliant, which created the opportunity to migrate the clients that the acquiring firm wanted. About six months before the end of 1999, it sent notices to the clients using the small system that they would be upgraded for Y2K compliance at the end of the year - at a cost of a million dollars, plus maintenance fees that would be twice what they'd been paying; and of course training would be available, for a fee, in the new system. And by the way, new hardware would be required.

The clients using the small package in this case were able to temporarily rescue themselves through legal action against the acquiring firm, forcing it to make the product Y2K compatible and giving the clients time to bail out in other directions. But in the long run, all the clients had to make conversions that had not been part of their long range plans, and incurred all the expenses of conversion on a short timetable.

All software sunsets eventually. With open source, the client can choose when this may occur. With proprietary software, the vendor has complete control over maintenance, upgrades, and access to data.

1.3.3 Upgrades occur on the Vendor's timetable.

In the proprietary model, the vendor determines when software is to be upgraded. An upgrade may be welcomed; but again, the management point is that the timing and cost of upgrades is not controlled by the client. In the situation where the vendor is trusted, this will not be felt to be an "issue." Again, my point is that with open source software this is never an issue; the client can use its current software version as long as it's cost effective to do so.

In a healthy vendor-client relationship, upgrades are indeed welcome and generally provide a welcome improvement in functionality and fix annoying but non-critical bugs. I do not mean to imply that the proprietary model is useless or consistently unsatisfying. And I don't mean to imply that open source software assuredly makes for an inherently more pleasant IT experience. Here, as everywhere, it is the execution that determines the worth of the relationship. My point is that using open source software puts control back in the hands of the management of the client firm.

Microsoft does not have a client-vendor relationship with any particular firm; their software is mass-produced, not aimed at meeting any particular need. This is not troublesome when Microsoft software works well "out of the box." The problem specific to Microsoft is that of mandatory upgrades and the deliberate absence of backwards compatibility, which forces a company to upgrades on Microsoft's schedule, and forces not only the company but all its customers or clients with which it shares documents to upgrade at the same time.

(By "absence of backwards compatibility" I mean, for example, that Word files created with version 6 cannot be read at all by version 5, nor can version 6 read files created by Word 97. This is what can force an entire industry to upgrade simultaneously, for example clients of US federal agencies who must read the Word documents produced by the agencies.)

1.4 Vendor-centric "community" development.

return to contents

Even when an industry's leaders realize there's a need for common standards or procedures, in the context of proprietary software solutions, vendor interests tend to control this process, as each vendor attempts warp the standards to its benefit. This may have the goal of avoiding the re-writing of its own software, or of forcing its competitors into a position of having to bear a crippling R & D burden to meet those standards, or of simply having the prestige and power of taking the leading role.

But standards development seldom has as its highest priorities simple effectiveness or client-firm needs.

Health care has tremendous common software needs. We work with the same types of clinical data, use the same pharmaceuticals, performs common services for patients - and patients migrate from one provider to another frequently, making interchange of data important for quality of care.

But the proprietary software model has put no priority whatever on interchanging data between providers for the patient's benefit. Each has focused on creating its own solution,and conducting its marketing and development with the goal of becoming the de facto standard through monopolization.

HL-7 has been somewhat less vendor-driven but has been vendor friendly. HL-7 did not from the beginning produce software tools, and followed the proprietary development model. As a result we have a giant, continually changing specification, and almost no common software modules that programmers can incorporate into their development process.

This is - how can I put this kindly - clueless. We have hundreds of brilliant, insightful, zealous, well-trained professionals working under a heuristic that does not have as its main goal solving the problem of connectivity, portability, and communication that HL-7 should have aimed from the beginning to solve.

1.5 Efficiency and control.

return to contents

Perhaps because health care professionals must be perfectionists in order to provide the best possible quality of care, our data systems tend to be overwhelmingly complex. They seem to try to do "everything" rather than identifying what's crucial and doing this well.

Because health care professionals are intelligent, responsible, and articulate, because medical practice is - in the care of the individual patient - an art, each practitioner evolves an work style and work habits that are idiosyncratic.

Because health care institutions tend to be somewhat inbred duchies, each likewise evolves its own idiosyncrasies.

Unfortunately, both individual practitioners and institutions come to believe strongly that their own habits are each the best and most effective. This is a great hindrance to achieving software that meets common needs.

1.6 Exploitation.

"Vendor lock" is a complex and sometimes subtle collection of influences that operationally ties the client to the vendor, making separation unduly expensive - to the benefit of the vendor, which may exploit this financially.

1.6.1 The Exploitation Model

The history of computers in business has been largely one of exploitation of client firms by vendors. The model to which software firms aspire is to achieve a functional monopoly and to exploit this for maximum financial gain. Pursuit of this model has led to a few wealthy, dominant firms and many business failures.

Exploitation is possible because of two factors:
   1.The complexity of IT systems in the enterprise makes migration expensive
   2.The rigid protections on intellectual property under current law require all- or-none conversions and give complete control to the vendor, which is permitted never to disclose its source code.

1.6.1.1 Reprise: Management Challenges to which We've Grown Accustomed

return to contents

The result for the enterprise is --

   - high software costs;
   - more importantly, loss of control of several key management parameters:
     * Cost control: vendors dictate license and support fees.
     * Ergonomics: vendors dictate user interfaces and connectivity capabilities.
     * Personnel: the vendor of course hires its own developers.
     * Planning: vendors dictate
       - upgrades,
       - sunsetting of products,
       - hardware re-capitalization.
   - The possibility of vendor business failure threatens basic enterprise functionality.
   - A threat on Microsoft platforms is of BSA (Business Software Alliance) enforcement actions, a concern because the complexity of Microsoft licensing schema makes perfect compliance impossible and because enforcement actions shut down normal functioning of the client enterprise during "reconciliation" and involve expensive penalties.

Thus the issues for the client enterprise are those of expense, management control, efficiency, and uncertainty or threat.

1.7 The Open Source Paradigm: A Way Out

The important characteristics of open source for management are not that it has brought us Linux distributions and usable applications, or that these are gaining widespread acceptance and use, but that it provides a paradigm of IT project management and a means of IT business management that is outside the proprietary-source box - yet without requiring that any vendor close down business; it's just a different way of doing business; as we said, it's a change from software as "product" to software as "professional service."

How Collaborative Development has Worked

Open-source collaborative development has already produced high-quality systems through distributed development. The GNU project showed the way, Linux development brought into being a reliable, robust operating system to which GNU tools could be fit, the Apache project was the first large collaboratively developed large application, and there have been many more since. A culture of professional support is emerging within this community.

How has this worked? Here is a description by Michael K. Johnson, a participant in Linux development since the beginning and Red Hat Kernel Group Manager.

          Chapter 2

Open Source Best Practices

Written from work of
Michael K. Johnson
Manager, Kernel Technology
Red Hat, Inc.

return to contents

2.1 Competition and Cooperation

Cooperation with competitors is possible. It is wasteful for competitors to duplicate fundamental efforts. If competitors work together to create the infrastructure they all need, this frees resources for each of them to focus on the service and customization that are the most important ingredients of competitive success. Consider any basic industry, such as automobile or furniture manufacturing. The basic design of a car or a couch is no secret; competing companies differ in their aims to satisfy the needs of individuals who have different needs or tastes. To the extent that competing vendors reject cooperation, they also lose opportunity to better meet the needs of individual customers.

When competitors wish to cooperate, a third-party organization can serve as a forum that defuses conflict and promotes objective debate. Within the UNIX/Linux community, these organizations include Unix vendors, Usenix, Open Group IETF, Linux distribution builders, Linux International. Linux Standard Base (LSB). And Xfree86.

Academics and the medical profession have been doing this for years. They compete, but share knowledge and tools. This is also how open source functions. This works for writing software as well as for doing research.

Consider, for example, Red Hat Linux. Competitors use it as base of their distributions. It may surprise you to know that Red Hat takes advantage of this. How does this work? The wide use of Red Hat's Linux distribution makes it the "trusted base." This recognition (and potential market share) allows Red Hat to provide more interface stability, because Red Hat can re-include competitors' changes and benefit from their experiments. Red Hat can take advantage of its competitors' mistakes.

Conversely, Red Hat's competitors take advantage of this situation by trying some modifications before Red Hat does. When they make a good modification, they get reputation and market.

End-users are advantaged by this, as competition enhances both Red Hat's products and its competitors'. The users have choice among distribution providers with different priorities.

Red Hat and its competitors still have proprietary processes; not all code and tools are publicly released. Red Hat does not publish schedules ahead of time (avoiding vaporware); they do internal development when appropriate; they don't always publish code the instant they write it. But Red Hat follows the open-source paradigm in that their default policy is not to distribute binaries without source except where legally constrained.

Red Hat prefers to develop in the open: It is their default policy; otherwise they would be just one more competitor for Microsoft to crush.

Developers, including vendors, do well to make a default policy of cooperating, and should choose secrecy only because of well-developed arguments. In planning open source development ignore vague fears, observe wise caution. Choose secrecy for modules rather than projects when possible.

Opening code has technical benefits as well .

2.2 Open Source Paradigms Improve Quality.

return to contents

How long does the average tech employee stay in one job? Open Source coding and documentation techniques simply produce code that is easier to maintain by the next employee. How many proprietary shops internally write modular, self- documenting, clearly versioned code that's up to the open source standard? How would it aid the task of enterprise IT management to use open source tools and code, and use its standards and techniques internally by adopting those as well as exploiting the availability of open source software?

1. Self-documenting code.

2. Concurrent documentation record.

3. Modular software techniques.

This has several intrinsic benefits:
   - Interface stability,
   - Debugability, and
   - Maintainability.

To use modular techniques does not mean just having multiple C++ files. It means having shared libraries, run-time loaded libraries, and separate programs.

Strong separation forces better design. It can help cleanly separate proprietary from public code, for example.

It also permits the developer to enjoy some of the advantages of Open Source without giving secret research away early, and without giving away proprietary components. It improves generalization, facilitates third party contributions, and facilitates transfer of maintenance.

(For a formal view of open source code quality, see the Reasoning white paper, How Open Source and Commercial Software Compare: A Quantitative Analysis of TCP/IP Implementations in Commercial Software and in the Linux Kernel.)

2.3 Case studies of the successful use of modular techniques.

UNIX text filters have extreme modularization, and have been a historical success.

GIMP plugins have very high modularization, which contributed strongly to the GIMP's meteoric success. Simplicity encouraged third-party participation.

In the Linux kernel, loadable modules make it easy to keep personal work private, and makes it harder to distribute binary-only.

          Chapter 3

A CIO's Perspective of Open Source

From Mike McCoy, MD
   CIO, UCLA Healthcare
   November, 2002

  
return to contents

Because I'm responsible for UCLA Healthcare's entire IT structure, ahead of all else I have to take a systems and enterprise perspective. The medical center - and therefore its IT structure - must function all day, every day, all year. This means that reliability and uptime is the primary goal; all else is secondary, even financial efficiency.

There's a misconception that open source software is costless. The truth is that open source software brings freedom to the user. Open source software is royalty-free. But 80% of software costs are not in royalties, but in hassles.

Open source software is without royalties or significant acquisition fees; but this low financial cost brings a social responsibility to the user: to help foster improvement in the software, and to return to the community improvements it makes. This is a small but important cost of free software, software libre.

Open source software liberates the user from expensive hassles and inefficiency that encrust the vendor-driven software markets:

The traditions of this market begin with expensive "consultants" who are constricted by their own traditions to recommend only solutions that yield for them and their business clients rich financial rewards, without really considering whether the solutions they recommend are in the short run ergonomically efficient for the client health care center or in the long run will be malleable to its evolving needs or to improving technology. The consultant is only the thin wedge of a process of sales - consulting, evaluation, and contracting - that seems to require a succession of expensive, minimally functional specialists who have no accountability for the eventual value of their proposed software "solution." for the center.

An example of contemporary hassle after installation is in regards to Microsoft licensing: their licensing arrangements are so complex that you need consultants to merely to figure out what you owe.

Proprietary, closed-source software involves many daily hassles in implementation and operations. Vendors use dongles, software time bombs and other nonsense to guarantee that use of purchased software is strictly within licensing limits; these devices create unreliability, inefficiency, and expense for the user.

The user is interested in service. This includes whatever helps make the software run reliably, helps it fulfill the actual work requirements of the user, permits reasonably-costed modifications for emerging special needs. If you're paying for service you're paying for what you need. There's no possibility of copyrighting or patenting service; you can't steal service from people! You can only ask for it, and you will only continue to receive it if the compensation for it is fair.

Vendors often waste vast tracts of time fussing about small details of contracts, details that have nothing to do with service and much to do with defending territorial rights to use of intellectual property that are of little or no interest to the user. Abandon fussiness, give me service.

Vendor-driven software is not developed to meet the needs of customers. "Functionality" is often touted, but without a prior examination of whether the product's built-in "functionality" matches the user's needs. This would be an easy challenge if functionality only were a matter of dollars, if the cost of software were somehow magically linked to its functionality. But the functionality of commercial software is miserable, and the cost is linked to what the market will bear, compared to the other miserable "solutions."

At UCLA, we spend $20M on capital IT structure, $20M on licenses, and $20M on software maintenance and internal development. What does that $20M in license fees buy us? The right to possess and use software that does not make us more efficient.

In the vendor market there isn't that much I want to buy! Quality is lacking: functionality, reliability, malleability, responsiveness all are deficient. On the other, hand the open source software we've used is of unbelievable engineering quality: it's humbling simply to look at it. Meanwhile the software code base of most major vendors is laughable. Vendors have 495 people in marketing and 5 in research and development. Their programmers are badly overextended; there isn't a rigorous review process; fix after fix is layered onto the code.

3.1 The Windows Equation

return to contents

Into this salad of expensive, badly conceived mediocrity is added the Windows Equation. Empty-headed prophets got millions of sweet, ignorant people to think that Windows Is The Future. Sadly, it will be if everyone thinks it should be. Among vendors there is a herd mentality, all stampeding to Windows as an "upgrade" on the server. Sadly, there are major software vendors who have working, scalable systems on Linux who are saying, "The customers want Windows." They end up with clumsy Windows systems that require much more hardware, more maintenance, and function less well that Unix or Linux systems.

For example, IDX has converted its radiology information system from Unix to Windows; on Unix it wasn't beautiful, but it was reliable, solid, inexpensive. Now only the "Windows upgrade" is supported; at UCLA, one Unix box has been replaced by a cluster of 20 NT servers - it's a horribly inefficient system. Furthermore, to generate reports from it, Crystal Reports is required; this additional package can't be run as added task on the existing 20 NT servers, it requires a new box.

This is a continuing frustration with Windows platforms: because their operating system design is deficient, if you deploy Windows you basically need a separate box for every process, increasing the number of licenses you have to buy from Microsoft and worsening electrical bills and hardware and network maintenance.

Because of these hassles, UCLA Healthcare avoids vendors that require Windows only; but the number of vendors offering cross platform applications is shrinking.

What health care centers need is vendors and software that deliver functionality. Unfortunately, buying the established vendors instead of open source software reduces short term personal risk for managers because no one ever got fired for buying (for example) Cerner, even though it's dogged with problems, because it's the conventional wisdom to do this.

3.2 The Consultantocracy.

return to contents

A frustrating challenge is surviving the "consultantocracy:" Nothing is done without consultants; but there's an incestuous relationship between consultants and vendors. The consultants are unable to look beyond their dysfunctional proprietary-software box to realize that in the open software market there are solutions which are of vastly superior design quality, functionality, and reliability.

We administrators need to ask ourselves: Why should software cost millions to install?

Consultants have no interest in anything that can be installed easily; consultants reward inefficiency because it creates more work for them! - more billable hours.

The long-term risk to health care centers is Microsoft's P/E ratio. As Microsoft's sales plateau, they can maintain their P/E ratio only by increasing revenues from current business - upgrades and licenses. The consequence is that they are ratcheting up license fees in every way imaginable. This is causing a great fiscal burden to many companies, not only health care.

Competition among proprietary vendors is fierce. But instead of competing for customers based on service, they are competing for total control of their customers. As companies struggle and fail, they are going bankrupt; they are being bought. But this is not resulting in the commodification of product that would occur in any other industry with resulting lower costs for their customers. Instead, the opposite: Because their grail is control and exploitation of this control by vendors, not service, the products of acquired companies are "sunsetted," and because the code is secret and cannot be acquired or maintained by the customer, the new owner forces unwanted software replacement at a cost of millions in license fees, hardware replacement, and employee retraining. All to replace systems that were already functioning, whose replacement was not planned or controlled by the user.

The stampede to Microsoft platforms by medical software vendors exacerbates this cost inflation and unwittingly protects Microsoft's PE ratio; the result is that Microsoft, secure in its control of the market, are migrating to a licensing scheme that allows them to increase revenue seamlessly; they can dial profit up as desired. As cash flow and new sales flatten out, users will pay dearly to keep the OS. Microsoft's control of its market and thus its cash flow makes it reluctant to sign a contract that goes beyond a year, Even the University of California is not large enough to negotiate with Microsoft; "take it or leave it" is the rule.

For UCLA Healthcare, the cost of moving to IDX on Windows was $5M. It was working fine on Unix; UCLA had other, more pressing institutional priorities than changing versions of medical software. But IDX was convinced that "Windows is the future;" production and support for its product on Unix was stopped, and upgrade to the Windows platform was required, regardless of the needs, financial pressures, and priorities of the institutional user.

3.3 The Functionality Vacuum

return to contents

There are tremendous challenges and opportunities for any firm willing to use the resources of the open source software community to deliver quality and value that are an order of magnitude above what is provided now by closed-source commercial vendors.

To permit this to evolve, awareness and advocacy is essential.

Managerial leadership is necessary, for consultants not aware of open source, and will ignorantly lead clients into inefficient and expensive closed-source solutions.

Within academic medicine the American Association of Medical Colleges (AAMC) may be a forum in which healthcare leaders can educate each other on the possibilities for more cost-effective software solutions that will also put users back in control of their IT budgets. It is possible to conceive of AAMC forming a software collaboratory to share development of high quality medical software.

It's important to publish best practices. For example, Clinicomp, which provides software for bedside charting, is so far the only vendor in US to offer Linux systems for enterprises. Clinicomp leaders are fanatical about quality and reliability; they changed from SGI platforms purely because Linux is much more robust for 24x7 real time ICCU monitoring. Their Linux system supports 1000 users simultaneously and runs for years non-stop on little inexpensive boxes. Compare this with Windows systems.

3.4 Bringing open source into the healthcare enterprise.

return to contents

Executives don't know what operating system is underneath their software, and they don't care as long as it's working. IT professionals are usually allowed considerable freedom in choosing their software tools, and so you can be a champion of open source simply by using it.

Where can open source be used in the healthcare enterprise?

First, it is ready for use in commercial support infrastructure. There are many open source tools that can be used to increase efficiency and reliability without disturbing contractual relationships.

What development models do we have? Where is the ActiveState (www.activestate.com) of healthcare applications? We need an independent, service-based, collaborative enterprise to develop language and modeling tools for healthcare. This does not seem to be Siemens or HBOC - but each of these firms has the size and leverage to become this if they could discover the open source model.

Who has a track record for true innovation? There's never been a Bell Labs of Medical Informatics; we'd be greatly benefited if there were one, but who will support it? We can begin by creating a "collaboratory" of open source development among healthcare institutions.

Innovation is important, but in a pragmatic sense is overrated: Most tools are copies of an existing design ... It's easier to design a complex system if you have a working model. But healthcare does not have a well-designed implementation that can be widely emulated: We don't have an EMR comparable to UNIX, the model and standard (POSIX - http://www.opengroup.org/onlinepubs/007904975/toc.htm) to which Linux was built.

If we want to make progress, we also have to avoid boiling the ocean. Medical care encompasses all aspects of human distress. In providing software "solutions" to aid our work, we try to tackle everything at once. Grandiose projects come from this mistaken ambitiousness. The level of abstraction these projects take on elevates them well above the level of practical effectiveness. We spend 5 years designing an abstract model; then we discover it's too complex to be implemented. Consider the HL-7 project and learn from it. If you want to make a career of drawing blueprints, this is your right, but most of us are working hard to built a basement and keep it dry and warm.

So if we want to make a difference, we need to make a little piece that does something nice first.

Another thing that hinders achievement is "religiosity" versus secular practicality. Being pure is admirable, but my job doesn't require purity; it requires that the IT infrastructure of UCLA Healthcare be up and running 24 x 7. Open source people hobble themselves by thinking it has to be all open source. Using non-open-source, Windows pieces pragmatically is OK. We can't cripple our enterprise by wishing an open source tool or application were available, we have to take whatever works, even if it's expensive, mediocre, or impure.

Even though open source is built on collaborative development, frameworks and methodologies for sharing are lacking in open source healthcare. Yet the building supplies are out there. Examples are XML - It's extensible; you can do a little piece first. Or Zope - you can download into an environment and find it works. Open source solutions are marvelous for Web services.

I keep hammering on vendors to open their source. I spare no vendor. For example, I told Cerner that UCLA didn't buy them because they don't run on Linux. We all need to do this because it saves cost and effort (personnel time and costs). Linux takes little support. OS/390 Linux saved us a bundle because we can take all these Wintel boxes and put them on the mainframe. All the disaster recovery stuff is then integrated with the mainframe backup.

A step in the right direction is Clinicomp's WINCIS frontend. Because it is CORBA-based, you don't need the source if you have standard connectivity. This should be standard practice.

What can the government do? There is an appropriate role of federal government in encouraging the dissemination of functionality. Yet right now public funds are put into the development of closed source programs, public money disappearing into proprietary systems and subsequently benefiting the public not at all - and sometimes not even the vendor unless the vendor can find a way to create and market a commercial application.

As a general principle, government funding of vendors development should add value to government-funded basic tools and paradigms that are then openly available to the healthcare community.

The entire University of California means nothing to Microsoft financially, so we have no bargaining position. We hope to make UCLA independent of Microsoft so that we can make business decisions. Linux systems run for years non-stop on nice little boxes. I can't imagine what a 1000 bed hospital would look like on Windows. Sometimes the CIO's don't know what OS their systems are running on.

How best can we make progress? Medical open source developers should focus on little useful tools rather than boiling the ocean. Managers can be open source advocates simply by using open source. It's not necessary to develop open source to support it. Just use it.

When UCLA Healthcare is all open source, that is going to be a powerful message.

         Chapter 4

Recommendations

return to contents What management steps can be taken to make use of the strengths of Open Source?

I: Employ Open Source software wherever it makes practical sense for your enterpise. To do this effectively, it's necessary to learn what resources that are available. This may require consulting your own technical staff or making use of technical consultants who are expert in open source resources.
   A: Infrastructure. Linux is a reliable, robust operating system. GNU/Linux tools are excellent for web pages servers, file and print service, communicatiosn, email, and other IT infrastructure tasks because of their reliability, design excellence, and resistance to security problems.
   B: Applications. Work with vendors to port current UNIX applications to Linux, in order to reduce license fees. Study open source application resources to determine whether they are adequate for our needs.

II: Seek platform and application independence within the enterprise to avoid unexpected vendor entrapment.
   A: Require internal documents and communications to be in non-proprietary or universally supported formats.
   B: Work toward removing our dependency on Windows where licenses are not nnecessary (e.g., smart terminals, email, simple word processing, most spreadsheets, "desktop" software).
   C: Pressure vendors to use Linux platforms, and not port applications working well on other platforms to Windows.
   D: Pressure vendors to provide platform-independent or X-windows frontends.

III: Work legally toward fail-safe mechanisms against unwanted sunsetting or business failure.
   A: Contractually require proprietary code to be provided with compiled software, under NDA, as part of contracts - with limited ownership rights if the vendor fails or is absorbed into another firm.
   B: Require new software acquisition projects to survey for open source applications, include the option of contracted or in-house adaptation of extant software that is not exactly suitable.

IV: Solicit support contracts from professional open source firms to encourage the growth of the professional commercial software model (i.e., firms that are not tied to specific applications).

V: Support industry-wide basic applications
   A: MPI (master patient index)
   B: Seek support models for universally-needed functionality through mechanisms other than licensing (e.g. CPT codes)
   C: Support open source HL-7 modules using software tools that are open.
   D: HIPAA security application plugins.

VI: Migrate to open source platforms and applications as a long-range plan for your IT system.

VII: Participate as founders in a medical open-source initiative with other groups.such as AAMC, AAFP, WorldVistA, OSHCA, OpenEHR, EU-Spirit, and others as they emerge and we discover them,with the following goals:
   1: Pragmatism: Create a usable software project to implement every cooperative project undertaken: code should be an integral part and necessary outcome of every joint project, whether for standards or implementation.
   2: Cake before frosting: Identify common essentials through debate; aim at prioritizing desirable non-essentials for subsequent work
.   3: Follow the discipline of collaborative concurrent documentation
.   4: Develop and respect leadership
     a: technical leadership
     b: medical leaders
     c: maintainers Copyright (C) 2003, Daniel L. Johnson, MD All rights reserved.

return to contents