Tales From the Crucible-Secrets of a High-Tech Expert

or "YOU CALL THAT A SYSTEM?? I'LL SEE YOU IN COURT!!"

by Warren S. Reid, ©Copyright, 1995, All Rights Reserved.

SYNOPSIS: This article discusses the increasing trend of litigation as a solution to failed systems projects. It presents the viewpoint and experiences of an expert in the systems planning, development, testing arenas. The legal process is briefly introduced as is the role of the expert witness. Then, three cases are presented showing: the systems and the parties; the complaint; the analyses, strategies, and opinions; and the verdict. Finally, some recommendations are shared that will help you if you happen to find yourself in court after an alleged failed systems project. **************************************************************

The Future Is Here!

Unfortunately, the future of your success as either a high technology user or a producer/developer of high technology hardware, software, and services, may not depend on the brilliance of your technical staff, or the speed of taking your product idea to market, or your global strategic alliances. In today's ever complex society, Silicon Valley and business users are finding the newest high tech growth industry to be -- litigation!

Your success today and in the future may actually depend on the skills of your legal department -- and not your other traditional production and service departments. The legal industry has been gearing up for the influx of new cases involving failed systems projects as well as intellectual property misappropriation. Organizations such as the Computer Law Association, the World Computer Law Congress, the International Computer Law Forum, the Software Publishers Association, as well as the International and American Bar Associations are devoting huge amounts of their resources and programs to educate constituents in litigating complex high technology matters.

Jury of Your "Peers"

The most disturbing prospect, is that these cases all begin as a good faith effort between you and the opposing party to develop and implement successful systems solutions to benefit you both. Instead, you will now have to duke out the merits of your case under the worst circumstances, including:

Worse than that, you could find yourself arguing your case in front of a jury much like those in Bobbit, Menendez Brothers, or Rodney King cases. These are the same jury pools from which your jury will be selected -- meaning that you may very well have your systems evaluated by a group of people often chosen specifically because they were economically disadvantaged or from under-educated backgrounds!

And I predict that this is just what is going to happen to many unwary vendors and users in the very near future. The amount of time, resources, and energies spent in developing complex systems is simply too great and the impact of failed systems is simply too visible to let a failed projects go without a fight, without explanation, and (depending on the mindset of the parties) without retribution.

Jury Selection

A few more words about juries and the legal process is in order so that no reader glamorizes what may happen after he/she threatens to sue the other party -- and gets his/her wish. Jury selection is not a science by any means. There are some jury psychiatrists and psychologists that would like to see overweight white women from the suburbs, or former male physical education teachers, or citizens with English as a second language be the appropriate jurors to judge whether or not your complex on-line, real-time, fault tolerant, mission critical system was adequately delivered -- and whether the bugs that have appeared are reasonable under the circumstances. In fact, there is no simple solution.

As you have probably noticed in the O.J. Simpson trial, histrionics and drama can in some cases overshadow the truth. I'm aware of a situation where the prosecution's main witness, who was obviously losing the case and also had the most to gain, began to wail like a baby. While the jurors eventually found the company innocent of any fraud (after four and a half weeks of deliberation), they made a very strange recommendation to the court. They awarded a seven figure sum to the "wailer" -- in my opinion, simply because he broke down so thoroughly. My belief was confirmed in interviews with two of the jurors after the judgment was given.

The result of any lawsuit is still somewhat unpredictable, and using the legal process is no substitute for being lazy and/or ineffective during the contracting, planning, management, systems development , testing or quality assurance processes.

Juries Don't Understand

Some believe that if they only get the right experts, they will win their case. In fact, some experts have been likened to prostitutes -- although I believe that is out of order. In my experience, any expert who really is or can be bought off (i.e., to testify to anything) is unlikely to ever get any more work and therefore the system weeds out such dishonest experts.

What is more likely to happen is that the experts counter-balance each other -- with one expert swearing 'the glass is half empty' and another swearing 'the glass is half full'. Similarly there are so called "experts" who believe that metrics and function points are useless tools and others who believe they are imperative to the development, control, and management of good systems development projects.

Sometimes, winning can be the issue of "which expert does the jury like more?". This is particularly distressing. For instance, an expert has a beard that is salt and pepper in color. If that expert reminds a juror of a deceased parent that the juror loved or enjoyed, such juror may be more likely to hear what the expert has to say and even side with that expert -- whether or not s/he really understands what the expert says. On the other hand, if that juror had a horrible relationship with that parent who had the expert's demeanor or looked like him/her, that juror may be unable to really hear that testimony or it may be uphill battle.

Thus, the issue of going into court, to seek real "justice", can be a slippery premise indeed. While my experience shows me that the legal system works in most instances, there is a chance that, even if you are absolutely in the right, an acceptable judgment may elude you.

Role of the Expert

What then, is the role of the expert? In my opinion, it is to educate the judge and the jury, that is to try and explain the complicated systems development processes, technologies, interfaces, and requirements to people who very likely have much less experience than you would hope. While educating, the expert must use appropriate gesturing, speaking style, and dress for that particular courtroom -- else his/her communication with the jury might be impaired. Also, remember that most parts of the trial can be very boring to jurors. Establishing rapport and the importance of the expert's testimony is critical in the first 5 minutes of his/her testimony. Developing a strategy for getting the complex points across to the jury is a critical service the best experts provide.

Another role of the expert is to explain the fact pattern, to put together the pieces of the mosaic so the jury can see what happened. Getting the jury to actively listen and "participate in the testimony" (for instance by saying "picture this" or "put yourself in the following situation" when answering the attorneys' questions) will improve the ability of the jury to put itself into the events in question and remember them better.

Thirdly, the expert will opine on the issues described in the complaint, explaining in an independent manner what this set of circumstances means in terms of the contract, in terms of acceptable performance, in terms of the flexibility of the system, or portability or maintainability or fault tolerance, or whatever the issues are. In giving his opinion, the expert tries to help the jury understand from his independent vantage point and professional experience just what happened and what a reasonable conclusion might be.

Lastly, the role of the expert is to assist and/or train the attorneys (with whom s/he is working) on the technical issues and questions that should be raised during the discovery process, while deposing witnesses, and in the trial. Competent experts tell the strengths and weaknesses of the case to the attorneys early on and at every step of the way. They have the courage to resign from the case when pressure is put upon them to flagrantly "spin" the facts or shave their testimony. Competent experts recommend to the attorneys that they are working with that they consider settling or dropping a lawsuit when the facts do not support the case from the expert's technical, systems, management, or business point of view.

Of key importance is that the expert must carry out the above role with independence. He or she is not an advocate. To help retain that independence, experts are retained by the attorneys that represent the clients and not by the clients themselves. Furthermore, they are paid before they render their opinions so that their fees are not contingent on what they say or how they perform during testimony.

Fabulous Failures

With all of the new wonderful "panaceas" given to software developers and users over the decades to help assure successful system solutions, such as rapid prototyping, structured methodologies, JAD (Joint Application Development), object-oriented programming, test coverage analyzers, COCOMO models, function point metrics and software tools, project estimation and management tools, etc. successful systems still elude many otherwise successful businesses. Indeed fabulous systems failures include the following:

The list goes on and on and will continue as other failures like the Denver Airport Baggage Handling System, California's Department of Motor Vehicles systems and other notable failures continue to come into public view.

Let's not forget one of the worst system's disaster ever -- Jurassic Park. Although Jurassic Park is fictitious, you may recall that its system was based on the false assumption that only female dinosaurs existed in the park (so once the system counted to the expected number of dinosaurs -- it stopped counting -- even though new dinosaurs were being born in the wild), and that one brilliant but disgruntled programmer (Mr. Nedry) that provided no systems documentation was enough to keep the system running forever.

Example Cases

With this setting re high-tech experts' roles, and the fact that failures and runaways will continue to happen, I would like to present a few cases where I was engaged as an expert.

In reviewing such highly summarized cases, it is easy for the reader to underestimate the amount of sleuthing and preparation required. Once you know the answer it's so simple -- but in some cases we have analyzed as many as 500,000 -700,000 pages of documentation to arrive at and support our conclusions. With two or more sides to every story and even more contradictions, it takes hard work and tenacity, along with clear, honest, and persuasive communications with judge, jury, and attorneys to win the day.

"The Dispatcher-Car 54 Where Are You?"

The System and the Parties

This case involved one of the world's largest vehicle dispatching organizations (hereinafter called the "Dispatcher") and most complicated computer-aided dispatch (CAD) systems. This CAD handles thousands of different vehicles, covers a million miles, requires a giant workforce, while managing special hazardous situations, assisting local police department dispatching and many other functions. It needs to "work" best and be absolutely non-stop during peak performance situations when natural or other disasters strike.

The opposing party, the "Integrator", was a billion dollar plus consulting firm entering the CAD industry, that rightly believed that being the Integrator for this dispatch system for this entity would give it worldwide renown and instant credibility. After many delays in delivery and poor testing results, the Dispatcher canceled the contract with the Integrator for cause.

The Complaint

The Dispatcher was sued by the Integrator for over 100 million dollars even though the contract was only for a fraction of that size. The Integrator's claim charged that the Dispatcher's unreasonable contract termination for delay, faulty design, and inability to complete the project negatively affected their future business worldwide -- and thus caused the Integrator's loss of bids to develop CADs in Korea and Europe. Among other things, they sued for future profits lost. Such illogical and vastly inflated sums are oftentimes just the starting point for negotiation and settlement. But because such erroneous logic can sometimes elude a jury, these inflated demands must be taken very seriously from the outset and defeated.

The Integrator further blamed its inability to finish the project timely on the Dispatcher's alleged continuous changes and unwillingness to freeze specifications. Despite this, the Integrator claimed it poured additional monies into systems development to responsibly keep the relationship and contract intact. It claimed to be on the "two-yard-line" with only a little more time -- weeks -- to deliver a fully tested and functional system.

The Analyses, Strategies, and Opinions

I and my team reviewed tens of thousands of documents during discovery. We interviewed Dispatcher witnesses and reviewed depositions from both parties. From this, we concluded:

The Verdict

The Integrator Company was an "employee-owned" company in a city then experiencing a harsh recession. While the local jury later told the attorneys that judging against and penalizing its own citizenry was extremely difficult, the Jury awarded not a single dollar to the Integrator -- and more than $1,000,000.00 to the Dispatcher to cover attorney fees. This was a stunning victory indeed!

"The Good, the Bad, and the Ugly"

The System and the Parties

This second example is interesting because it started out as a copyright and trade secret infringement case and it ended up being won mostly based upon testing objectives. Sometimes you'll never know where the cases may take you.

In this case, one party was a small systems applications developer an IBM Value Added Reseller (the "Developer") that developed, licensed and maintained vertical software for the wholesale distribution business. The Developer formerly an electronics wholesaler, developed his own systems solution when no appropriate ones were available. Eventually he enhanced and globalized it so that it was appropriate for the industry. Over time, because of his 25 years as an "insider" (i.e., a wholesaler electronics distributor) and a visible presenter at major industry conferences, his product became extremely robust with functionality and very popular. Now in his seventies, and with a terminally ill son, he wanted to sell the software company, but in such a way that his employees would have a place within the acquiring company.

The opposing party (the "Acquirer") was a billion dollar systems developer and integrator, that already enjoyed success with its products in manufacturing arena. It believed that having an outstanding complementary distribution product would critically complete its product line offering. It believed it was presently losing customers who wanted integrated manufacturing and distribution packages.

The Complaint

The Acquirer desired to get software with the same look and feel as its manufacturing package as soon as possible. Therefore, it went on the acquisition trail. After looking at many available systems, the Acquirer gave the Developer a multi-million dollar letter of intent to purchase the assets and software of the latter company with the right, in its sole judgment, to cancel the deal if after intense examination the software didn't meet the needs of the Acquirer. Preliminary discussions and visits had already taken place so that the Acquirer generally knew about the organization and the software, its history, its pricing and costs, its competitors, its functionality and capabilities.

The Acquirer claimed it now needed to review the Developer's software code, technical architecture and structure in detail; approve its design; interview the customers, programmers and testers, trainers, implementors, maintenance staff, hot-line group, marketing staff, and sales and sales support staff; review standards and documentation; review and analyze the testing suite; etc. and then apply its own tests, to be sure that the system worked as promised and could be maintained by the Acquiring company.

This all sounded reasonable at the time, especially since applicable law required that the Acquirer act in good faith -- even though it could cancel the deal in its sole judgment. This means that the Acquirer could not just change his mind for no reason, some immaterial reason, or for a fraudulent purpose. The letter of intent also provided that the intense review within 45 days.

Bang! On the 45th day exactly, the Developer received a one page letter which indicated that the Acquirer would not complete the acquisition because the system did not meet the performance requirements promised by the Developer and the profitability requirements needed by the Acquirer. In addition, the system could not be backed up in less than two hours using multiple tapes -- thus necessitating an employee to stay an additional two hours each day. There were two other small points in the letter of rejection.

The Developer at first charged the Acquirer with copyright infringement because the Acquirer in a very short time came out with a similarly configured system. Later, after we arrived and based upon our opinions, the attorney also charged "bad faith" and "trade secret misappropriation" (i.e., Acquirer entered into contract with no intention to complete it in order to learn the successful Developer's secrets to best develop its own competing product) -- a very difficult thing to prove!

The Analyses, Strategies and Opinions

During our research we discovered the following:

The Verdict

After four years of arguing this case, it was scheduled to go to court on one particular Monday. On Friday night, as I packed my bags to leave, the case was settled almost literally on the courthouse steps in favor of a happy Developer and his attorneys.

"How Many Ways Can You Slice a Pizza?"

The System and the Parties

This complicated case was eventually fought in a Federal courtroom in Des Moines, Iowa. The system itself was an ingenious system that in fact would turn one of the world's largest pizza chains, Pizza King, into over 5,000 self contained pizza manufacturing and distributing plants. It was a system designed to capture customers' pizza orders by phone and get them into the customers' homes ASAP -- where every minute counts! Its functionality includes:

This case involved three parties:

  1. Party "A" - a developer for point of sale (POS) equipment and with clients including Denny's, Winchell's, Church's Fried Chicken, Burger King, etc. (The attorney for this party engaged us.)
  2. Party "B" - also in the fast food POS business boasting clients including McDonald's and Taco Bell.
  3. Party "C" - formed by a group of professionals and financiers from IBM, Big Six Accounting Firm, and MBAs from top U.S. business schools. Their role was to finance these systems at $700-$900 per month over 5+ years (most pizza store owner/operators cannot afford to pay $35,000 or more for a multi-register system that's integrated with back office operations).

The Complaint

During the development of the system by Party "A", Party "B" after a long and detailed review of "A's" system and assets, acquired Party "A". Thus "B" was now responsible to complete the system and develop a long term maintenance capability. Party "C" was in a very big hurry to finance the systems and discount the paper to other financial institutions. Well into the project, "C" unilaterally decided "the system didn't work and would never work -- and that it was fatally flawed", and that it should be paid for $80,000,000 in profits it could have made over the next 5-10 years had the system been rolled out to Pizza King stores internationally.

Party "B" decided to join with "C" in saying that the system "didn't work." "B" alleged that "A" lied to it regarding the quality and status of the system-in-progress that it acquired (even though "B" was certainly capable of performing such an independent audit and analysis). They figured that it would be easier to prosecute "A" and perhaps win a settlement amount than to defend the quality of the system currently in its second pilot testing phase at some twenty sites and perhaps pay no monies (or receive no monies as "C" was really a shell company with no assets).

The Analyses, Strategies, and Opinions

To win this case in court would require our defining what "the system works" actually means. We would also have to show that "A" didn't misrepresent (or worse) the facts to "B". More importantly, to win the case, the attorneys and we believed it would be imperative to get "B" over to "A's" side defending the quality of the pilot system and the development efforts. But how could we achieve all this?

After reviewing the documents and learning more about this case and the roles of each party, we developed a three point approach that would define to what extent the system "worked" by:

  1. The Hardware Works -- Comparing it against the contract specifications for mean time between failure (MTBF) for the systems' hardware components
  2. The Software Works -- Defining a metric against which the readiness and health of the system could be compared
  3. Users Liked It -- Getting affidavits and confirmations about how the system worked from pilot store users and, if possible, from "C's" own internal documents

The Hardware Works (MTBF) -- to prove this point we analyzed every single error report and problem report that the system experienced in the five months it had been pilot tested in the stores. We did this for every component at every store (such as IBM PS/2, a make-station printer, a network controller, a bar code printer, order entry terminals, voice stations, wall masters, network components, etc.)

We eliminated from our analysis data involving "bad sites." The contract permitted this. A bad site was one which had an average MTBF of more than twice the average of all of the pilot sites including that store. This would eliminate special one-time burn in situations and poorly managed stores.

We also adjusted the data for situations that would impact true MTBF calculations but should not be counted in our opinion. Further analyzing the true failure occurrences, we were able to prove that while the system did not perform adequately during the first 4-5 weeks in the field, it began to meet and exceed the required MTBF after that time. Once it surpassed requirements in week 6, it continued to improve -- never falling below the contracted rate again. What was so powerful was that the contract provided for more than six months before that phenomenon was required to meet the contract!

Our challenge, was to explain to a jury that while the MTBF for a particular component was 260 days, or 730 days, or 2,300 days, that the MTBF for the whole system of 22 components mathematically came to just 35 days per site. While the opposing side argued, "How would you like a car that needed to go to the shop every 35 days?" (something the jury could relate to), we had to explain and convince the jury that if one order entry terminal went down or one of the ticket printers was unavailable for a few hours, the system still worked and still met the performance requirements of the store -- and still allowed a pizza to get out of the store two to three minutes faster than with the old system (a critical requirements for performance).

The Software Works -- to proving the software "worked" or was about ready for full production release was more difficult because there were no standards in the contract, and because all non-trivial systems have defects. What are acceptable defects and acceptable defect levels depends upon the specific case and the instant facts.

What we did here is identify, by combining all relevant problem software logs (kept by "A", "B", "C" and in the stores), every software defect and its severity level for the 9 months of systems testing and pilot testing. We then developed simple graphics to show the jury that the system had reached the kind of "stable state" that we would expect before a production release.

We charted software defects opened and closed for each monthly period; totals by month of defects opened and closed by severity level; average time to address/correct defects. We found that over time the defects found went from 90 to less than 10 per month; that severity of new defects went from "catastrophic" and "must fix immediately" in the early period to "deferrable" and "cosmetic changes"; that the defects backlog went from a high of over 100 in the early months to less than 10 per month; and average days to fix a defect was dramatically reduced from 84 days to 5 days.

Combined with the facts we had already shown that the system contained the contracted for functional requirements and that quality control was adequate, these new statistics and trends proved to me and the jury that the system was stable and, just about ready for production from a software standpoint.

"Users Liked It" -- Proving this was tricky - but luck was with us! During discovery, we learned that "C" was going to develop its own pizza system and, in our opinion, appeared to promise to use functionality that looked a lot like "A's" and "B's" system -- although they would recreate it in a different language on a different hardware platform. "C's" own business plan to raise money for this new business endeavor indicated how much current users liked "A's" and "B's" system and how the current users would be either references or the next buyers of "C's" new system -- a system which "C" would offer with more beneficial financing (based on transaction usage) to pizza store managers.

"C" went on to praise the functionality, beneficial usage of the system, and their own experiences with "A's" and "B's" system to their bankers and shareholders as well. "C's" own project status reports indicated that many (not all) franchisees were complimentary about the system or its potential; and they included letters from users specifically praising the system -- although the system was not perfect!

We discovered that no user rejected the system during the entire pilot test and that, even though the parties have stopped supporting the system after the lawsuit began, half of the customers continued to use the system daily for another 220 days to 352 days. This usage, in an environment where even a minute or two of delay or downtime can hurt reputations and guarantees of "half-hour piping hot delivery", proved to me and to the jury that the system worked.

"The Turning Point' came, interestingly enough, after my deposition was taken for 4 days regarding the above facts and opinions, Party "B" fired its expert who tried to prove that the system didn't work -- and joined forces with "A" to win the case -- an important accomplishment. This made sense because "B" had actually finished up the system and performed over half of the pilot installations. Part of their arrangement to join "A" was that they be able to retain WSR Consulting Group as their own experts as well as "A". After the testimony above, "B" simply couldn't continue to argue that the system did not work. Now "C" would have to prove that alone.

"It Keeps Going and Going and Going ..." Just like that famous battery we further proved that "C" continually kept changing the system's requirements to get as much as they could from their relationship to "A" (and now "B") and refused to freeze requirements. We particularized 106 changes (some very significant) over and above what the contract required "A" and "B" to deliver. We further showed that "A" and "B" in an attempt to keep the project alive and "C" happy was issuing new enhancement releases of the software at the rate of two per month.

The Verdict

After 4.5 weeks of deliberation (so even with all of the above it wasn't an easy decision for the jury) the jury agreed that the system "worked" and they gave a stunning victory to the "A" and "B" team.

What Can You Do to Prevent These Things, or Win in Court

While there are no guarantees for winning your case, consider a following maxims for systems development projects that may very well lead to successful systems solutions and that if needed in a lawsuit may help:

Some Frequently Asked Questions

Now that we have gone through some cases and cautionary recommendations that can help you stay out of court -- or win in court -- I'd like to share with you some questions I am frequently asked regarding how to approach large scale systems failures and relationships with attorneys:

What should an attorney look for in an expert?

The short answer is look for someone that you can work with. The litigation will have short deadlines at times, and will probably require several changes in strategy as more information becomes available. You need someone who can thrive in that environment. Look for people that can come to preliminary conclusions quickly based on incomplete information. Look for someone who has been through it before: i.e., from a technical standpoint, and from the legal process.

Like a marriage or the hiring of a key executive, if the initial selection decision is wrong, a lot of therapy and grief will follow. Make sure the personalities are a good match and that the expert is: articulate; empathetic; loves to teach; can establish rapport with a jury (and judge); is credible; knows what he/she is talking about; is detailed; can simplify the complex; and can put words together into pictures (word pictures and graphics). Don't be lazy, check references! Note this is pretty much the same advise that I give experts re the homework they should do re attorneys that have approached them -- before they agree to render services!

What do you look for when you start and engagement?

In addition to the usual documents (chronology of events, names and roles of each key person involved, contracts, complaints, critical business and legal documents), large scale systems projects have their own language and documentation. Such documents include the following (and this is by no means and exhaustive list):

  1. Systems Development Methodology Used
  2. Standards Used (in programming, design, testing, documentation, reporting, etc.)
  3. Systems Requirements/Specifications (including functional, performance, growth, security, futures, etc. targets)
  4. Project Feasibility Report
  5. Systems Overview Chart
  6. System Test Procedures, Plans, Results, and Discrepancy Management Reports & Statistics
  7. Acceptance Criteria and Test Plan (with results)
  8. Design Overview (and Preliminary and Detailed System Designs)
  9. Detailed Workplans and Work Programs (staff estimates by task and actual staff Hours)
  10. Automated Tools Used (for testing, capturing defects, documentation, estimation, progress reporting, coding, conversion, etc.)
  11. Systems Conversion and Hardware and Software Installation Plans
  12. All Deliverables
  13. Various Inspection, Walkthrough and Review notes
  14. Configuration Management Procedures, Tools And Utilities Used
  15. Quality Assurance Plan
  16. Project Management Reports
  17. Contingency Plans

It's the piecing together of these documents and many more, along with interviews of percipient witnesses, knowledge of the technology and/or specific client's industry, and knowledge of the computer industry's "standards of quality", where the expert can really make a difference!

What one thing is critical to the success of the lawyer-expert working relationship?

Open communications between the parties. The attorney needs to make and keep his needs known to the expert and vise-versa. As strategies change, and legal issues or positions change, the expert must be kept aware and contribute appropriately to those strategies and changes. The expert must be allowed to disagree and challenge the attorney's points of view where the technology, project, or industry records warrant such behavior. Keeping the expert in the dark re progress in the case that impacts the expert is a road to surprise -- and isn't the whole legal system filled with enough uncertainty already?

How do you approach a large scale systems failure job or other high-technology matter?

Large high-technology projects, whether systems failures, runaway projects, getting a product to market, protecting or infringing intellectual properties, etc. are really a nexus of many factors (just like pieces of a mosaic) that I believe must be analyzed and pieced together wherever possible. While it is beyond the scope of this article to discuss all of the pieces and their complex interrelationships, the main pieces for a systems failure matter might include the items summarized below:

  1. The Strategy Track
  2. The User Track
  3. The Project Management Track
  4. The Technology Track
  5. The Quality Track (Closely linked to other tracks)

A Final Thought

Judge Edenfeld once mused that the jargon used in the computer field makes the legal Law Latin or Norman French seem as clear as the Ten Commandments -- and that while using exactly the same words, experts uniformly disagree as to precisely what they mean.

Now there's a word to the wise!