Waldo Jaquith

I write a lot about how government should procure custom software, but all of my high-falutin’ theories are built atop some crucial prerequisites that I don’t often write about. I recently wrote about the enormous value of hiring people who understand how technology works, and in the spirit of that, I want to point you to Will Callaghan’s recent blog entry on the subject: “How to be a better collaborator.”

Though I want to paste in the whole thing, I’ll give you the seven major points he reviews:

1. Consider other people’s needs as well as your own
2. Share everything openly
3. Acknowledge what you know and don’t
4. Have a shared endeavor
5. Find simple ways to stay aligned with others
6. Always be iterating
7. Work in an organization that supports all of the above

Will’s audience is individual collaborators, but I want you to think of this organizationally. Successful government software projects have created an environment in which all of these things are possible. And that’s hard! Sharing everything openly runs counter to many agencies’ instincts. Acknowledging what you don’t know can be scary, especially for anybody from a marginalized demographic. Working iteratively is a battle against the headwinds of long-term planning. In many organizations, it’s not on individuals to work in this way, but on leaders to foster an environment that provides the circumstances and psychological safety necessary to allow people to work in this way.

No amount of procurement, budgeting, or oversight changes will bring about a successful software project. They can create the circumstances under which it’s possible for a project to succeed. For that to happen, the project should be run as Will describes.

When government buys custom software (whether built from scratch or having commercial software customized), it costs a lot of money. For both federal and state government, $100 million is not a shocking price tag for a large project. I’ve worked on software projects that top $1 billion. The question I get all the time about this is, simply: why?

There are dozens of reasons why government software is often both expensive and bad, but these are the biggest reasons:

1. The price tag isn’t just for software—it’s everything. In fact, actual software development is a minority of the cost. Agencies toss everything up to and including the kitchen sink into these contracts: creating and staffing the program management office, running the help desk, providing tech support, hosting the software, monitoring performance and uptime, maintaining the software, outsourced oversight (IV&V), preparing innumerable reports and PowerPoint presentations, and just a smattering of actually creating software. Putting most or all of that on a single contract inevitably means that most or all of it gets outsourced, with associated markups and inefficiencies.
2. We’re at least 30 years into a pricing death spiral. When the average price tag is 310% of the originally budgeted price, it stands to reason that you should expect the next project to cost more, and vendors adjust their bids accordingly. (This is adjacent to Hofstadter’s law: It always takes longer than you expect, even when you take into account Hofstadter’s law.) Prices are a one-way valve—they only ever go up.
3. Price tags are a funhouse mirror. Neither program nor procurement staff know what software should cost. After a career of eye-popping price tags, staff have become inured to them. $10 million for a trivial change order is just normal. $300/hour for a junior software developer seems reasonable. $50 million as the all-in cost for a modernization of a major agency system feels like the going rate. Folks from the private sector balk at these prices, and they’re right to do so. These prices aren’t tethered to reality. But if those price tags are all you ever see, they just seem normal.
4. Agencies’ solicitations aren’t set up for the best software developer to win—they’re set up for the best bidder to win. There are very few vendors equipped to write a 500-page proposal in response to one of these kitchen-sink solicitations—this is not a competitive market. The few vendors who can manage that are the ones who get contracts over and over again. Is their work any good? That’s beside the point. The important thing is that they can get contracts. They’re very good at that.

There are other reasons for the large price tags on government software—waterfall development, vendor lock-in, not conducting user research, diffuse responsibility, lack of in-house knowledge, and so on—but it’s my experience that these aren’t the major culprits. Want the biggest reduction in pricing? Attack those big problems.

My job is working with state and local agencies on the budgeting for, procurement of, and oversight of the major software projects that undergird every agency. I have lots to say about how to get that right, but it can really all be boiled down to one piece of advice: hire people who understand how technology works.

By hiring coders, designers, user researchers, product owners, etc., agencies can stop getting ripped off (and stop publishing RFPs in which they absolutely demand to be ripped off).

The mission of nearly every government agency is intermediated by software. If the software fails, the agency fails in their mission. If a state unemployment department’s UI system goes down, there’s really no sense in which they’re an unemployment department. If a state Medicaid agency’s Medicaid Management Information System is broken, there’s really no sense in which they provide Medicaid coverage.

And yet state agencies outsource this work. The thinking is that they’re not in the business of software, they’re in the business of unemployment insurance or health insurance or whatever. Anything to do with the software is seen as the domain of software vendors, up to and including the oversight of how and what those vendors are doing. But the software is the UI and the health insurance.

I’ve spent the past decade on helping with this problem in a bunch of ways, but all of the solutions I propose are inferior to simply hiring software developers. These are the people who can evaluating agencies’ existing software, understand the needs of the users, perform market research, test software, help prepare solicitations, evaluate proposals, and oversee the work of vendors. Heck, with enough software developers, the agency can even write software themselves, instead of outsourcing it.

So, sure, you can get better vendors to bid, do a better job of checking vendors’ references, and do a better job preparing to outsource, but better than all of those is simply hiring people who understand how software is made.

When government agencies need to have custom software built, as a matter of course they require that the bidding vendors have built that exact thing before. I understand why this seems like a good idea, but it’s a mistake.

At best, state agencies are willing to say (for example), “our state’s higher education financial aid processes are unique, but we’d be willing to tolerate a vendor who has previously built financial aid systems for state higher education systems.” On its face, this seems sensible. Getting a vendor with extremely similar experience would seem to reduce risk.

Here’s the catch: there are very, very few vendors with experience building that extremely niche software. Like…three? And that’s not to say that they did it well. Were their past projects completed on time, within budget, and within spec? Probably not. Did the delivered systems address the needs of the end users well? Almost certainly not. This is not a competitive vendor pool. The odds of success are not good. Imagine finding a homebuilder by limiting your pool to contractors that had built the exact house that you want—the same layout of rooms, same siding and roofing material, the same fixtures. You could do that, but no reasonable person would.

The solution is to expand the definition of “past performance.” Has built exactly this thing before is too narrow. The way to expand the vendor pool, increase competition, and get more bids from competent vendors is to use a more common analog, something that actually exists broadly in the commercial market. To return to our example of a state higher education financial aid system, it’s better to think of that as a case management system. Students complete a form, the application is subject to a series of automated checks, some applications are queued for review by a state employee, and ultimately each application is either accepted or declined, with a dollar value attached to an acceptance. I appreciate that there’s more to financial aid systems than that, but the core of a financial aid system sounds like a straightforward case management system with a public-facing component.

Case management systems: now there’s something that lots of vendors have experience with. A huge percentage of government software systems are just case management systems in a $50 million trench coat. There are commercial case management systems that can perhaps be configured to serve the current need, there are open source case management systems that can perhaps be modified, and even writing a new case management system isn’t wildly difficult. There are hundreds of vendors with experience implementing case management systems. Some of those vendors probably have experience doing things that are functionally near-identical to financial aid systems, despite the purpose of the implementation being to source auto parts or handle customer complaints or manage travel reservations.

With few exceptions, specialized government software can be understood to be minor variants of extremely common software that’s widely used in the private sector. Agencies will be best served by understanding what software that their need is a minor variant of, and seeking vendors with experience at that. Changing the scope of “similar experience” from highly specific government experience to something broader—but equally relevant—will increase competition, lower costs, and will damned sure make success more likely.

Agencies sometimes tell me that they employ some software developers and that they intend to assign those developers to supplement the labor of the vendor team that they have hired for an Agile software development project.

This is a terrible idea. Please don’t do this.

I sympathize with the underlying motivation! We’re contracting for developers, and we already employ developers who have a lot of existing knowledge about our systems, so surely it makes sense to have these groups work together. With that level of detail, yes, this is very sensible. But the sensibility ends there.

The vendor team is a cohesive unit, working under a single organizational structure, under a single manager, under a shared HR system, using common tools, norms, and practices. Your agency’s developers are part of none of that. Worse still, your agency’s developers are on the wrong side of the fence, the customer side of the fence, putting the vendor team in a really uncomfortable position if they disagree with a technical decision made by your devs. If the agency contracting officer believes that the vendor is not in compliance with the contract in some way, and the agency’s developers are involved in that non-compliant work, now you’ve got a mess on your hands.

So what can those agency developers do? Well, one of them can act as the technical lead for the agency, working under the government product owner to ensure that the code delivered by the vendor on a sprintly basis is in compliance with the quality standards laid out in the contract. (As a rule of thumb, this work requires ¼ of an FTE per scrum team.) And another can be that technical lead’s backup, so that the lead can take leave without creating trouble. Agency developers can also work in parallel with the vendor team, building or improving supporting infrastructure or related infrastructure, applying the knowledge learned by the vendor team as they conduct user research. Sometimes those developers can do all of the technical work within the agency’s IT boundaries, allowing the vendor to work exclusively within their environment. But whatever they do, they need to leave the vendor team to do their work in their own way under their own terms.

I love hearing that agencies employ developers. I love when agencies want those developers to support the work of vendor scrum teams. But it’s crucial to position that support correctly to avoid accidental self-sabotage.

There’s a simple thing I preach that is worth writing down: a good model for a pull-request-length relationship between a government agency and its software development vendor. I didn’t create this approach—this was the model that we used on the projects I worked on at 18F, starting nearly a decade ago. Here’s the approach.

The agency creates a repository in GitHub.¹ If this is for a completely new project, the repo may consist only of a README—that’s fine. If the repo is for something that already exists, then of course the existing code is put into the repo. The vendor forks the repo. The vendor is given no permissions for the agency’s original repo—they have no power to change anything there. The vendor does all of their work in their fork. This is safer for both the vendor and for the agency.

At the end of each sprint (or user story, or feature, or whatever milestone you want to use), the vendor files a pull request against the agency’s upstream repo. The pull request is the vehicle for the agency conducting a review of the work, to ensure that it meets the code quality requirements found in the contract, and to ensure that the completed stories meet the definition of done. Designated agency employees can comment, request changes, and do all of the normal things one does when reviewing a pull request. Once the pull request is judged to be satisfactory, the agency can accept it (I recommend having the technical lead sign off on it in the form of a comment, leaving it to the product owner to actually accept the PR). This is the point when the agency is saying “yes, this work meets our standards,” so it’s possible that the contracting officer’s representative may want to be notified or otherwise play a role here.

(Relatedly, it’s important to use the agency’s repo for all user stories etc., not the vendor’s repo. Agencies need to be able to end a relationship with a vendor without losing their backlog.)

And that’s it, that’s the simple approach that protects both the agency and the vendor, allowing a proper review process, and keeping the contracting officer happy.

1. I say GitHub, but I assume this works equally well in any Git-based repository, e.g. GitLab. As long as there’s a mechanism for a user to fork a repo and subsequently file a pull request to commit changes to that upstream repo, this approach should work. ↩︎

When embarking on a new software project in government, “custom or COTS” is a presented as the initial dichotomous decision. (“COTS” meaning “commercial off-the-shelf” software.) But in terms of technology, that’s a false dichotomy, propped up by both vendors, for their benefit, and government employees, following a policy-prescribed process. The truth is that nothing is purely custom software, and nothing is purely COTS. There is, instead, a continuum, and it’s a short one.

Open Source Software is COTS

Let’s define our terms. COTS is any software that is commonly used in the private sector, can be purchased trivially, and is available for use as-is, whether licensed from a vendor or used under an open source license, whether purchased or acquired at no cost. In fact, let’s look at the Federal Acquisition Regulations’ definition of COTS, though note that it covers all manner of commercially available off-the-shelf products, and not just software:

Commercially available off-the-shelf (COTS) item —

(1) Means any item of supply (including construction material) that is–

(i) A commercial product (as defined in paragraph (1) of the definition of “commercial product” in this section);

(ii) Sold in substantial quantities in the commercial marketplace; and

(iii) Offered to the Government, under a contract or subcontract at any tier, without modification, in the same form in which it is sold in the commercial marketplace; and

To go one step deeper, let’s look at the definition of “commercial product”:

(1) A product, other than real property, that is of a type customarily used by the general public or by nongovernmental entities for purposes other than governmental purposes, and–

(i) Has been sold, leased, or licensed to the general public; or

(ii) Has been offered for sale, lease, or license to the general public;

(2) A product that evolved from a product described in paragraph (1) of this definition through advances in technology or performance and that is not yet available in the commercial marketplace, but will be available in the commercial marketplace in time to satisfy the delivery requirements under a Government solicitation;

(3) A product that would satisfy a criterion expressed in paragraph (1) or (2) of this definition, except for-

(i) Modifications of a type customarily available in the commercial marketplace; or

(ii) Minor modifications of a type not customarily available in the commercial marketplace made to meet Federal Government requirements. “Minor modifications” means modifications that do not significantly alter the nongovernmental function or essential physical characteristics of an item or component, or change the purpose of a process. Factors to be considered in determining whether a modification is minor include the value and size of the modification and the comparative value and size of the final product. Dollar values and percentages may be used as guideposts, but are not conclusive evidence that a modification is minor;

(The definition goes on, but it won’t add to our understanding here.) We can see that, per the FAR, open source software is a “commercial product.” Because government had some some rousing battles about this in the early ’10s, the Department of Defense put together an Open Source Software FAQ that eliminates any uncertainty that might be left by navigating a web of regulatory language: “[N]early all OSS [open source software] is ‘commercial computer software’ as defined in US law and the Defense Federal Acquisition Regulation Supplement, and if used unchanged (or with only minor changes), it is almost always COTS.”

Custom Software is Mostly COTS

The layperson may envision custom software as something completely bespoke, each underlying line of code lovingly hand-crafted by an artisan dev, using only organic, locally grown electrons. But custom software is more like building a Lego set of your own design, with each piece a separate open source software program. Some amount of the software will be written from scratch (casting Lego pieces in your own molds, to stretch the metaphor), but as measured by lines of code, even a big, complicated system will be overwhelmingly comprised of open source (COTS) Lego bricks—some north of 90%.

COTS is Made of COTS

It sounds like a truism to say that a COTS application is made up almost entirely of COTS, but what I mean is that, just like open source, it too is comprised overwhelmingly of open source Lego bricks. The major distinction between custom software and a COTS application is licensing—in the former, the custom portions are exposed to you, but in the latter, it’s custom to the vendor, not to you. Having passed through the layer of the licensing vendor the custom bits are transformed into COTS. A 2024 report by Synopsis, based on their analysis of 950 commercial codebases, found that 96% contained open source software, and 77% of the code within those code bases was open source. No doubt most of those commercial codebases contained a substantial amount of sub-licensed COTS components, too.

I occasionally encounter IT agencies that are stuck in the 1990s, who will tell me that that “open source isn’t secure.” Not only is this profoundly wrong, but COTS is made almost entirely of open source! There is no escaping open source software. The world runs on it.

Custom or COTS is a False Technical Dichotomy

The decision of whether to buy a COTS system or build a custom system is legitimately important, but for reasons that have nothing to do with the software itself. The technical distinctions between the two are basically academic. There is no pure COTS, there is no pure custom, there is only an enormous stack of open source software with a thin layer of original code atop it.

“COTS versus custom” should be a decision based on matters of cost or support or stability or lock-in or risk. But not based on differences in the software. Because they’re pretty much the same.

I recently wrote about how independent verification and validation (IV&V) is almost always useless for custom software projects, and nodded toward how it could actually be beneficial. Now I want to go into depth about what useful IV&V looks like.

When I wrote that earlier blog entry, there was a model for useful IV&V—18F. At the end of February, that federal team was shut down by a vindictive Trump administration, as a part of their project to demolish any part of government that is functioning well.

The ostensible purpose of IV&V is to ensure that a government procurement is going the way that it’s supposed to—that the vendor is adhering to the requirements of the contract and the deliverables are up to spec. IV&V comes into play when government lacks the in-house expertise to enforce the contractual requirements. Sometimes the agency decides to hire an IV&V vendor, but more often a funder imposes an IV&V requirement on the agency to guard their investment. IV&V doesn’t actually do that well for custom software projects, but it does provide an elaborate performance of doing that, which is apparently good enough.

18F never billed itself as providing IV&V, but that’s pretty much the process that we put together in my time there, under the umbrella of our acquisitions work. After helping agency partners through the procurement process, we’d help them in the post-award phase, teaching them how to work with an Agile vendor. (I think of this as the Cyrano de Bergerac problem—after whispering all of the right words in the partner’s ear throughout the procurement process, it’s important to stick around after the contract is awarded, lest the vendor find the agency’s high-falutin’ Agile talk isn’t supported by experience or ability.) We developed a standard process and set of services designed to help the agency ensure that the vendor was performing well, though also to ensure that agency was able to keep up with a high-performing vendor. This involved somewhat more than verification and validation, but I’ll explain the whole suite of services, for the sake of completeness.

Help the Agency Keep Up with the Vendor

For a lot of agencies, contracting with a good Agile vendor is like dropping a Ferrari engine into a school bus. There’s going to be a lot of wasted potential. Very few agencies are set up to work with such a vendor at the pace that those vendors work. The vendor’s scrum team will have questions about the product, the business requirements, and the tech stack that the agency cannot respond to fast enough to avoid leaving the scrum team twiddling their thumbs.

At 18F, we’d show up before the procurement process, not just to walk them through the procurement, but also to prepare them to work at vendor speed. That often meant building a prototype with Agile methodologies, which would allow the agency’s new product owner to practice actually being a product owner and allow us to test the process for getting to production, and we could use that as an excuse to both experience and improve the process of providing vendors with access to their environment. This meant that, once a contract was awarded, vendors could get to work immediately, instead of spending months navigating onerous agency processes that are optimized for decade-long tech projects that are outsourced lock, stock, and barrel to a single enormous vendor.

Coach the Product Owner

New product owners should go through a training program (I generally recommend Scrum Alliance’s Product Owner training), but that classroom training only prepares somebody so much to actually be a product owner. At 18F, we’d provide them with ongoing coaching as they learned to do the job. That meant ensuring that knew what it means to ruthlessly prioritize the MVP, helping them to maintain a backlog of user stories that are sized well and supporting a properly incremental development pattern, giving them the confidence to lead scrum ceremonies, and generally showing them what good looks like.

Enforce the Code Quality Requirements

The contract should (must, actually) prescribe standards of quality for the work, probably in the form of a quality assurance surveillance plan (QASP) that speaks to code quality, accessibility, documentation, security, etc. It’s rare for an agency to have anybody on staff whose job description encompasses reviewing vendors’ work for conformance with technical requirements…but it turns out that their IT shop is likely to have one or more people who have those skills, if unused from 9–5 Monday through Friday. At 18F, as with product owners, we’d coach these folks as they learned to play the role of technical lead. We’d train them on how to evaluate the sprint’s deliverables in the terms laid out in the QASP, how to perform a code review, and how to work respectfully and productively with the vendor team. It took a long time to get somebody in place to serve competently as the technical lead, so somebody from 18F would serve in this role on an interim basis, gradually replacing themselves with the agency’s technical-lead-in-training.

Start Strong, Fade Out

18F’s work on these projects always started off strong, with a team of 2–4 people joining every scrum ceremony, holding multiple standing weekly meetings with the partner, and generally filling roles that would be better filled by the agency’s employees. But the goal was always to transition all of that work to the partner agency. At one point we formalized this, setting up a big spreadsheet with a row for every task that 18F was performing, and two column headers: one labelled “18F,” one labelled “Agency Partner.” Our job was to transition every one of those tasks from the 18F column to the agency partner column. That would take months, even years, but it meant that, one day, the partner could ask “so…what is it you do for us again?” And that was when 18F’s IV&V role was no longer needed.

I’m not sure that it makes sense for a commercial IV&V vendor to strive for uselessness. It’s financially irrational to work yourself out of a job. But IV&V contracts are often imposed by external forces (e.g., federal funders), who could hypothetically require that IV&V not merely provide (often useless) oversight, but also train grantee agencies in how to oversee vendors’ work themselves.

Of course, 18F was killed by the Trump administration, so that non-conflicted partner is no longer an option for agencies. And federal grants have also become somewhere between unreliable and mirages, so that force for IV&V is much reduced. But state and local agencies‘ need for functioning software and project oversight is unchanged, and may actually become more urgent as they need to step up where the federal government is crumbling while lacking federal funding to do so. IV&V can be an important part of ensuring that systems are delivered on-time, on-budget, and within spec, but only if it’s IV&V that’s actually useful.

I’ve long worked in the open, overwhelmingly for one reason: it increases enormously the surface area for success. That might mean thinking out loud on social media, documenting ideas as blog entries, or publishing software as open source. Here are some of the benefits.

My ideas improve by being shared freely. If I’ve hit on something good, more people will know about it if I share it openly. Or if I’ve come up with something that’s actually bad, then people will have the chance to tell me that, so I can fix it or reverse myself. I’m not aware of ever having had a good idea on my own—it’s only in conversation (literal or figurative) that my ideas mature.

I am often wrong about where my work applies. Something that I produce openly can spread beyond what I think its audience is. Often I’m wrong about the highest and best use of a concept, a program, an illustration, whatever. Perhaps I think I’m making a compelling case for agencies publishing software as open source, but in fact I’ve made a really great case for why the EU should support open science hardware standards (which I know nothing about).

I am often wrong about when my work applies. Sometimes, a thing I’ve produced isn’t wrong, I’ve just created it at the wrong time. It’s out of step with the zeitgeist, or maybe it solves a problem that people don’t know they have. But if I’m patient, circumstances will change and something I did years ago will become timely. Its ongoing availability means the work is always waiting to be useful.

My work can be improved upon. Sometimes I’ve created a platform that others can use to create something better still (if inadvertently), whether incorporating my work or replicating it with improvements. Sometimes that benefits me (by improving the thing I created), sometimes it doesn’t, but it’s yet to harm me.

An odd effect of working in this way is that I am often credited for doing things that I don’t feel that I deserve any credit for. I do 10% of the work, others do 90% of the work, and I get credit for the result. But those others will tell me that they wouldn’t have done the last 90% if I hadn’t done the first 10%, perhaps because they wouldn’t have thought of it, or perhaps because they couldn’t stand seeing something so obviously incomplete.

Working in the open means that whatever I’m trying to do is far more likely to happen, or that my inchoate thoughts are more likely to be turned into action by somebody else. I’ve been at it for my entire adult life, and don’t intend to stop. Recommended.

I don’t love government’s entrenched habit of outsourcing anything related to technology. If I had my way, government would be one of the United States’ largest employers of software developers. But if we accept the frame that government is going to continue outsourcing software development, then it’s important that we reconsider how much to outsource. I don’t mean what percentage of all software development, I mean the extent of each project that is handed off to a vendor. What might it look like to reduce the outsourced extent, if that would reduce risk? Where could we draw the line?

I think the line is found on the far side of control. For reasons—some good, some bad—agencies often outsource projects so thoroughly that they give up their ability to control the projects’ outcomes. Agencies think that they’re owning control in the form of the requirements in the contract (dozens or hundreds of pages of shall statements), but they’re not. Writing a really whiz-bang set of requirements is empirically unrelated to project success. Requirements are put together without performing user research, they can’t change as the project team learns more, and it’s impossible to make them so comprehensive that the vendor can’t get them wrong. Requirements are not control, they are the illusion of control.

What does it look like to retain control? Agile gives us the needed framework. Control is embodied in the form of the empowered agency product owner: somebody who prioritizes the backlog, verifies the completion of user stories, and oversees user research and design to ensure that it’s sensitive to policy requirements and the demands of leadership. Control is also embodied in the form of the agency’s technical lead—somebody who works at the direction the product owner to ensure that the vendor’s technical work is meeting the quality standards within the contract and and is appropriate within the agency’s environment and standards. These are the control tasks. Once you outsource any of these tasks, you are taking government out of the user research ←→ software development cycle, and you’re right back to dropping your agency’s fate in the hands of the usual suspects.

You can outsource your programming and your design and your user research and your hosting and your O&M and your help desk and your documentation, you can outsource all of it, but if you outsource the control tasks, you haven’t meaningfully reduced your risk of failure, you’ve just compartmentalized it.

This is not easy advice to act on. Agencies often don’t want to hear that they need to name a product owner, empower them, and name a technical lead. Those aren’t normal job titles or job descriptions for agencies. They don’t have somebody on the bench ready to take on an entirely new full-time set of tasks. They don’t have software developers at all, let alone available to start overseeing vendors’ work. But, I argue, it is better to take on those battles and own your agency’s fate than to risk blowing another $50 million on another failed software project.