What Went Wrong With The 'Tiny Nonprofit' Behind The State's Beleaguered Vaccine Site

When Gov. Charlie Baker released vaccine eligibility to a million more Massachusetts residents last week, state websites crumpled under a virtual stampede of people rushing to secure appointments. Users were left in online limbo for hours, and many gaped at how something so critical could be entrusted to, in their eyes, flimsy web services and fragile program designs.

“Foolish system. Ill-Designed. Who are the people that designed it?” seethed Steve Rosenfeld, 70, from Worcester in a written comment to WBUR after he failed to get an appointment.

The nonprofit Maryland Partnership for Prevention runs one of the crucial websites that crashed last week, a service it provides in more than two dozen states across the country. The company's software allows people to register their COVID-19 vaccines and make appointments. In Massachusetts it is used, among other things, to manage signups at state-sponsored mass vaccination sites.

By its own account, the company is a “tiny little nonprofit.” For two decades, it built and managed simple software programs designed to streamline flu vaccine consent forms and registrations in Maryland school districts.

Then, in just the last year, this "tiny" nonprofit transformed its software into a massive program to manage COVID vaccinations. It’s called PrepMod. It powers the appointment systems of vaccination sites and health clinics in 27 states including Massachusetts.

Although local health clinics say, by and large, the software helped their sites run smoothly, the failure last week exposed glaring flaws in PrepMod’s design and management choices.

These flaws and the problems they caused were preventable, according to Jay Jamison, an executive at Boston-based web technology company Quickbase. Developers have long had solutions to what he calls the "news day problem" - typically when media reporting causes a surge in traffic to a normally sleepy website.

“These are known problems that you can design around,” Jamison says. “These are problems that many firms face, and many others have solved the problem of how do you stand up applications that get a ton of traffic quickly, and how do you do it reliably."

Companies just have to plan for it, Jamison explains, and those with a deep pool of experience in web application development wouldn't make what he sees as, essentially, rookie mistakes.

I knew what the system needed to do. I designed the screen and the functionality because it’s the work that I do.

Tiffany Tate, executive director at the Maryland Partnership for Prevention

When Tiffany Tate, the executive director of the Maryland Partnership for Prevention, began working for the nonprofit, she was fresh out of graduate school. She saw the position as a stepping stone to bigger, more lucrative corporate opportunities.

“I thought, I’ll work this job for a couple of years, and then I’ll go on and make the big bucks and run a company,” she says.

The nonprofit had a single purpose: boost flu vaccination in Maryland school districts. At the time of its founding, two decades ago, Tate says she and her co-workers were floundering under piles of paper consent and registration forms. The information on those hard copies had to be manually transcribed into electronic school and state systems, and it took forever.

“We said, 'there’s got to be a better way.' So, we got this idea to create software to collect consent electronically and manage that information by school,” Tate says.

This concept led to ReadiConsent, the precursor to PrepMod. Tate says it worked fantastically, saving her organization time and increasing the flu vaccine coverage rate in Maryland schools. Districts in other states also began using the software.

Tate is proud of the impact her work has had, and her passion for vaccines bloomed. Instead of gunning for a cushy corporate management job, Tate dedicated herself to immunizing students, particularly in those in underserved schools.

“I love immunizations. Vaccines save lives,” she says. “For 24 years, I’ve been working in public health. It’s all community-based work where my beginnings are in Baltimore City, trying to reduce the disparities between African-Americans and whites for flu vaccinations. Making sure people who need vaccines have access to them.”

Tate believes her experience managing ReadiConsent and other vaccine programs for the flu prepared her — and the nonprofit she now runs — for the massive undertaking of rolling out critical software during the pandemic. She'd been thinking about just such a possibility since the H1N1 scare of 2009, she says, and knew that many of the features she’d implemented in ReadiConsent would come in handy for clinics rolling out COVID vaccines. And so, in late 2020, PrepMod was born.

“I knew what the system needed to do. I designed the screen and the functionality because it’s the work that I do," she says. "When the pandemic came along, we said, ‘well let’s evolve [our software] and add a few more features and make it available to everyone."

If you’re going to pay a lot of money, no matter what that number is, you want it to perform.

Jay Jamison, executive at Quickbase

According to documents provided by the state, the Maryland Partnership for Prevention, under a subsidiary called the Multi-State Partnership for Prevention, invoiced Massachusetts $318,000 for a lifetime license to PrepMod and other programs, including the flu vaccine software ReadiConsent. The nonprofit invoiced another $120,531 for personnel costs over three months, starting in January 2021. It appears to be a better deal than what some other states got – Pennsylvania paid over $850,000 for its contract.

The new income must been a welcome sight for the Maryland Partnership for Prevention. Financial filings from the last decade show the nonprofit has been operating in the red since 2014. Its most recent public tax form, from 2017, lists Tate as the only "key employee." The nonprofit lost almost $150,000 that year.

Tate says even with all the PrepMod licenses sold during the pandemic, Maryland Partnership for Prevention will likely not make it into the black. The price charged to states for its licenses was only designed to cover the cost of developing the software, Tate says, not to stuff her pockets with public health dollars.

“We’re here to serve, and we have not positioned ourselves to make, you know, millions and billions of dollars,” Tate says. “We have a lot of expenses that are probably going to cause this to break even or go in the red.”

But Maryland Partnership for Prevention’s lack of stable revenue is a concerning sign to Quickbase's Jamison. From running a tech company of his own, Jamison knows how costly developing web products can be – particularly when those products handle health information. That also means a tech firm needs to have good cash flow to develop and deliver a good service, he says.

“A vendor having financial stability is a really important signal of being able to maintain and continue to deliver a repeatably above expectations experience for their users,” Jamison says. “I think that’s a really important frame to think about.”

The money the state spent on PrepMod seems not quite worth it to Jamison. It’s not just that nearly half a million dollars is a lot — although, “that’s a lot of money for software, there’s no doubt about it,” Jamison says. It’s that Maryland Partnership for Prevention failed to deliver a reliable and robust product. The website the nonprofit set up to handle Massachusetts vaccinations ended up being an obstacle to residents.

“If you’re going to pay a lot of money, no matter what that number is, you want it to perform. For me, it seems like a lot, mainly because it didn’t meet expectations,” Jamison says.

[PrepMod]’s really been helpful. It’s taken the burden off a local board of health... it's been a great tool.

Kristin Black, Health Agent at Northborough

Local health clinics that have used PrepMod for weeks disagree with Jamison on some points. Kristin Black, a health official for Northborough, says PrepMod was one of the reasons why the vaccine clinic at the Northborough Senior Center was able to run smoothly.

“It’s really been helpful. It’s taken the burden off a local board of health,” she says. “There’s currently a 24-hour requirement for reporting vaccines administered, and so I think it was only because of that tool we were able to really meet that requirement of 24-hour reporting. So, it was a great tool.”

For Black, PrepMod streamlined the registration process. It made it easy for her to send the vaccination data to the state’s immunization information system and helped manage appointments. For day-to-day clinic operators, Black says PrepMod wasn’t hard to use – although it required some training and had “a big learning curve.”

Tate’s background in influenza immunization also meshed well with the operations of local clinics. Black says PrepMod was good fit for small administrative jobs like those at local boards of health.

Black does have some criticisms, however. Mainly, she says a couple of design flaws created inefficiencies. For one, she and her staff had to start a completely new registration for every patient’s second vaccine dose. She wished the system would remember patients, so they didn’t have to walk people, particularly seniors, through the electronic process twice. She also knows the sign-up process hasn’t been easy for a lot of her residents.

Worse, Black says some boards of health experienced crashes with PrepMod. In Hopkinton, the software went down during a vaccine clinic. The staff had to revert to paper records until the system came back online. 

"It was a mess," says Hopkinton's health director, Shaun McAuliffe. Luckily, he had printed all of the registrations at home the evening before the clinic.

"Had I not, I [would] have had to run the clinic blind, unable to confirm which resident of the Commonwealth had registered," he tells WBUR. "Since the system was down, I had to enter and close out every registrant by hand and then enter everything digitally."

I would get a message that sorry, there were no slots available at that time. So, I’d go back … and do it all again… I was able, at the end of 5 hours, to see how many times I’d tried: 57

Sarah Metcalf, 67, from Northampton

On the morning of Thursday, Feb. 18, an estimated million Massachusetts residents suddenly became eligible for COVID vaccines, and the appointment system's flaws became catastrophic. Millions of clicks flooded the PrepMod website each hour. It sputtered in the sea of traffic.

“Typically, we’re dealing with about 60,000 queries an hour,” Maryland Partnership for Prevention’s Tate says. “It went up to 70,000 hits per minute… about 4 million an hour. Our system just didn’t catch up with the demand. The sites were impacted – they weren’t prepared.”

Tate describes the problem as a lack of "web provisioning." This includes server capacity and personnel tasked with maintaining the site. If traffic exceeds a website’s resources, it will slow down or fail, as many users experienced that day. Despite the problems, 60,000 people managed to secure appointments on Thursday, Tate points out.

But Tate agrees mistakes were made.

“I think there should have been things in place that would have made for a better, smoother experience for the people of Massachusetts (…) We’re accepting responsibility for the challenges people experienced with our site,” she says. “This is an opportunity to apologize and do better. We just know that as we move froward, we are now ready for whatever comes our way because we’ve had this happen, and we won’t have it happen again.”

As for the design flaws with PrepMod, Tate says that’s less about the program and more about how Massachusetts customized it. Each state that uses PrepMod can tailor the interface to its needs.

“The system is highly configurable. The system does what the state asks it to do, so we have versions that handle appointments differently than your state does,” Tate says.

California also uses PrepMod, and the interface in the Golden State is far different from Massachusetts. It also allows residents to select an appointment time before filling in pages of personal information rather than the other way around, one criticism that Massachusetts residents had, especially as the website failed, and they found themselves entering that information over and over and over again.

But California has also encountered some problems with the service. Health officials there griped that the program slowed down appointment scheduling. In Pennsylvania, some health bureaus stopped using PrepMod after finding software glitches like duplicated entries for people who had only been vaccinated once.

And customization shouldn't come at the expense of robustness, says Quickbase's Jay Jamison. Designing software that is easy to use and able to handle failures– like saving work when something goes down – are things that tech companies can do.

"It has to do with people who are experts on the processes talking through how they want [the software] to flow," he says.

It was a bad user experience, which we own. We need to do better.

Governor Charlie Baker

After the PrepMod website and, independently, the state’s vaccine finder website went down, Gov. Baker was livid. He vowed to improve the system and make sure that such crashes and slowdowns didn’t happen again.

“It was a bad user experience, which we own,” he said this week. “We need to do better and people have been working to improve both the user experience, but also the capacity of the system to manage millions of people at one time.”

Baker has announced 11 regional vaccine sites that will have improved server capacity and a digital waiting room to allow users to queue up for appointments during times of high traffic. Maryland Partnership for Prevention’s Tate says she’s also taken steps to improve the robustness of PrepMod and the state's vaccine signup website.

As the rollout continues, increasingly younger and computer-savvy populations will begin jostling for appointment slots. They will test whether the improvements were sufficient, or whether the problems that plagued the state's websites last week will only return with greater intensity.