In their last year of medical school, med students get to look forward to a once in a lifetime holiday that’s a mix between Christmas and Tax Day. Match Day is the day when medical students find out which post-graduate residency training program they will be attending for the next couple of years. The National Residency Match Program (NRMP) was created in 1952 in response to an intern appointment process that become so chaotic, students were expected to accept internship offers over the phone as soon as they received them. With the Matching process, applicants apply to programs, typically through the Association of American Medical Colleges (AAMC)’s Electronic Residency Application Service (ERAS) software. Programs interview applicants, and both applicants and programs send rank lists to the NRMP where they are fed into an algorithm that matches applicants with programs based on their preferences. The result is that both parties get a better outcome than if residency followed a traditional hiring process. But the Match, like all system, has its flaws.
Like most things in our lives, the Match was not unaffected by the COVID-19 pandemic. Programs had to switch to virtual interviews, which had a large impact on Match outcomes. The Match rate for graduating seniors at American MD programs has remained 94% for a long time…until this year where the Match rate declined to 92.7%, which is the lowest Match rate for US seniors since 1997 [PDF]. While it may not seem like such a big deal, a 1-1.5% decrease kind of is when we’re talking about matching algorithms. But my main concern isn’t the Match rate, it’s the Match results. The COVID-19 pandemic didn’t cause the inefficiencies and inequities of the Match, it exposed longstanding ones that were already there.
The Match has become increasingly inefficient over the years, to the point where I would argue that it’s no longer achieving its intended goals. Applicants are spending more money with each passing year, and getting worse results. In 2010, 56% of Matched applicants got their top ranked program. But in 2020, only 49.2% Matched their first ranked program. Program directors are increasingly experiencing applicants that cancel interviews at inopportune times, and are flooded with applications. The Match is increasingly a topic of discussion in medical journals because it seems program directors aren’t happy. I’d argue that neither medical students nor program directors are being served by the Match in its current form. When neither of your system’s stakeholder groups are being served by your system, you don’t need to wait for the x-rays to come back. The residency Match is broken and I want to talk about why.
Application fever
Every major problem inherent to the Match can be tied to a simple phenomenon: application fever. Application fever is the result of a positive feedback loop that has led to an increase in the number of programs applicants every year. In 2007, the average American senior applied to 30 residency programs in ERAS. In 2018, they applied to 60 programs. It’s important to point out at this point: applying to more programs isn’t free. Applicants pay money for every program they apply to and there is a non-linear increase in fees. Applying to 30 programs in ERAS costs $419 while applying to 60 costs $1,200.
Obviously, applicants don’t want to spend all of this money. So, why are they doing it? Well…they have to, because that’s what every other applicant is doing. Applicants need to attend a certain number of interviews to essentially guarantee their chances of Matching. It’s in every applicant’s interest to get as many interviews as possible. The limiting factor for programs on how many applicants they interview is how many they need to before it becomes a waste of time. That number is basically constant for programs based on the number of positions they have. The only limit for applicants on how many programs they apply to is how much money they have, and how many programs there are. This mismatch in limits is what leads to application fever – applicants have incentive and the ability to apply to as many programs as they can afford, while programs have no way to limit applicants or increase the number of available positions.
Since the number of interview positions is constant, that means that the probability of receiving an interview invitation decreases as the number of applications to that program increases. So, the number of programs candidates have to apply to has to be based upon the probability that any given application will lead to an interview. A good way to figure out that number is to look at the average number of programs the average Matched applicant applied to the past cycle. But if you aren’t so sure about your chances, applying to more programs is the easiest way to improve your odds. When you’re $1,000 in, what’s $200 more? Applying to fewer programs is only a viable option if you are absolutely sure of your chances. Other than that, applying to fewer programs than the average means you risk going unmatched. The result is a residency application process that doesn’t work for anyone.
Program directors are flooded with applications – the average orthopedic surgery program received 150 applications for every open position in 2020. Program directors can’t review all of those applications by hand, at least not thoroughly, so they come up with a standardized process to filter them. The problem is that when program directors use a standardized process for filtering applications, there’s going to be a significant degree in overlap in their filtering criteria. When every applicant is applying to more than half of all orthopedic surgery programs, there’s going to be an overlap in who gets through, so program directors are increasingly selecting for the same small group of top applicants. In the 2016 otolaryngology Match, just 26% of applicants accounted for half of all possible interview positions. While programs interview the same number of applicants every year, they send an increasing number of interview invitations annually. That means that more interview invitations get declined or there are a lot of no-shows on interview day.
With virtual interviews, the cost of attending interviews is much lower. There’s no disadvantage for highly competitive applicants to attend every interview they’re invited to. That’s ultimately what lead to the reduced Match rate this year. I was actually surprised the Match rate wasn’t lower. So will things go back to normal after the pandemic? Not necessarily, and that question assumes that everything was fine before. I just don’t think that’s the case.
This ain’t a scene, it’s an arms race
While it can seem like a small issue, the effects of application fever are far-reaching. If you look at any big problem in medical education, that issue is often tied to application fever.
Let’s start with how application fever affects the environment of medical school itself. The residency Match has become increasingly competitive over the past two decades. When the Match is more competitive, medical education as a whole becomes more competitive. As programs move to more quick metrics to filter applications, applicants start trying harder to get higher board scores, more publications, and other things to stand out. In 2007 the average medical student at a US medical school who Matched had a USMLE Step 1 Score of 221, a USMLE Step 2 Score of 226, two research experiences, and two abstracts, presentations or publications. In 2018, the average medical student at a US MD medical school who Matched had a USMLE Step 1 score of 230, a USMLE Step 2 Score of 246, about three research experiences, and four abstracts, presentations or publications. And it doesn’t stop there – medical students today are getting more graduate degrees, and going to more interviews than they did in the past.
There is serious competition for research opportunities in medical school, because having research publications will make them more competitive. It also means students will take any opportunity they can get, even when those opportunities are terrible. A perfect example is MedBikini kerfuffle on social media. Back in August the Journal of Vascular Surgery published a paper titled “Prevalence of unprofessional social media content among young vascular surgeons”, which has since been retracted. The study was taken down following social media backlash, and the study just being really, really, dumb. However, other medical students on Twitter felt the need to harass one of the med students who was second author on the study, causing him to delete his account. I would think they would feel some sort of empathy for their fellow med student because the only reason he contributed to that paper was to get publications to put on his vascular surgery application.
The high costs of applying to competitive specialties such as dermatology is viewed as a barrier to entry for applicants from low-income and minority backgrounds. It’s not because people from underrepresented backgrounds aren’t qualified or interested – it’s because they’re playing a pay-to-win game in a country with an inverse relationship between the melanin in your skin and the money in your pocket. It’s no coincidence that the medical specialties with the highest number of applications per applicant are the least diverse in medicine. This highly competitive residency selection also creates a toxic environment in medical education. The increased focus on Step 1 Scores created what some students describe as:
[A] tiered system of “worthiness” among students, which residency program directors use as part of their selection process. Those with high scores are steered toward “competitive” specialties, whereas those who do not are restricted to “noncompetitive” ones.
It was announced recently by the National Board of Medical Examiners that the USMLE Step 1 would be converted to a Pass/Fail scoring system, but it is likely this will just turn Step 2 CK scores into a filter metric. I also expect this is shift in focus is going to negatively affect osteopathic medical students who previously would only take Step 1 in addition to the COMLEX series of exams.
While I could go on, the point is the problems with the Match don’t happen in isolation. It’s about the hundred other problems the Match causes or makes worse. The question isn’t “should the Match be changed?” but rather how it should change. It turns out there is an easy fix.
Patching the match
Application fever is the lead in medical education’s water supply. I don’t have time to cover every single issue it causes, so I will refer you to Dr. Bryan Carmody’s blog which covers the state of the Match in much more detail if you’re interested. So, what can be done to fix the Match for program directors and students, and what obstacles stand in the way?
It’s always going to be within applicants’ interests to apply to as many programs as possible if they want to maximize their odds of Matching. But when that comes at the cost of programs not being able to tell who is actually interested in their program, it leads to the process not working for anyone. Any solution to application fever has to limit or increase the cost to apply to programs. While some professional societies have tried preference signaling programs, it’s the equivalent of putting a Band-Aid on a bullet wound.
Application caps are the best solution to every problem with the Match because they address application fever directly. Application caps also make the Match fairer and more predictable. Under application caps, applicants can maximize their chances of Matching by simply applying to the programs they’re interested in and competitive for. Applicants would have a much higher chance at receiving an interview at their preferred programs, minimizing the cost of applying. In addition, the ERAS fee structure should be changed to a one-time cost to discourage applicants from applying to an excessive number of programs.
Application caps are controversial on places like StudentDoctorNetwork and Reddit, but application limits would benefit the vast majority of applicants and residency programs. Those same SDN nay-sayers also don’t like it when programs start requiring things like CASPer as part of their application process as a means to have more quick data points to filter applications. I also don’t want residency programs do business with Altus Assessments, and wrote an entire piece about why they shouldn’t. The only data I could find showed that 60% of surveyed otolaryngology applicants would be in-favor of some form of application limit. But there are an increasing number of articles in medical journals arguing for residency application caps.
So, what’s stopping application caps from happening? Well, many people are against application caps to begin with. But more importantly even if the majority of the medical community were in favor of application caps, the organization with the power to make that happen really, really doesn’t want that to happen. The AAMC brought in $94 million in 2019 from ERAS fee revenue alone. If you’re wondering if even a fraction of the cost goes into the operating costs of ERAS, just know that $94 million is enough to cover the entire production cost of The Witcher 3: Wild Hunt. You could use the remaining 13 million dollars increase the AAMC’s IT budget by 3 million. Now sure, you might say it’s more complicated than that. Is it though? ERAS is not providing value to its stakeholders anywhere near in-proportion to its revenue. Applicants pay ERAS enough money that the AAMC could provision them their own personal ERAS server. Yet this year the ERAS portal crashed during the start of the first Supplemental Offer and Acceptance Program (SOAP) cycle. The pricing structure of ERAS is predatory, and that in itself warrants more criticism from the medical community.
So, let’s say all of the advocacy of the world wouldn’t get the AAMC to implement application caps. While program directors have their problems with ERAS and the Match, pulling out still might not be worth it for them. Does that mean they just have to accept the process as it is? There’s no action they could take to achieve the outcome they want? Of course not. Program directors have all the power with regards to how their residency selection process works for their program. If they want to pull out of the Match, they can pull out of the Match. But if program directors want application caps, their only option is to hope the AAMC puts application caps in ERAS? Of course not. What if I told you there was a way to implement application caps in ERAS without the approval of the AAMC, so long as enough programs agreed?
To implement application caps, all you would have to do would be to create a website where programs that want application caps would sign-up. They would make it as clear as possible that they won’t consider applications unless an application token has been sent through the website. Applicants would sign up to the website and be given a limited number of application tokens. When it’s time to send their applications, applicants send a token to the program through the third-party website, then send the program an ERAS application like they would normally. Then the Match would proceed normally.
Work would need to be done to determine the ideal application limit based on the number of sign-ups from programs and applicants. You would need to do some simple tooling to remove applications from program director’s portals for applicants who haven’t sent the program tokens. But after that, application caps would effectively be implemented for programs as a residency application process with application caps. The only obstacle would be getting enough programs to participate for this option to have a functional economy. It wouldn’t affect the general rhythm of the application process in a substantial way other than clicking a few buttons for applicants, a far less significant burden than what programs and applicants are dealing with now.
A question from a program director might be: “How do you know this will work?” There are no certainties in life, but I’m as sure as I can be. The only major obstacle outside of getting enough programs to participate would be not scuffing the technical execution. So long as there is a large enough pool of programs, the payoff would be huge. Application pressure would be reduced by 50-75%. Program directors could be sure that the reduced number of applicants they do receive are actually interested and would show up for interviews. Applicants would be heavily incentivized to use the website because it’s the only way to have their application considered by participating programs. It’s also worth the little extra effort for applicants because their odds of getting an interview invite at participating programs are much higher. The system gives everyone what they want, and it could probably be a free service developed using open-source tools.
Another reasonable question from program directors might be “What if things go wrong?” Well, the first thing to understand is that even the most catastrophic outcome you could imagine isn’t that bad. Let’s say for the sake of argument all of your slots go unmatched. You could still use the SOAP program to fill all of your slots; there are still more residency applicants than residency positions. Even applicants who don’t Match are highly qualified, and you’ll probably perfectly happy with them. That’s the worst possible outcome, and literally isn’t possible unless you don’t submit your rank list or something. But let’s look at some realistic pitfalls and how they could be avoided.
Let’s say some programs don’t have enough applicants to be able to Match all of their slots. There are lots of ways to fix this issue after it’s been raised. Those programs could be put into a special pool and tokens could be granted to applicants to apply within the special pool. The program could simply drop the token requirement and get the word out to applicants. Finally, even if the program had unmatched slots, which still happens under the current system, they could still SOAP their unmatched slots.
The biggest issue that could happen early on is with a small number of early adopter programs and a huge number of applicants that want to apply to programs such that the token economy wouldn’t work. The only thing you could do is run the system sub-optimally by being overly stringent with tokens or overly generous. But so long as there are a sufficient number of programs that want to participate to have a functional economy, this shouldn’t happen.
On the technical side, nothing about this is complicated. You’re basically just storing a list of participating programs, and who sent tokens to what programs. The only hard part is calculating what the ideal cap should be based upon the number of programs and applicants. A lot of that is going to come down to community feedback. It’s not a particularly sophisticated solution; the hard part is convincing program directors that the results would be worth it, and actually doing it.
Conclusion: Status calamitous
I’ve been wanting to write this piece for about three years now. As a software engineer, I’ve always found the problems with the residency Match incredibly interesting. The reason why I find the Match interesting…is because it’s hilarious. The Match is a brilliant idea, in theory. In practice applicants are spending $1,100 for API requests. Lifeguard certification is about to become a graduation requirement for orthopedic surgery programs because residents need to know how to rescue program directors from drowning in ERAS applications. The fee structure of ERAS is so predatory it makes Ticketmaster look like a charity. I’ve tried to think of policy changes that would make the Match substantially worse, but so far I can’t. It’s patently absurd. How is the Match like this? Do people actually like it this way?
As software engineer, sure, the Match is hilarious. But I also find the issues that come from the Match deeply concerning. The system isn’t efficiently achieving its intended goals. I think it’s safe to say that everyone is being done dirty by the Match. It’s within everyone’s interest to advocate for change. But I don’t think any progress is being made here. In 2016, the AAMC sent program directors a survey on their top three pain points. The large volume of applications, lack of information about which applicants were in genuinely interested in their program, and applicants cancelling interviews were some of the biggest complaints. To those program directors: has the AAMC done anything in the past 5 years to address your concerns? I’m going to guess the answer is no. They have a growing number of reasons not to each passing year. Things aren’t going to get better on their own, and I don’t think the AAMC is on your side.
I think application caps are the only logical solution to most of the problems people have with the residency Match. While there are a lot of ways the Match could be changed, application caps are the only solution that fixes the problems caused by application fever without introducing new problems or significantly changing the flow of the Match. At the Council of Orthopaedic Residency Directors spring conference this year, one of the main topics on the agenda was “What should we do differently?”, where application caps were mentioned. Unfortunately, after mentioning application caps, they stated that application caps were not within their control. The AAMC won’t implement application caps which means a 3rd party service for application limits is really the only way forward.
The good news is that the implementation is so easy that a random software engineer could probably develop such a service on their own. It also seems that program directors have a random software engineer in their corner. While I won’t make any promises, developing an open-source application limit service is something I might be interested in. The only thing I need to know is if that’s something that programs are interested in even just talking about. I’m not asking anyone to commit to anything, just testing the waters. I don’t want to make money off this, I’d like this to be a free service. Even talking about application caps as a possible solution if programs want them would be a huge step forward.
To program directors who might be reading this: I’m not trying to tell you what to do or how you should feel about the Match. I’m just giving my take based on what I’ve read based on the data and what program directors and applicants have been saying. The power to shape the residency selection process has always rested with individual program directors doing what they think is best. So, if you’re a program director who isn’t happy with the way things are going, you have a lot of power to change the outcome. Even just saying that you’re unhappy with aspects of the Match is doing something. But choosing to do nothing is still a choice, and I’d argue it’s the wrong choice.
The biggest obstacle to changing the Match is the belief that the Match can’t be changed. Proposed changes to the Match have always contentious and people are always going to have different ideas about what an ideal residency selection process looks like. If you aren’t happy with the way things are going, just know that things aren’t going to get better on their own. If they do improve, it will only be by comparison. From where I’m sitting, things can’t get much worse. The bar for improvement is a tripping hazard.