Increasingly artificial intelligence (AI) models are being used as expert systems to augment clinical decision-making. At present, at least officially, AI is not being used to replace professionals, but rather solely to advise and augment their decisions. A human is always in the loop.
Even still, it is critical to understand the nature of the advice being given. AI models are not independent moral agents. They have not conscience, no moral judgment, and no feelings. But large language models (LLMS) do a good job of mimicking human language and through that, emotions. They do engage in their own type of reasoning. The temptation is to treat their reasoning as if it were human, because they communicate like humans, but in reality it is not.
A recent study highlights some of these differences, specifically with moral reasoning. The authors created a case of two patients who both require a kidney transplant, while only one kidney is available. This exact paradigm has already been studied with human non-expert subjects, so there is a database of responses from people for comparison. They varied the details of the two cases to see how they affected the decisions of several LLMs in terms of which patient gets the kidney. In addition, the study design measure confidence in the decision, partly by offering a random fair alternative, like flipping a coin.
Several features stood out to the researchers. The first is that the moral decisions of the AIs did not align well with the decisions of human subjects. Second, there was a difference in process. Humans tended to weigh all of the variables in their final decision. The AI’s, by contrast, not only weighed the variables differently, they tended to fixate on one variable (such as the extent of current alcohol use).
Further, the AIs displayed a much higher degree of confidence in their decision than the humans. Specifically, they were much less likely to replace their decision with a random alternative, like coin-flipping.
In short, the AIs made overly simplistic and overly confident decisions that did not align with human values. However, the authors also state that “low-rank supervised fine-tuning” of the AI models did improve their performance, aligning them better with human responses and calibrating their confidence levels. Supervised fine-tuning means that the models were trained on specific examples provided by the researchers – essentially giving them the human responses so that they respond more like the humans. Low-rank adaptation means that a few specific parameters were tweaked, without changing the majority of the trained model.
Again – no one should be relying on LLMs for moral decision-making. But increasingly they are being incorporated into expert decision-making, even though the final decision still lies with the human experts. LLM responses can still have a significant influence on human decisions. Their deterministic over-confidence is easy to confuse for reliability. And they communicate like humans, which makes it easy to respond to them as if they have human-level intelligence, when they don’t. Further, they do not inherently reflect human morals.
It is critical, therefore, for anyone using LLMs in this capacity to understand their nature and their limitations. Even if you do understand this, it is easy to be overly influenced by their output, which can falsely seem to be objective and definitive – even when they are completely wrong, or simply making judgments. A great follow up study would be to see how human expert decision-making is influenced by receiving input from an LLM – does it help or hurt? Even though the scope of this study was narrow, the implications are far-ranging, not only for all clinical decision-making, but for many areas of expert and non-expert use of AI to augment decisions.
There are several lessons to take away from all this. First, do not confuse the comfortable human-like style of LLMs for actual human-type or level intelligence. Yes, they can be powerful, and do have their own logic, intelligence, and reasoning. It’s just not human, and we must actively resist the urge to anthropomorphize them. Further, it is helpful to understand their specific weaknesses. They tend to act overconfident, to state as fact whatever their output is, and to be flattering and friendly. These features are easy to confuse for reliability, but we know that LLMs are not always reliable, and can even hallucinate.
Second, LLMs are not moral agents, in any sense. Whatever decisions you make with AI support are still 100% your decisions. AI’s cannot actually make moral judgments based on any inherent values, they just reflect their training data. They can be made, however, to align their output with certain values if you specifically control their training data and tweak their parameters. It is important to recognize, however, that this would just simulate moral values, and that simulation can break down unpredictably.
Going forward the question is – can we make AIs that have a functional equivalent of moral values? This is more than just tweaking their training data. We can make AIs so that they reflect human values in specific ways, such as not encouraging teenagers to kill themselves, or to reflect human choices in allocating scarce medical resources. But these are specific fixes, not global moral reasoning. We need to design global moral filters for AI.
Efforts to develop such moral filters are already underway. Anthropic, for example, has developed a “constitutional AI“, which is essentially a list of behavioral rules that their AIs must follow. Perhaps more capable is the Delphi project, which, which is an AI trained to predict the moral judgments of humans. The idea is to have Delphi be an AI moral filter on other AI systems. This does produce useful results, but there are still limitations – “Delphi has limited cultural awareness and is susceptible to pervasive biases.”
The question of AIs and moral judgment is a rapidly moving target, as AI development is proceeding quickly. This is partly why there is an increasing call to slow down the pace of AI development, meaning that we should not be more focused on making them more and more powerful than making them more moral and safe. We may want to shift resources towards questions such as – how do we make our AI models reliably reflect human morality – rather than just making them more capable and ubiquitous.
