When AI Gives Moral Advice, It Often Misses What Makes Us Human

As generative AI tools become everyday companions, people are increasingly turning to them for advice on deeply personal questions: Should I confront a friend? Am I being unfair to my partner? Was I wrong to put my children first?

These are not technical questions. They are moral ones. And while large language models (LLMs) such as ChatGPT have become remarkably capable conversationalists, a new study suggests they still struggle with one of the most human aspects of decision-making: navigating moral disagreement.

A paper presented at the 19th Conference of the European Chapter of the Association for Computational Linguistics finds that AI systems tend to reproduce majority opinions while overlooking the diversity of values that humans bring to difficult ethical situations. The researchers call this phenomenon the “pluralistic moral gap.”

Can AI understand moral complexity?

The study was conducted by Debora Nozza, Dirk Hovy (both of Bocconi University Department of Computing Sciences, Bocconi Institute for Data Science and Analytics), Giuseppe Russo (EPF Lausanne), and Paul Röttger (University of Oxford).

Their starting point is a simple observation: many moral dilemmas do not have a single correct answer, and instead invite a spectrum of perspectives. In everyday life, people often disagree—sometimes profoundly—about what constitutes the right thing to do. Yet most evaluations of AI moral reasoning focus on whether a model reaches the “correct” answer, typically defined by a majority opinion.

The researchers argue that this misses the point. Real moral reasoning is pluralistic. It involves competing values, different life experiences, and legitimate disagreement.

“Moral dilemmas rarely have a single correct answer, and instead invite a spectrum of perspectives.”

This observation is more consequential than it may appear. Most AI evaluations assume there is a single correct answer against which a model can be measured. The authors challenge that assumption. In many everyday disputes—between partners, relatives, colleagues, or friends—the interesting question is not whether the machine identifies the majority view, but whether it can represent the range of reasonable positions that humans actually hold.

Building one of the largest datasets of real-world moral dilemmas

To investigate how closely AI mirrors human moral judgment, the team created a dataset containing 1,618 real-world moral dilemmas and more than 51,000 human judgments, collected from Reddit’s popular “Am I The Asshole?” community. Each dilemma includes both a verdict and a written explanation of the reasoning behind it.

Rather than reducing responses to a simple yes-or-no label, the researchers preserved the full distribution of opinions. This allowed them to compare how humans disagree with how AI systems disagree—or fail to. The result is one of the most detailed examinations yet of how language models behave when faced with messy, ambiguous ethical questions.

AI agrees with us—until things get complicated

The findings reveal that when humans largely agree about a moral issue, AI systems perform well. Models such as GPT, Claude, Llama, Mistral and Qwen often reach conclusions similar to those of most people. But when disagreement increases, AI alignment deteriorates rapidly.

The researchers note that this is precisely where moral advice matters most:

“It is presumably in this grey area where people depend most on moral advice.”

In other words, AI is strongest when the answer is obvious—and weakest when the question is genuinely difficult.

The real problem is not judgment—it is values

The most important discovery emerged when researchers looked beyond final decisions and examined the values used to justify them. Using a taxonomy of 60 moral values, the team analyzed thousands of human and AI explanations.

At first glance, humans and AI seemed surprisingly similar. Both frequently referenced concepts such as autonomy, care, compassion, respect and well-being. But a deeper analysis revealed a crucial difference: humans draw on a wide variety of values depending on context. AI systems, by contrast, repeatedly rely on a small set of dominant principles.

• The top ten values account for 81.6% of all value references in AI-generated explanations.
• The same ten values account for only 35.2% of human explanations.

Humans were significantly more likely to invoke values such as inclusivity, communication, self-care, emotional sensitivity and child welfare—considerations that often emerge in nuanced interpersonal situations. This led the authors to identify what they call the pluralistic moral gap:

“A systematic tendency of LLMs to rely on a narrower set of moral values that closely resemble the majority judgments.”

The Dynamic Moral Profiling

The researchers did not stop at diagnosing the problem. They developed a technique called Dynamic Moral Profiling (DMP), designed to expose AI systems to a broader range of human value perspectives before generating an answer.

Instead of prompting a model to reason from its default perspective, DMP exposes it to value profiles derived from human responses to similar types of dilemmas. The method does not tell the model which conclusion to reach; rather, it encourages consideration of a broader range of human values before generating an answer. The result is a 64.3% improvement in alignment with human judgments and a marked increase in value diversity.

Perhaps most importantly, the approach worked not only on the original dataset but also on entirely new dilemmas, suggesting that the method generalizes beyond the training examples.

The risks of oversimplification

The implications extend far beyond academic research. Millions of people already consult AI systems for relationship advice, workplace conflicts, family disputes and other ethically charged situations. If these systems consistently gravitate toward majority views, they risk overlooking minority perspectives that may be equally valid and important.

The authors are careful not to conclude that AI should never provide moral advice. Instead, they argue that current models tend to simplify ethical complexity.

Their conclusion is particularly relevant as AI becomes more integrated into everyday decision-making:

“Exposure to pluralistic perspectives is not just beneficial but essential for supporting thoughtful, context-sensitive decision-making.”

The authors suggest that the future of trustworthy AI may not depend on finding a single “correct” moral framework. Instead, it may require systems capable of reflecting the richness, diversity and sometimes uncomfortable disagreements that characterize human morality itself.