Moral red line 06 Human good · moral agency

High-Stakes Decisions and Human Moral Agency

A patient, a family before a court, or someone seeking public assistance needs a person who can listen, explain, reconsider, and take responsibility for a consequential decision.

Can the person reviewing an AI recommendation genuinely change the outcome?

The moral boundary

Decisions involving lethal force, liberty, culpability, irreversible denial of essential care, family integrity, safeguarding, canonical authority, or spiritual discernment must remain with competent human persons and institutions. The boundary is crossed when AI becomes the de facto authority or when opacity, speed, scale, or workflow design reduces human review to approval without independent judgment. This protects conscience, due process, mercy, meaningful human control, and responsibility toward the person affected.

The pro-human direction

Use AI to improve evidence and prediction while preserving human authority, independent judgment, explanation, appeal, reconsideration, and traceable responsibility.

Arizona appeals-court ruling

Sentence vacated

On 30 September 2026, the Arizona Court of Appeals upheld the conviction in State v. Horcasitas and vacated the sentence. It found that an AI-generated statement in the deceased victim's voice contributed to fundamentally unfair sentencing.

When advice becomes authority

A patient waits to hear whether treatment will be approved. A family comes before a court. Someone seeking essential assistance is told their application has failed. Each needs a responsible person who can hear the circumstances, explain the reasons, and reconsider what happens next.

AI can organize evidence, identify patterns, and improve prediction. As institutions rely on it, the authority carried by its recommendations grows. The way evidence is presented, the time allowed for review, and the cost of disagreement can determine how much freedom a professional retains.

Manufactured expression in a real courtroom

In State v. Horcasitas, the victim’s family prepared an AI-generated video in which the deceased Christopher Pelkey appeared to speak, including words of forgiveness. The sentencing judge considered it genuine and allowed it to influence the sentence. On 30 September, Arizona’s appeals court upheld the conviction and vacated the sentence, finding fundamentally unfair sentencing. The case shows how generated personal expression can acquire authority over a consequential human judgment.1

The institution must preserve judgment

Québec’s August judicial guidelines keep decision-making human while allowing bounded preparatory uses. NHS Resolution’s September guidance addresses professional duties, liability, and the use of AI in care. California’s September healthcare and legal measures require safeguards around clinical judgment and verification of legal work; implementation follows the statutes’ timetables.234

The August UN–ICRC appeal for a binding agreement on autonomous weapons gives this duty its gravest setting. If AI selects targets, compresses time, and presents a recommendation with institutional authority, a human authorization needs conditions that preserve independent judgment and refusal.5

The same responsibility reaches safeguarding, canonical authority, and spiritual discernment. Oversight has to be organized around the person whose life is affected and the reviewer who must be able to change the outcome.

Moral foundation

Conscience, prudence, and responsibility

Conscience calls a person to judge what they ought to do and answer for it. Gaudium et Spes describes this inner responsibility; the Catechism calls prudence the virtue that discerns the true good in each circumstance and chooses the right means. Serious decisions require attention to the particular person, justice, mercy, and the possibility of reconsideration.7, 8

Magnifica Humanitas keeps technological judgment connected to conscience and responsibility. Antiqua et Nova requires identifiable human accountability, including in healthcare. A professional needs the knowledge, time, and freedom to disagree. Families and affected people need someone they can approach for reasons and a fresh hearing.9, 10

In warfare, Magnifica Humanitas calls for human control where life is at stake. Cardinal Parolin's September UN address places technological governance within a duty to serve peace. Church institutions carry parallel duties in safeguarding, canonical authority, and spiritual discernment, where trust and responsibility belong to competent people.9, 11

01

Conscience and prudence

A decision calls for judgment about the good of a particular person and responsibility for the means chosen.

02

Justice and mercy

Those affected deserve to be heard, receive reasons, and have their circumstances reconsidered.

03

Accountable care

Professionals and institutions need real power to protect the person whose life is changed by their decision.

From principle to practice

Oversight that can change the outcome

Madeleine Clare Elish's moral crumple zones describe situations in which a human operator receives blame while having little practical control. Institutions must give reviewers sufficient information, time, authority, and freedom to refuse, alongside routes for affected people to seek reconsideration and remedy. Responsibility should follow actual power across developers, deployers, and authorizing institutions.12

On 6 October, the UK government accepted the healthcare AI Commission's 44 recommendations, with staged implementation ahead. This offers an opportunity to organize regulation around patient safety, professional judgment, and accountability. Recommendations, accepted commitments, and implemented protections must remain clearly identified at each stage.13, 14

Public institutions

Define AI's advisory role, identify the decision-maker, give reasons, and provide a fresh human appeal.

Professionals

Examine original evidence and uncertainty. Preserve the freedom to delay, disagree, refuse, and stop the process.

Church institutions

Reserve safeguarding, canonical authority, pastoral care, and spiritual discernment to competent people with clear accountability.

Minimum conditions

  • Knowledge — access to evidence, uncertainty, and the system's limits
  • Time — room to examine the case and hear the person affected
  • Authority — freedom to disagree, override, refuse, or stop
  • Recourse — explanation, appeal, reconsideration, and remedy
  • Responsibility — identifiable people answerable for the outcome
Selected sources Every source is drawn from the white paper and opens the primary document or publication page where available. View sources
  1. 1
    State v. Horcasitas — 1 CA-CR 25-0191

    Arizona Court of Appeals Division One · 30 September 2026 · Judicial opinion; conviction affirmed and sentence vacated

    LegalLegal judgment / statute / filing

  2. 2
    Judicial guidelines on the use of artificial intelligence

    Court of Appeal of Québec · Adopted 31 August 2026; reported 4 September · Judicial guidelines

    LegalLegal judgment / statute / filing

  3. 3
    Guidance on scheme coverage and liability issues concerning the use of artificial intelligence

    NHS Resolution · 10 September 2026 · Professional guidance

    Government / regulatorOfficial primary source

  4. 4
    California's nation-leading AI framework just got stronger

    Office of the Governor of California · 30 September 2026 · Enacted SB 574 AB 1979 and SB 503; statutory implementation timetable

    LegalLegal judgment / statute / filing

  5. 5
    Renewed call regarding autonomous weapons

    UN Secretary-General and ICRC President · 25 August 2026 · Appeal for a legally binding instrument

    Government / regulatorOfficial primary source

  6. 6
    General audience — The Human Community

    Leo XIV · 16 September 2026 · Papal catechesis

    ChurchOfficial primary source

  7. 7
    Gaudium et Spes, §16

    Second Vatican Council · 7 December 1965 · Pastoral constitution

    ChurchOfficial primary source

  8. 8
    Catechism of the Catholic Church, §1806

    Catholic Church · 1992; definitive Latin edition 1997 · Catechism; teaching on prudence

    ChurchOfficial primary source

  9. 9
    Magnifica Humanitas, §§71–72, 102–106, 198–200

    Leo XIV · 15 May 2026 · Encyclical

    ChurchOfficial primary source

  10. 10
    Antiqua et Nova, §§39, 44, 46–47, 74

    Dicastery for the Doctrine of the Faith and Dicastery for Culture and Education · 28 January 2025 · Dicastery note

    ChurchOfficial primary source

  11. 11
    Address to the UN General Assembly

    Cardinal Pietro Parolin · 26 September 2026 · Holy See diplomatic intervention

    ChurchOfficial primary source

  12. 12
    Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction

    Madeleine Clare Elish · 2019 · Engaging Science, Technology, and Society 5: 40–60

    AcademicPeer-reviewed

  13. 13
    Government response to the National Commission's recommendations on the regulation of AI in healthcare

    UK Government / MHRA · 6 October 2026 · Recommendations accepted; staged implementation

    Government / regulatorOfficial primary source

  14. 14
    What the National Commission's recommendations mean for patients and the public

    UK Government / MHRA · 6 October 2026 · Government explanation of accepted recommendations

    Government / regulatorOfficial primary source