A Virtuous AI is an Existential Risk

תקציר מקורי באנגליתarXiv:2606.13739v2 Announce Type: replace-cross Abstract: This paper examines trade-offs between AI safety and well-being relative to (i) one of the most promising methods for finetuning super-capable AIs, 'Constitutional AI', and (ii) one of the most influential approaches to understanding complex ethical decision making and the conditions for the well-being of rational agents, 'Virtue Ethics'. We finetune various models using a 'Virtuous agent' constitution, a 'Subordinate agent' constitution, and a 'Generic agent' constitution, and evaluate them on 'general safety' (toxic behaviors, misinformation, etc.) and also on their willingness to endorse and act on a wide-range of behaviors that, if adopted by a super-powerful AI, would significantly increase the level of existential risk for hum