The AISG AI Research Symposium was Produced by the AI Governance Pillar at AI Singapore, now part of the NUS Artificial Intelligence Institute

Introduction

At the AI Research Symposium on 27 May 2025, as part of the 2025 ATxSummit Village events, the AI Governance team from AI Singapore brought together academia, government, and industry to discuss the governance of AI agents. The discussion spanned across two sessions, the first on “Introducing AI Agents”, in which panellists focused on identifying the challenges of real-world deployment, where current research on agentic AI systems is headed, and some possible directions to govern AI agents. The second session on “Regulatory and Ethical Questions”, discussed the moral underpinnings of agentic AI systems, as well as required shifts in public policy and international coordination approaches to govern such AI agents.

Session 1: Introducing AI Agents

Moderator: Prof Jungpil Hahn, Deputy Director of AI Governance, AI Singapore

Panellists:

• Ms Eunice Huang, Head of APAC Trade Policy, Google
• Prof Mohan Kankanhalli, Deputy Executive Chairman, AI Singapore and Director, NUS AI Institute
• Dr Jasmine Begum, Director of Legal, Corporate & Government Affairs, Microsoft ASEAN and New Markets

Agentic AI systems are gaining traction as tools for reducing friction in daily life by enabling the delegation of complex, multi-step tasks to autonomous digital agents. These systems are underpinned by reasoning, planning, and memory, allowing them to function beyond the reactive limits of chat-based models. For instance, they can complete end-to-end processes like buying concert tickets with minimal user supervision. However, the absence of a shared industry definition of what constitutes an “agent” remains a foundational gap. Without this consensus, coordinated research, interoperable standards, and public understanding of the technology are undermined.

At the same time, as shared by Prof Mohan Kankanhalli, agentic AI presents a suite of technical and research challenges. Chief among them is value alignment: human preferences and values are difficult to encode, and while developers rely on preference datasets to fine tune models to respond safely to sensitive prompts, these datasets will always be incomplete and are highly context-dependent. This leads to ongoing risk that agents may generate harmful or inappropriate outputs. Hallucination, the generation of false or nonsensical content by LLMs, is another endemic issue. While mitigations like guardrails or delegating specific tasks to more reliable tools exist (such as an agent using a calculator), they cannot fully eliminate the problem. Additionally, adversarial robustness is a growing concern, particularly the susceptibility of agents to manipulation via social engineering tactics that mirror human coercion. These vulnerabilities extend further when agents operate in multi-agent ecosystems. While individual actions may be benign, their collective behavior can lead to emergent, unanticipated risks. The lack of formal guarantees around the safety of composite actions in these systems underscores the fragility of current safety assurances. A further, underexplored risk arises when agents begin to interface with physical systems such as controlling doors, vehicles, or infrastructure where misjudgments can lead to material, real-world harm.

These growing capabilities are simultaneously catalyzing an economic transformation. Dr Jasmine Begum framed this shift through the lens of “Agents as a Service,” forecasting the emergence of new professional roles particularly in AI auditing. This allows the democratisation of auditing services which benefits MSMEs because they now can rent such services instead of buying them. The economy is expected to become increasingly reliant on agent-to-agent interaction, with humans stepping into roles centered on strategic oversight and exception handling. However, this transition invites deep concerns around job displacement, especially as the pace of technological change far outstrips past industrial revolutions, leaving little time for large-scale workforce reskilling. Questions of economic benefit distribution sometimes referred to as “wallet sharing”, as well as unresolved questions of accountability in the event of agentic error, compound the stakes.

To address these challenges, both companies and researchers have proposed a range of governance approaches. Ms Eunice Huang emphasised that Google adopts user-centric design principles by clearly defining agent scopes, ensuring that agents seek explicit permission before accessing personal data or performing sensitive tasks such as payments. This “control-by-design” ethos aims to balance autonomy with trust, though it can lead to usability issues if overly intrusive. Practical constraints like limiting an agent’s scope of action to a specific domain (e.g., email organization) and implementing incremental deployment through controlled testing serve as additional layers of control and refinement designed to build public trust by allowing Google to refine the agent’s behavior, assess its real-world utility and safety, and adapt its design based on emerging issues long before any wide scale deployment. This release strategy was used for tools like Astra and Meridian were first launched as research previews and later made available to a small group of trusted testers.

Dr Jasmine Begum stressed the need to rethink regulatory frameworks from first principles. She argues that many governments are attempting to adapt analog-era regulations to digital-era systems, resulting in mismatches that leave policy gaps and blunt oversight tools. AI policies, especially those governing agentic systems, need to be developed from the ground up. Regulation must be rethought and not simply revised so that it both supports innovation and ensures public safety and trust, especially in regulated industries like healthcare, finance, or defense, as well as government use-cases for agentic AI in citizen services. These uses raise important questions around personal data access and protection. Ultimately, there must be a close and ongoing partnership between governments, industry, and civil society that embraces a ground-up model of responsible AI governance, rather than merely adapting existing regulatory frameworks.

Finally, the panel emphasized a more philosophical and risk-oriented view of agentic AI governance. A key insight shared by Prof Mohan Kankanhalli is the recognition that not all risks can be predicted in advance. Strategies such as differentiating between high-stakes and lower-stakes applications allow for graduated oversight and more experimental deployment in less critical contexts. Yet history has shown, through analogies like 9/11, that threat models often fail to anticipate transformative adversarial events, which are also called “out-of-model attacks.” In a recent paper “Bullying the Machine,” Prof Mohan and other researchers showed that users could coerce models into providing inappropriate responses simply by mimicking coercive or emotionally manipulative behavior. This suggests that agentic systems are vulnerable not just to technical exploits, but to forms of social engineering that mirror human dynamics. This underlines the importance of realism in governance: acknowledging that no safeguard is foolproof, and that adaptive, flexible approaches must be built to contend with a landscape defined as much by uncertainty as by innovation. Governance, then, must be not only technically proficient but socially grounded, ethically aware, and politically adaptive such that it is capable of evolving alongside the agents it seeks to constrain.

Session 2: Regulatory and Ethical Questions

Moderator: Prof Simon Chesterman, Senior Director of AI Governance, AI Singapore

Panellists:

• Assoc. Prof Jennifer Ang, SUSS
• Mr Juraj Corba, Chair of AIGO OECD and Co-Chair of GPAI
• Ms Lee Wan Sie, Cluster Director, AI Governance and Safety, IMDA

The panel’s discussion on the ethics and regulation of agentic AI was wide-ranging, but its insights coalesce around several major themes. One foundational theme is the moral asymmetry between humans and AI agents. As Assoc. Prof. Jennifer Ang highlighted, true moral agency requires more than the ability to plan and act. It also demands self-awareness and the capacity to deliberate on values. These are required for an entity to be held morally responsible. Today’s AI systems, despite their sophisticated behaviors, lack these attributes. Yet we are increasingly entrusting them with tasks that carry real moral weight. This creates a troubling dynamic where morally consequential decisions are delegated to systems that cannot be held responsible while the developers and users who can are distanced from the outcomes. This is compounded by the “black box” nature of LLMs, and the diffusion of responsibility across complex technical processes. Even AI developers often lack full insight into how their contributions will be deployed, let alone the consequences. This problem extends to end-users as well, who may retain the final say over an agent’s recommendation but still feel unable to justify their acceptance or rejection of that output. The overall opacity of these systems means that transparency and traceability which are essential components of ethical accountability, are increasingly absent.

Related to this is the concern of moral disengagement and psychological distancing. As AI intermediates decision-making, humans may feel less directly involved and thus less morally engaged, which is a phenomenon similar to the distance seen in drone warfare or the classic trolley problem. Over time, such distancing risks cultivating “moral blindness,” where habitual deferral to machines dulls our ethical instincts. This cultural erosion may lead society to normalize ethical disengagement, mistaking technological functionality for moral adequacy. However, Assoc. Prof. Jennifer Ang shared a hopeful counterexample: her experience with STEM researchers who initially dismissed ethics as irrelevant but gradually came to appreciate its importance after targeted, context-specific workshops. These efforts suggest that cultivating responsible practitioners, not just responsible systems, is a promising path forward.

Another major theme revolved around the challenges and directions of regulation. Ms Lee Wan Sie emphasized that agentic AI dramatically expands the surface area of risk due to its capacity to act autonomously across broad domains. These risks include technical malfunctions, malicious misuse (e.g. scams or misinformation), emergent frontier threats (e.g. manipulation or blackmail), and systemic societal disruptions. In response, Singapore’s approach continues to follow a risk-based regulatory model, but requires increased specificity. General-purpose tools are no longer sufficient; targeted mechanisms are needed for agentic systems, including context-aware requirements for human oversight, calibrated risk assessment, and robust post-deployment monitoring. Future investments must strengthen real-world testing and sociotechnical research, with a focus on understanding the broader impacts of agentic AI systems on employment, trust, public and human values.

International coordination formed another important strand of the conversation. Mr Juraj Corba pointed out that the challenge of governing agentic AI mirrors that of nuclear nonproliferation: collective risks exist alongside strong national incentives to compete. However, AI adds a socio-technical layer of complexity. The governance challenge lies not only in the technology itself, but also in the diverse institutional, cultural, and policy environments in which it operates. Forums like OECD and GPAI aim to bridge this gap by facilitating structured global dialogue to build consensus on standards, risk categories, and governance norms. Still, the pace of development and disparities in political will pose significant obstacles. The emergence of multi-agent systems that can adapt, negotiate, or even deceive further complicates the landscape, making it harder to assign responsibility. Nevertheless, the principle that humans must remain ultimately accountable for agentic systems was reaffirmed as a non-negotiable foundation of governance.

A persistent blind spot discussed was the limited understanding of malicious AI use. As Ms Lee Wan Sie noted, most governance tools assume cooperative, well-intentioned actors, yet growing AI capabilities are increasingly accessible to bad actors whose behaviors remain largely unobserved and understudied. Regulatory mechanisms are better suited to structuring ethical practices among compliant developers and users than detecting and countering misuse. The scarcity of reliable data and poor visibility into unregulated environments makes this an urgent but under-addressed area. Addressing it will require not only technical tools, but also a strategic shift in how regulatory systems think about adversarial behavior and off-the-grid risks.

Finally, the panel tackled the cultural and institutional challenges of ethical governance. One critical point raised in the Q&A was the difficulty of translating abstract regulatory principles into practice, especially given the contextual variability of agentic AI applications. In domains like healthcare, even closely related use cases like diagnostics, summarization, and billing require different levels of oversight. Instead of rigid benchmarks, emphasis was placed on developing shared, flexible processes for validation and deployment. The concept of regulation by design also came under scrutiny. While embedding safeguards in software can enhance system security, efforts to over-regulate human behavior risk undermining human agency and creativity. The goal should be to design systems that support and extend human moral judgment, not constrain it.

Ethics education was similarly highlighted as an area needing nuance. Standardized training is insufficient in a world of diverse cultural norms and values. Effective ethics training must be responsive to context and sensitive to social and cognitive diversity. The case of caregiving robots in Singapore where design choices may conflict with values like filial piety illustrates how ethical missteps can stem from cultural disconnects rather than malicious intent. A deeper commitment to human-centered design, as opposed to merely user-centered efficiency, was seen as essential. Without this shift, Assoc. Prof Jennifer Ang warned, we risk building AI systems that may be technologically powerful but ethically hollow solutions that respond to the surface of human need while neglecting the deeper layers of meaning, dignity, and social connection.