During the UN General Assembly week in September 2026, ECNL, Humane Intelligence PBC and the Office of the United Nations High Commissioner for Human Rights (OHCHR) hosted a roundtable on agentic AI. It brought together people from international organisations, government, industry and civil society to map the emerging risks of these systems and brainstorm how to address them. What follows is a synthesis of the ideas that came out of the session.
A fast-moving technology with an existing framework
Agentic AI is advancing faster than our ability to understand and address its risks. These systems leave less control for the people affected, especially marginalised groups, amplifying familiar harms such as discrimination and surveillance.
Agentic AI covers an enormous range, from booking a haircut to swarms of agents probing government systems. A conversation about agentic AI risk that doesn't name the use case will stay too abstract to be useful, so being precise about what a system does, and to whom, is a precondition for meaningful safeguards. Nevertheless, there exists a core shift from systems that provide information to systems that act. Agents remember more, draw inferences, pull data across services and hand tasks to other agents. This includes agentic systems that make profiles from persistent memory and execute autonomous actions. That changes the nature of the risk:
- Harms compound. They build across steps, have downstream impacts, and spread into other systems.
- Consent gets lost. Consent given at the start may not apply to future and downstream steps by the agent.
- Harms get harder to see. The people affected find it harder to notice or document harms, let alone contest them.
The UN Guiding Principles on Business and Human Rights (UNGPs) already set out what states and companies must do: the tools need to be adapted to apply the human rights framework to frontier AI. Risks should be identified through a human rights lens, combining due diligence and impact assessments with independent evaluations and benchmarks. Yet identifying risks is only the first step; it has to lead to mitigation that works in practice. Both steps should centre the people most affected, particularly marginalised groups and communities in the Global Majority. Users can already exercise some oversight, as they can read reasoning traces, ask follow-up questions, and ask an agent to explain why it acted as it did; yet this is not enough to constitute proper control and harm prevention.
Due diligence, risk mitigation and transparency must be legal requirements, with real consequences for non-compliance. Voluntary commitments did not prevent the harms already documented, from discriminatory automated decision-making to biometric mass surveillance. They cannot be enforced, and there is no basis for relying on them going forward.
The urgency is sharpened by an inverted policy landscape. AI companies increasingly set the agenda for their own governance, while human rights are often left out of the high-level forums where that agenda is shaped. Furthermore, there are hardly any resources that go into human rights due diligence, both within and outside companies.
Human rights risks
Agentic tools already operate in high-risk sectors such as policing and immigration enforcement. The most autonomous uses tend to target the people least able to push back.
- Immigration enforcement. Palantir's work with ICE combines link analysis, predictive targeting, biometric matching and case management into a single high-autonomy pipeline. It is aimed at a largely non-criminal population. 404 Media has also reported that CBP uses financial data to predictively target migrants.
- Predictive policing. Tools trained on historical arrest data flag already over-policed communities even more.
Inference combined with memory is a distinctive risk. Language models have already inferred location, income and sex from real Reddit profiles with up to 85% accuracy, and anonymisation didn't reliably prevent it. Agents go a step further and act on what they infer. A workplace agent might infer pregnancy or disability, while a financial agent might infer economic distress. Wrong inferences cause harm, too.
Memory adds to the problem. It stitches scattered details such as a job change, frequent travel, a visit to a medical provider, or daily purchases, into a profile no one consented to. Moreover, consent is weakest exactly where the risk is highest. Workers, benefit recipients, and migrants often can't meaningfully opt out, and the harms fall hardest where identity characteristics intersect, such as gender, race and ethnicity, sexual orientation, disability, age, or socio-economic status.
In the Global Majority, these harms are likely to be sharper and less visible. As documentation comes from the US and Europe, much remains unknown. Shrinking civic space, rising authoritarianism, and transnational repression raise the stakes.
The 2026 Jantar Mantar protests in Delhi are the clearest recent example. Police used AI-enabled cameras on telescopic masts and smart glasses, cross-referenced national fingerprint databases, and recorded protesters' Instagram handles. In settings like this, network mapping becomes activist mapping. Link-analysis platforms already decide which connections merit scrutiny, and a government agent can infer political affiliation and act on it.
Language gaps increase harm, as in under-resourced languages, agents can be less accurate while sounding just as confident. AI developers themselves have reported their models being misused for national and transnational repression.
Commitments for human rights protection
- The rules for this technology cannot be written by a few people in a few countries who decide which rights are expendable. Governance has to be global and participatory, with the communities most affected involved in shaping it.
- Human rights due diligence must be rigorously conducted wherever a system is built, sold or deployed, including in the Global Majority.
- Due diligence cannot be a ‘check the box’ exercise at a unique stage in development. Agentic systems change as they accumulate memory, gain tool access, and are put to new uses, so due diligence has to run through the entire lifecycle in an iterative manner.
- Some decisions should never be delegated. When an outcome affects someone's rights, a person must make the call and be accountable for it. An agent should not make that call.
- People need to know when agents are acting on them and with what effect. When the gap between what companies know and what the public knows is wide enough, people cannot even tell that their rights have been violated. Closing that gap is a precondition for any use and then remedy.
AI neutrality is a myth, so transparency is required
AI systems are not politically or socially neutral. Every output reflects choices about data, framing, and priorities. Rather than chasing a single “neutral” answer, systems should be open about how outputs are curated, and users should have tools to probe a model's assumptions. That transparency should be legally required, with meaningful access for independent researchers.
Operationalising commitments through tooling
The biggest shift called for is from high-level commitments and taxonomies to benchmarks, tools, and evaluations that actually change how models operate.
Under the UNGPs, due diligence means identifying impacts, acting on them, tracking whether responses work, and communicating about it. Evaluations can be an efficient method for gathering evidence. While developers are often best placed to run them, because they control memory, tool permissions, access between agents, and how autonomously a system can act, they are beholden to corporate interests, so independent evaluations are the only way to have unbiased results.
Test the system, not just the model. One preprint study estimates a 37% gap between lab benchmark scores and real-world performance for enterprise AI agents, and agents that succeed 60% of the time on a single run succeed only about 25% of the time when they must get the same task right repeatedly.
To count as due diligence, evaluations should be:
- Co-designed with affected communities. Including speakers of non-dominant languages and people in Global South contexts, so evaluations measure what matters rather than what is convenient.
- Grounded in local context. Impact assessments and red teaming should draw on local knowledge and lived experience, not only technical or academic expertise.
- Tied to clear red lines and risk tiers. Firm prohibitions at the top, backed by the power to stop or limit deployment (including which customers and contexts a system is sold into), with proportionate safeguards for lower-risk harms.
- Continuous. Memory and aggregation risks grow over time, so one-off pre-deployment testing is not enough.
- Publicly accessible. Open evaluation tools and results let civil society, journalists, researchers and the public scrutinise systems on their own terms.
- Independently audited as routine. AI auditing should become as standard and credible as financial auditing.
Leverage points
Regulation is not the only lever. Investors, pension fund managers, and procurement teams can embed human rights requirements through due diligence, standards, and evaluation criteria. Public procurement matters especially, because governments are major buyers and can shape what gets built. States and other public-interest stakeholders are natural partners for this work.
Looking ahead
Several questions remain open. How can safeguards stay locally relevant and community-led while still working at scale across regions? What new harms is agentic AI creating, including effects on mental health, society, democracy and civic space? And how do we translate human rights principles into technical benchmarks and metrics?
At their core, these are questions of power. The future of agentic AI should not be decided by a few people in a few countries. Transparency, public metrics and accessible evaluation tools give civil society and human rights organisations a real ability to scrutinise these systems and shape their direction.
ECNL, Humane Intelligence PBC and OHCHR look forward to continuing this work together and alongside other emerging initiatives. We invite civil society, technologists, funders, academics, investors and anyone else interested to join this broader effort.