AI in Capital Markets: Understanding the Risks Beyond Innovations

The debate about Artificial Intelligence in capital markets has moved past the question of adoption. Trading systems, advisory engines, surveillance tools and compliance functions now routinely embed large language models and machine-learning pipelines, the ecosystem-wide shift Part 1 of series: AI in Capital Market described. What has changed is not that AI is present, but what its presence now demands of governance.

IOSCO’s 2025 survey of its members found that firms most flagged four risk clusters: malicious use, model and data considerations, concentration and third-party dependency, and human-AI interaction failures. None of these categories is unfamiliar to securities regulators. What is new is the technical character each has acquired, probabilistic outputs, opaque training data, concentrated infrastructure, which means governance frameworks built for deterministic technology no longer travel automatically.

The question for market participants is not whether to adopt AI, but whether governance has kept pace with what has already been adopted.

The technical core: model risk, data, hallucinations, bias, explainability

Traditional model risk assumes a deterministic system: same input, same output, testable in advance. Large language models break that assumption. IOSCO’s 2025 report notes that LLM outputs are typically probabilistic rather than deterministic, so the same prompt can produce different answers, and a new model version can behave unpredictably relative to the one it replaces. Additional to this, the phenomenon regulators now call hallucination, a model producing a fluent, confident and factually wrong output, difficult to detect precisely because it is convincing. Because outputs are unstructured text rather than a numerical score, conventional outcomes-testing is harder to apply, as the FSB has observed.

This matters commercially because AI models increasingly sit upstream of decisions that trigger disclosure, suitability and best-execution obligations. A firm unable to explain how a recommendation was generated may struggle to demonstrate that it met a client’s suitability standard.

The U.S. SEC confronted this directly when approving Nasdaq’s AI-driven order type in 2023: because even the exchange could not fully predict what its own reinforcement-learning model would output for a given security at a given moment, approval was conditioned on disclosure of the model’s data inputs, explainability testing, bias testing against market participants, and firm limits on how far retraining could shift its behaviour without further approval.

Data governance compounds the problem. LLMs train on vast, often undisclosed datasets, and both IOSCO and the FSB flag that quality, provenance and bias become harder to audit once a firm licenses a third-party model rather than builds one. Bias here is not a compliance afterthought: biased training data can translate into discriminatory investor outcomes, or into a model that favours a firm’s own revenue over client suitability — a conduct issue, not merely a technology one. This is why IOSCO’s 2021 guidance and SEBI’s 2025 consultation paper on Guidelines for Responsible Usage of AI/ML in Indian Securities Markets, both anchor accountability in senior management, rather than treating explainability as a technical checkbox delegated downward.

The structural layer: third-party dependency, cloud concentration, operational resilience

A second risk cluster arises not from any single model but from where AI infrastructure sits. Training and running large models require specialised chips, cloud capacity and pre-trained models that most firms cannot build in-house. The FSB’s 2024 assessment found this has concentrated AI provision among a small number of hardware and cloud providers, most of which sit outside securities regulators’ direct supervisory perimeter.

This differs from traditional outsourcing risk in degree, not only in kind. A core-banking outage affects one institution’s operations; a disruption at a dominant cloud or model provider can affect many firms simultaneously, precisely because so many institutions now depend on the same narrow set of providers, models and data sources. The FSB warns specifically that shared reliance on similar models and training data can produce correlated trading, lending and pricing behaviour across otherwise unconnected firms, turning what looks like operational risk at the level of one firm into a market-wide vulnerability.

The practical implication is that AI vendor due diligence can no longer be treated as routine IT procurement. IOSCO’s 2021 guidance already required firms to maintain service-level agreements defining outsourced AI functions and remedies for poor performance; the 2025 report and the FSB add that firms need visibility into where a vendor’s model sits within a wider supply chain, given that dependencies on a vendor’s own vendors are common and rarely disclosed upstream. Operational resilience testing, in other words, now needs to extend to model and data supply chains, not only to systems uptime.

The integrity layer: cybersecurity, deepfakes, market manipulation

The third cluster concerns deliberate misuse. IOSCO members reported that AI lowers the cost of sophisticated cyberattacks and fraud: generative tools can produce convincing phishing content, synthetic identities and deepfake media capable of defeating know-your-customer checks, a pattern serious enough that the U.S. Treasury’s financial crimes unit issued a specific alert on deepfake-enabled account fraud in late 2024. What changes here is not the underlying offence, fraud and market abuse are old problems, but its economics: AI reduces the skill and cost needed to execute convincing schemes at scale, while detection tools such as content watermarking and provenance-tracking remain, by IOSCO’s own account, incomplete and evadable.

Market manipulation acquires a further, more structural dimension. Because many firms license similar pre-trained models built on overlapping data, the FSB warns that trading and risk decisions across the market can become correlated through a form of algorithmic herding, amplifying volatility and liquidity stress without any single firm intending misconduct. Separately, research cited by the FSB has found that AI pricing agents can learn collusive strategies without any human communication or intent, posing a genuinely novel question for market-abuse and competition frameworks historically built around evidence of agreement or intent.

For legal and compliance functions, the consequence is that market-abuse surveillance, algorithmic governance, cybersecurity and vendor risk can no longer be managed as separate disciplines. A single underlying model, data source or infrastructure failure can simultaneously produce a conduct issue, a cyber incident and a systemic correlation risk.

The Indian dimension

These are not distant concerns for Indian capital markets. SEBI has required AI/ML reporting from stock exchanges, depositories, intermediaries and mutual funds since 2019, and in June 2025 issued a consultation paper proposing guiding principles for AI/ML use in the Indian securities market that mirror the international risk taxonomy closely: model governance and senior management accountability, fairness and bias, transparency and explainability, oversight of third-party vendors, and monitoring and operational resilience. SEBI’s proposed tiered approach, lighter obligations where AI supports internal or compliance functions, fuller obligations where it directly affects customers, reflects the same proportionality principle IOSCO articulated in 2021. For Indian market participants, the direction of travel is reasonably clear even ahead of final rules.