Supreme Court of India flagged misuse of AI citing fake judgments, informally urging Bar Council of India to form expert panel. Bench of Justice P S Narasimha and Justice Alok Aradhe sought report.
The Supreme Court of India flagged serious concerns about the increasing misuse of Artificial Intelligence (AI) tools in court-related work especially the tendency to cite judgments that do not exist, often referred to as “hallucinated” decisions. The Court informally requested that the Bar Council of India (BCI) set up an expert committee to examine the issue.
A Bench led by Justices P S Narasimha and Alok Aradhe clarified that it was not issuing any formal order at this point, but that the request had been communicated orally.
The Court said,
“We want the Bar Council of India to constitute a committee of experts, including outside experts, and file a report before us. Please communicate. We are not passing any order…,”
The Bench stressed that judges cannot treat AI-generated outputs as automatically reliable, and it highlighted that judicial decision-making must remain accountable. It warned that if courts or lawyers begin to depend on unverified digital material, the credibility of the adjudication process can be undermined.
The Court also pointed to the need for stronger technological systems such as the creation of a sovereign legal database and an AI model specifically designed for India to lessen reliance on foreign or openly available tools.
The dispute began as a property case before a trial court in Andhra Pradesh. In that case, the trial court rejected the objections to an Advocate Commissioner’s report and relied on certain judicial authorities. Later, it came to light that the cited judgments were not real and were possibly produced with the help of AI tools.
The aggrieved parties challenged the trial court’s order, arguing that basing the decision on fabricated citations compromised the fairness of the proceedings. The case ultimately reached the Supreme Court, putting a spotlight on the overlap between technology and judicial ethics.
Also Read: Will AI Replace Lawyers? The Future of AI in Law and Legal Practice
During the proceedings, the Court sought responses from the Attorney General for India, the Solicitor General, and the Bar Council of India.
It also appointed Senior Advocate Shyam Divan as amicus curiae to assist the Court in addressing both the legal and technological aspects of the concern. In the most recent hearing, Attorney General R Venkataramani told the Court that he had suggestions, but said he first wanted to consult the Ministry of Electronics and Information Technology (MeitY).
At this stage, the Bench observed regarding regulatory developments:
“The AI committee is looking into it. It is almost at the verge of finalising the draft regulations. It will then be thrown open for consultation.”
However, the Court’s concerns were not limited to regulation alone. Speaking to Senior Advocate Shyam Divan, the Bench made it clear that accountability cannot be diluted just because the source of error involves technology.
It emphasized that both lawyers and judges must ensure the authenticity of material submitted on record.
The Bench asked,
“Research is also undertaken at the end of the judge, whether by mistake or otherwise, wherein a non-existent judgment is cited. Where then does the obligation and responsibility lie to ensure accountability, and ultimately, integrity in the conduct of proceedings?”
Also Read: “AI For All”: Centre Introducing Legislation To Regulate Artificial Intelligence (AI)
In response, Divan submitted that the Supreme Court’s Centre for Research and Planning has already prepared a white paper on AI in the judiciary. He noted that the document offers useful recommendations and could provide a starting point for addressing these concerns.
The Bench also noted that the problem is worsened by the lack of sovereign Large Language Models (LLMs) built for India’s legal environment.
It pointed out that using open-source or foreign AI systems increases the chances of hallucinations, making errors more or less unavoidable.
This has led the Court to consider the need for an indigenous AI approach backed by a verified legal database.
The issue also comes against the backdrop of rapid digitisation in the Indian judiciary, including initiatives like e-Courts and online legal research platforms. While these developments are intended to improve efficiency and expand access to justice, the present case demonstrates the dangers of using emerging technology without adequate safeguards.
In sum, without issuing a formal order, the Supreme Court has nonetheless indicated that AI-related challenges require urgent attention. By asking the Bar Council of India to form an expert committee and submit a report, the Court has started an important discussion on how to balance technological innovation with professional responsibility.
The case is now set to be heard again in the last week of May.
Case Title: Gummadi Usha Rani v. Sure Mallikarjuna Rao
FOLLOW US FOR MORE LEGAL UPDATES ON YOUTUBE