A new Senate bill introduced in California aims to curb how lawyers and arbitrators and lawyers can use AI to support their work, with a focus on data protection and preventing so-called ‘hallucinations’ from fabricating false information.
Introduced by Democratic senator Thomas Umberg, amended Senate Bill 574 would require legal professionals to ensure confidential information is not entered into AI systems, to personally review any AI generated work and to prohibit delegation of decision making to machines. The bill sits within Part I of Umberg’s six bill 2026 package alongside measures on elections, substance control, treatment safety, contractor licensing and media regulation.
‘Each of these bills respond to a gap in existing law whether its election eligibility, the use of artificial intelligence in the courts or oversight of substance use treatment. This package is about accountability, fairness, and making sure Californians are protected as our systems and technologies change,’ Umberg said when announcing the group of bills.
Speaking during the January 13 Senate Standing Committee on Judiciary hearing, Umberg stressed that AI now permeates the legal field and still makes mistakes, which necessitates human responsibility and disclosure when AI is used. SB 574 would formalize that expectation, making human oversight a statutory requirement rather than an implied professional norm.
For in-house counsel, the bill carries practical and cultural implications. Legal departments already experimenting with generative AI for contract review, research or compliance support would need tighter internal controls, clearer usage policies and closer collaboration with IT and security teams. Confidentiality safeguards would move from best practice to legal obligation, while review requirements could slow down workflows built around rapid AI assisted drafting. In-house lawyers advising business units on AI adoption may also find themselves under pressure to explain why legal cannot move faster while still meeting the bill’s standards.
Critics question whether SB 574 adds meaningful protection or simply restates existing duties in statutory form. The requirement that attorneys personally verify every citation, including those generated by AI, is seen by some as a time-consuming burden, particularly when courts already sanction inaccurate filings. Others point to the lack of new enforcement mechanisms, arguing that the bill relies on existing court rules and state bar discipline, raising doubts about whether it meaningfully addresses AI hallucinations or merely signals concern.
There is also unease around the bill’s broader language. Obligations to ensure AI tools do not discriminate against protected classes or produce harmful content could invite wide interpretation, future litigation and uncertainty over compliance standards.
SB 574 comes amid a flurry of AI related laws and proposals in California that signal the state is adopting a multi-layered governance approach rather than leaving AI oversight to federal action or voluntary corporate policies.
In2025 the California legislature passed the Transparency in Frontier Artificial Intelligence Act (SB 53), a first-of-its-kind law requiring large AI developers to publish safety frameworks, report critical incidents and protect whistleblowers.
Alongside SB 53, other AI rules target specific use cases.Senate Bill 243, which is now law, regulates AI ‘companion chatbots’, requiring user notices, safeguards against harmful content and protocols for minors’ interactions.AB 853delays implementation of the AI Transparency Act while expanding disclosure and detection obligations for AI content platforms.
SB 574 points to a broader strategy: regulators are not only concerned with the outputs of AI technology at the developer level but are also pushing accountability into professional practice and domain-specific use cases. For general counsel, this means staying ahead of a patchwork of rules that govern how AI can be used, disclosed, verified and audited