Long Beach, California, tested “Ask Elby,” an AI assistant on the city website that answered residents’ questions about city services. City staff worked with experts from each department through three rounds of testing to build its knowledge base. In a January 2025 post, the city reported that in a 30-day trial, 4,072 residents asked it a question and it handled nearly 7,000 communications.
On November 6, 2025, at the GovAI Coalition summit in San José, the city’s technology partnerships officer, Ryan Kurtzman, described what came next, GovTech reported. Feedback in the satisfaction survey was “very middle of the road.” The chatbot did not noticeably improve the user experience or resolve issues better than a search bar. It often gave outdated or incorrect information, missing links, and other dead ends. Residents looking for 2024 content got news items from 2020.
The city paused the rollout and decided against full implementation. Instead of adding an AI layer, it chose to improve the website and its search. Kurtzman said the city may revisit Elby, but first wants to validate the problem and learn what users need.
Accounts of the test’s size differ. GovTech reported more than 4,000 users and 15,000 queries in a month. The city did not disclose a vendor or cost, and its January 2025 page describing the trial is still posted without an update.
The chatbot repeated the website’s problems. Long Beach fixed the website.
Why it matters
A pilot that ends in no is still a pilot that worked. Long Beach measured the chatbot against what residents already had, a search bar, and it did not clear that bar.
The failure mode applies to any assistant built on existing content. A chatbot drawing on outdated pages serves outdated answers faster and with more confidence. When the problem is the information, an AI interface does not fix it.
There is also a record gap. The launch got a city announcement. The decision not to scale surfaced at a conference.
Questions before your next chatbot pilot
- What does the pilot have to beat, and have you measured that baseline?
- Who owns the content the chatbot draws from, and when was it last reviewed?
- What result would end the pilot, and did you write it down before it started?
- How will you tell the public if you stop?
- Is the underlying problem the interface, or the information behind it?
A pilot that ends in no has done its job. Long Beach said why in public.
Sources