Google Drops Embeddings From AI Conversation Classification

Google researchers have unveiled 'CALLIOPE,' a technique that automatically classifies the topics of generative AI conversations without the conventional approach of 'embeddings' — a technology that converts sentences into numeric vectors to group similar ones together. The technique can be used to organize vast logs of AI usage by topic. Edith Cohen and other researchers from Google Research, Google, and Google DeepMind published the CALLIOPE paper on arXiv on the 21st (Korea time). The paper describes the procedure for building topic hierarchies from unstructured documents, along with evaluation results.

Conventional methods center on embedding-based clustering, which converts sentences into numeric vectors and groups materials with similar meanings. CALLIOPE eliminates this step entirely.

CALLIOPE's differential privacy is applied at the keyword aggregation stage. The researchers designed it to add noise to keyword frequencies and filter out rarely appearing items, limiting the influence any single document can have on published results. The paper explains that strengthening privacy settings can make classifications broader and coarser, but that the sharp quality degradation can be reduced compared to existing embedding-based methods. However, the current guarantee applies only to a single document. The paper notes that user-level differential privacy — protecting all conversations a person leaves over multiple sessions as a group — requires further research.

The researchers compared CALLIOPE with CLIO and URANIA across six datasets, including the multilingual utterance corpus MASSIVE, the banking customer inquiry dataset Banking77, and movie reviews and news articles. According to the paper, CALLIOPE outperformed URANIA on NMI and ARI metrics measuring alignment with ground-truth categories and on some topic evaluations, and scored comparably or better on metrics assessing internal cluster structure.