Data Science @ Regensburg

Speaker 1:
Adam Roegiest (VP, Research & Technology at Zuva)

Title:
Aboutness Is Not Equivalence: The Limits of Embedding Similarity

Abstract:
The cosine similarity of text embeddings is often framed as measuring the "semantic" similarity of the text fragments, reflecting their origins in distributional semantics. This shorthand is often reasonable among technical practitioners who understand how embeddings are calculated and the limitations that entails. When terms like "semantic similarity" are used in front of non-experts, this can encourage misleading intuition about what the measure captures. For example, a legal practitioner might reasonably expect two "semantically similar" legal clauses to impose the same rights, requirements, and obligations, or very nearly so, but is this what cosine similarity actually measures?

In this talk, we show that such an assumption does not hold, at least for contractual language, and that care is warranted when calculating domain-specific measures of similarity. In a first study, we compare a reference clause with two variants: a minimally edited version that negates/reverses its legal meaning and a substantially reworded version that preserves its intent. Across embedding models, we see that the negated clauses are consistently more "semantically" similar than intent-preserving rewordings. In a second study, we explore whether explicit reasoning over clause obligations can better approximate lawyer's assessment of clause similarity than do embeddings. While embeddings remain competitive in this more naturalistic setting, prompt-based methods outperform them in most settings and provide interpretable explanations, as opposed to the opaque scores of embeddings. These results provide a broader lesson extending beyond the legal domain. Embeddings are effective at identifying pieces of text that are about the same thing, but domain-specific similarity will likely rely on which differences matter to the end user.

---

Speaker 2:
Yiran Hu (PhD student, University of Waterloo)

Title:
Evaluation of LLMs for legal applications

Abstract:
When large language models (LLMs) are deployed in domain-specific tasks, it is essential to evaluate their trustworthiness. In the legal domain, robustness and fairness are two particularly critical aspects.

In this talk, I will first discuss about some existing ways for integrating legal knowledge into LLMs, then I will present two of my recent papers: “J&H: Evaluating the Robustness of Large Language Models under Knowledge-Injection Attacks in the Legal Domain” and “LLMs on Trial: Evaluating Judicial Fairness for Large Language Models.” The first paper investigates whether LLMs rely on genuine legal domain knowledge when performing legal reasoning, particularly under knowledge-injection attacks. The second paper adopts a counterfactual evaluation framework grounded in legal theories of judicial fairness to assess fairness-related behaviors of LLMs in judicial decision-making scenarios.