Two Frontiers of Transformers: Life Trajectories and Tabular Learning

Life Sequence Transformer: Generative Modelling of Socio-Economic Trajectories from Administrative Data by Alberto Cabezas and Carlotta Montorsi
Generative modelling with Transformer architectures can simulate complex sequential structures across various applications. We extend this line of work to the social sciences by introducing a Transformer-based generative model tailored to longitudinal socio-economic data. Our contributions are: (i) we design a novel encoding method that represents socio-economic life histories as sequences, including overlapping events across life domains; and (ii) we adapt generative modelling techniques to simulate plausible alternative life trajectories conditioned on past histories. Using large-scale data from the Italian social security administration (INPS), we show that the model can be trained at scale, reproduces realistic labour market patterns consistent with known causal relationships, and generates coherent hypothetical life paths. This work demonstrates the feasibility of generative modelling for socio-economic trajectories and opens new opportunities for policy-oriented research, with counterfactual generation as a particularly promising application.
The Rise of Tabular Foundation Models: From Zero-Shot Inference to the Security Frontier by Karim TIT
Traditional tabular machine learning has long been dominated by gradient-boosted decision trees and complex AutoML pipelines that require extensive search, hyperparameter tuning, and feature engineering. In this talk, I will first introduce the emerging paradigm of Tabular Foundation Models (TFMs), which leverage Transformer architectures and massive synthetic pre-training datasets to achieve state-of-the-art performance via zero-shot in-context learning. I will conduct a deep dive into the TabPFN family, explaining the Prior-fitted Network paradigm and its latest evolutions, including Real-TabPFN, which integrates curated real-world data to handle larger, more complex datasets. Building on these foundations, I will provide a tour of other diverse SOTA approaches like TabDPT and TARTE, highlighting how different architectural choices impact scalability. Finally, I will shift to a security-researcher perspective to discuss the hidden vulnerabilities of this new paradigm. I will present recent results from our paper (accepted at SatML 2026) showing that TFMs are highly susceptible to structured test-time attacks and can even be repurposed as tools to attack traditional models. I will conclude by introducing Adversarial In-Context Learning (AICL), a novel defense that hardens these models by optimizing their data context, offering a path toward more robust and reliable tabular AI.