• Thomas Kipf | Relational Structure Discovery

    Online event

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: Relational Structure Discovery (Thomas Kipf is a Research Scientist at Google Research in the Brain Team in Amsterdam)

    Abstract: Graphs are a powerful abstraction: they allow us to efficiently describe data in the form of entities and their pairwise relationships. The past four years have seen an incredible proliferation of graph neural networks (GNNs): neural network architectures that are effective at learning and reasoning with data provided in the form of a graph. Rarely, however, do we ask the question where and how the entities and relations are obtained from in the first place on which we deploy our models, and how we can infer effective relational abstractions from data in cases where they are not available. This talk focuses on the question of how we can build effective relational machine learning models in the absence of annotated links or relations, or even in the absence of abstractions such as entities or objects in the first place. I will give a brief introduction to graph neural networks (GNNs) and cover work on GNN-based link prediction, on Neural Relational Inference, and more recent work on object discovery and relational learning with raw perceptual inputs, such as images or videos.

    Bio: Thomas Kipf is a Research Scientist at Google Research in the Brain Team in Amsterdam. Prior to joining Google, he completed his PhD at University of Amsterdam under Prof. Max Welling on the topic "Deep Learning with Graph-Structured Representations”. His research interests lie in the area of relational learning and in developing models that can reason about the world in terms of structured abstractions such as objects and their relations.

    The discussion will be facilitated by Johannes Klicpera, a PhD student in the Data Analytics and Machine Learning group at TU Munich, currently interning at Facebook AI Research. His research is centered on machine learning for relational data, from web-scale networks to small molecules, with a focus on graph neural networks. Before starting his PhD, he studied Computer Science and Physics at TU Munich and the University of Cambridge.

    5
  • Anima Anandkumar | Al4Science: A Revolution in the Making

    Virtual London Machine Learning Meetup -[masked] @ 18:15

    We would like to invite you to our next Virtual Machine Learning Meetup.

    The discussion will be facilitated by Edgar Schönfeld. Edgar is a Ph.D. student at the BOSCH Center for AI in Germany, working on generative models and out-of-distribution generalization. He holds an MSc degree in AI from the University of Amsterdam and a BSc degree in Nanobiology from TU Delft.

    Agenda:
    - 18:10: Virtual doors open
    - 18:15: Talk
    - 18:55: Q&A session
    - 19:15: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: AI4Science: A Revolution in the Making (Anima Anandkumar is a Bren Professor at Caltech and Director of ML Research at NVIDIA)

    Abstract: AI holds immense promise in enabling scientific breakthroughs and discoveries in diverse areas. However, in most scenarios, this is not a standard supervised learning framework. AI4science often requires zero-shot generalization to entirely new scenarios not seen during training. For instance, drug discovery requires predicting properties of new molecules that can vastly differ from training data, and AI-based PDE solvers require solving any instance of the PDE family. Such zero-shot generalization requires infusing domain knowledge and structure. I will present recent success stories in using AI to obtain 1000x speedups in solving PDEs and quantum chemistry calculations.

    Bio: Anima Anandkumar is a Bren Professor at Caltech and Director of ML Research at NVIDIA. She was previously a Principal Scientist at Amazon Web Services. She has received several honors such as Alfred. P. Sloan Fellowship, NSF Career Award, Young investigator awards from DoD, and Faculty Fellowships from Microsoft, Google, Facebook, and Adobe. She is part of the World Economic Forum's Expert Network. She is passionate about designing principled AI algorithms and applying them in interdisciplinary applications. Her research focus is on unsupervised AI, optimization, and tensor methods.

  • Xavier Bresson | The Transformer Network for the Traveling Salesman Problem

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: The Transformer Network for the Traveling Salesman Problem (Xavier Bresson is an Associate Professor in the Department of Computer Science at the National University of Singapore (NUS))

    Abstract: The Traveling Salesman Problem (TSP) is the most popular and most studied combinatorial problem, starting with von Neumann in 1951. It has driven the discovery of several optimization techniques such as cutting planes, branch-and-bound, local search, Lagrangian relaxation, and simulated annealing. The last five years have seen the emergence of promising techniques where (graph) neural networks have been capable to learn new combinatorial algorithms. The main question is whether deep learning can learn better heuristics from data, i.e. replacing human-engineered heuristics? This is appealing because developing algorithms to tackle NP-hard problems may require years of research, and many industry problems are combinatorial by nature. In this project, we propose to adapt the recent successful Transformer architecture originally developed for natural language processing to the combinatorial TSP. Training is done by reinforcement learning, hence without TSP training solutions, and decoding uses beam search. We report improved performances over recent learned heuristics.

    Bio: Xavier Bresson is an Associate Professor in the Department of Computer Science at the National University of Singapore (NUS). His research focuses on Graph Deep Learning, a new framework that combines graph theory and neural network techniques to tackle complex data domains. In 2016, he received the US$2.5M NRF Fellowship, the largest individual grant in Singapore, to develop this new framework. He was also awarded several research grants in the U.S. and Hong Kong. He co-authored one of the most cited works in this field, and he has recently introduced with Yoshua Bengio a benchmark that evaluates graph neural network architectures. He has organized several workshops and tutorials on graph deep learning such as the recent IPAM'21 workshop on "Deep Learning and Combinatorial Optimization", the MLSys'21 workshop on "Graph Neural Networks and Systems", the IPAM'19 and IPAM'18 workshops on "New Deep Learning Techniques", and the NeurIPS'17, CVPR'17 and SIAM'18 tutorials on "Geometric Deep Learning on Graphs and Manifolds". He has been a regular invited speaker at universities and companies to share his work. He has also been a speaker at the KDD'21, AAAI'21 and ICML'20 workshops on "Graph Representation Learning", and the ICLR'20 workshop on "Deep Neural Models and Differential Equations". He has been teaching graduate courses on Graph Neural Networks at NTU, and as a guest lecturer for Yann LeCun's course at NYU.

  • Frank Willett | High-performance brain-to-text communication via handwriting

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: High-performance brain-to-text communication via handwriting (Frank Willett is a Research Scientist working in the Neural Prosthetics Translational Laboratory at Stanford University)

    *Paper:
    High-performance brain-to-text communication via handwriting | Nature
    https://www.nature.com/articles/s41586-021-03506-2

    Abstract: Brain–computer interfaces (BCIs) can restore communication to people who have lost the ability to move or speak. So far, a major focus of BCI research has been on restoring gross motor skills, such as reaching and grasping, or point-and-click typing with a computer cursor. However, rapid sequences of highly dexterous behaviours, such as handwriting or touch typing, might enable faster rates of communication.

    Here we developed an intracortical BCI that decodes attempted handwriting movements from neural activity in the motor cortex and translates it to text in real time, using a recurrent neural network decoding approach. With this BCI, our study participant, whose hand was paralysed from spinal cord injury, achieved typing speeds of 90 characters per minute with 94.1% raw accuracy online, and greater than 99% accuracy offline with a general-purpose autocorrect.

    To our knowledge, these typing speeds exceed those reported for any other BCI, and are comparable to typical smartphone typing speeds of individuals in the age group of our participant (115 characters per minute).

    In an effort to engage the machine learning community, we have publicly released all data, consisting of the neural activity recorded during the attempted handwriting of 1,000 sentences (43,501 characters) over 10.7 hours. The data can be used to explore Important next steps, which include reducing the amount of data needed to train the neural network decoder, and eliminating the need to recalibrate the system when neural activity changes over time.

    Bio: Frank Willett is a Research Scientist working in the Neural Prosthetics Translational Laboratory at Stanford University. His work is aimed broadly at brain-computer interfaces and understanding how the brain represents and controls movement. Recently, Frank has developed a brain-computer interface that can decode attempted handwriting movements from neural activity in motor cortex. Frank has also worked on understanding how different body parts are represented in motor cortex at single neuron resolution. This work led to a surprising finding: what was previously thought to be “arm/hand” area of motor cortex actually contains an interlinked representation of the entire body. Prior to working at Stanford University, Frank earned his PhD in the Department of Biomedical Engineering at Case Western Reserve University.

  • Omer Levy | Natural Language Processing without Big Labeled Data

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: Natural Language Processing without Big Labeled Data (Omer Levy is a senior lecturer at Tel Aviv University's school of computer science and a research scientist at Facebook AI Research)

    Papers:
    https://arxiv.org/pdf/2101.00438.pdf
    https://arxiv.org/pdf/2010.11982.pdf

    Abstract: The latest methods in NLP involve pretraining a model on unlabeled text and then fine-tuning it on annotated input-output examples from the target task. While plain text is abundant, labeled examples must be collected by humans in a time-consuming and expensive process, which is often impractical to scale up via crowdsourcing due to the language (e.g. Hebrew) or domain (e.g. medicine). If collecting vast amounts of high quality labeled data can be prohibitive, perhaps we should find ways to train models with less? In this talk, I will present our first steps in exploring how to train NLP models without large amounts of labeled examples.

    Bio: Omer Levy is a senior lecturer at Tel Aviv University’s school of computer science and a research scientist at Facebook AI Research. He completed his BSc and MSc at Technion and his PhD at Bar-Ilan University, and did postdoctoral research at the University of Washington. Omer's research is in the intersection of natural language processing (NLP) and machine learning. He has worked on a variety of topics including semantics, self-supervised learning, machine translation, language modeling, dataset annotation, and explainability. Omer is particularly interested in creating NLP models that can generalize well.

  • Christian Szegedy | The Inverse Mindset of Machine Learning

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup.

    The discussion will be facilitated by Max Jaderberg, a researcher at DeepMind leading the Open-Ended Learning team, driving the intersection of deep learning, reinforcement learning, and multi-agent systems. His recent work includes creating the first agent to beat human professionals at StarCraft II, and creating algorithms for training teams of agents to play with humans in first-person video games.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: The Inverse Mindset of Machine Learning (Christian Szegedy is a machine learning and AI researcher at Google Research, currently focusing on formal reasoning and natural language processing using deep neural networks)

    Abstract: Here, in this talk, I will give examples of how large areas of machine learning promotes and requires a mindset different from more to traditional areas of computer science. While most of computer science is focused on disciplined, efficient solutions for most problems, large parts of machine learning and AI is focused on finding good tasks and curricula for certain domains. While machine learning still requires classical optimization and efficient solutions, a lot of the work shifts towards encoding and creating interesting problems and exploring the power and limits of new solutions. For example, transformer networks act as a powerful, self-routing structure and pre-training with the correct, conceptually relevant tasks has become a large area of research. Here we will examine several examples of the working of this mindset in practical examples.

    This presentation will assume some familiarity with transformer networks and the basics of contrastive training approaches.

    Bio: Christian Szegedy is a machine learning and AI researcher at Google Research, currently focusing on formal reasoning and natural language processing using deep neural networks. He holds a PhD in applied mathematics from the University of Bonn, Germany and worked on algebraic combinatorics, placement, routing and timing optimization of chips, logic synthesis via combinatorial optimization, advertisement pricing optimization and computer vision using deep convolutional networks. He is best known for discovering adversarial examples and co-inventing Batch Normalization. He has designed the vision deep networks of the Inception family and has co-authored the first deep-learning paper on mathematical theorem proving at a large scale.

  • Noam Brown | AI for Imperfect-Information Games: Poker and Beyond

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup. Please read the papers below and help us create a vibrant discussion.

    The discussion will be facilitated by Michal Šustr. Michal is a PhD student at Czech Technical University, interested in large-scale imperfect information games. He is one of the main contributors to OpenSpiel, a framework for reinforcement learning in games and co-organizer of the new Hidden Information Games Competition, a benchmark of game-playing AIs.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: AI for Imperfect-Information Games: Poker and Beyond
    (Noam Brown is a Research Scientist at Facebook AI Research working on multi-agent artificial intelligence, with a particular focus on imperfect-information games)

    * Papers:
    http://www.cs.cmu.edu/~noamb/papers/19-Science-Superhuman.pdf
    https://proceedings.neurips.cc//paper/2020/file/c61f571dbd2fb949d3fe5ae1608dd48b-Paper.pdf

    Abstract: The field of artificial intelligence has had a number of high-profile successes in the domain of perfect-information games like chess or Go where all participants know the exact state of the world. But real-world strategic interactions typically involve hidden information, such as in negotiations, cybersecurity, and financial markets. Past AI techniques fall apart in these settings, with poker serving as the classic benchmark and challenge problem.

    In this talk, I will cover the key breakthroughs behind Libratus and Pluribus, the first AI agents to defeat elite human professionals in two-player no-limit poker and multiplayer no-limit poker, respectively. In particular, I will discuss new forms of the counterfactual regret minimization equilibrium-finding algorithm and breakthroughs that enabled depth-limited search for imperfect-information games to be conducted orders of magnitude faster than previous algorithms. Finally, I will conclude with a discussion on recent work combining the previously separate threads of research on perfect-information and imperfect-information games.

    Bio: Noam Brown is a Research Scientist at Facebook AI Research working on multi-agent artificial intelligence, with a particular focus on imperfect-information games. He co-created Libratus and Pluribus, the first AIs to defeat top humans in two-player no-limit poker and multiplayer no-limit poker, respectively. He has received the Marvin Minsky Medal for Outstanding Achievements in AI, was named one of MIT Tech Review's 35 Innovators Under 35, and his work on Pluribus was named by Science Magazine to be one of the top 10 scientific breakthroughs of the year. Noam received his PhD from Carnegie Mellon University, where he received the School of Computer Science Distinguished Dissertation Award.

  • Irwan Bello | LambdaNetworks and Recent Developments in Computer Vision

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup. Please read the papers below and help us create a vibrant discussion.

    The discussion will be facilitated by Gül Varol, an Assistant Professor at the IMAGINE team of École des Ponts ParisTech. Previously, she was a postdoctoral researcher at the University of Oxford (VGG), working with Andrew Zisserman. She obtained her PhD from the WILLOW team of Inria Paris and École Normale Supérieure. Her research is focused on human understanding in videos, specifically action recognition, body shape and motion analysis, and sign language.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: LambdaNetworks and Recent Developments in Computer Vision
    (Irwan Bello is a Research Scientist at Google Brain where he works on Deep Learning)

    * Papers:
    https://arxiv.org/abs/2102.08602
    https://arxiv.org/abs/2103.07579

    Abstract: The first part of the talk will be dedicated to LambdaNetworks: Modeling Long-Range Interactions Without Attention. Lambda layers are a scalable alternative framework to self-attention for capturing long-range structured interactions between an input and contextual information. Similar to linear attention, lambda layers bypass expensive attention maps, but in contrast, they model both content and position-based interactions which enables their application to large structured inputs such as images. The resulting neural network architectures, LambdaNetworks, significantly outperform their convolutional and attentional counterparts on ImageNet classification, COCO object detection and instance segmentation, while being more computationally efficient. We design LambdaResNets that reach state-of-the-art accuracies on ImageNet while being 3.2 - 4.4x faster than the popular EfficientNets on modern machine learning accelerators. In large-scale semi-supervised training, LambdaResNets achieve up to 86.7% ImageNet accuracy while being 9.5x faster than EfficientNet NoisyStudent and 9x faster than a Vision Transformer with comparable accuracy.

    Second, I'll discuss a recent preprint Revisiting ResNets: Improved Training and Scaling Strategies. Novel vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies. Our work revisits the canonical ResNet and studies these three aspects in an effort to disentangle them. Perhaps surprisingly, we find that training and scaling strategies may matter more than architectural changes, and further, that the resulting ResNets match recent state-of-the-art models. We show that the best performing scaling strategy depends on the training regime and offer two new scaling strategies: (1) scale model depth in regimes where overfitting can occur (width scaling is preferable otherwise); (2) increase image resolution more slowly than previously recommended. Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1.7x - 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet. The training techniques improve transfer performance on a suite of downstream tasks (rivaling state-of-the-art self-supervised algorithms) and extend to video classification. We recommend practitioners use these simple revised ResNets as baselines for future research.

    Bio: Irwan Bello is a Research Scientist at Google Brain where he works on Deep Learning. His research interests primarily lie in modeling, scaling and designing architectures that process structured information while trading off scalability and inductive biases.

    1
  • Mike Lewis | Beyond BERT: Representation Learning for Natural Language at Scale

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup. Please read the papers below and help us create a vibrant discussion.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: Beyond BERT: Representation Learning for Natural Language at Scale (Mike Lewis is a research scientist at Facebook AI Research in Seattle)

    * Papers:
    https://arxiv.org/abs/1910.13461
    https://arxiv.org/abs/1911.00172
    https://arxiv.org/abs/2006.15020

    Abstract: Natural language processing has been revolutionized by large scale unsupervised representation learning, in which ever larger models are pre-trained on unlabelled text. I will describe some of my work on this topic, including the widely used RoBERTa and BART models. I will also discuss how to efficiently increase the capacity of models beyond brute force scaling, such as the non-parametric language model kNN-LM and recent work on large, sparse models. These approaches allow highly expressive models with greatly reduced training costs.

    Bio: Mike Lewis is a research scientist at Facebook AI Research in Seattle, working on representation learning for natural language. Previously, he was a postdoc at the University of Washington (working with Luke Zettlemoyer), developing search algorithms for neural structured prediction. He has a PhD from the University of Edinburgh (advised by Mark Steedman) on combining symbolic and distributed representations of meaning. He received an Outstanding Submission Award at the 2014 ACL Workshop on Semantic Parsing, Best Paper at EMNLP 2016, Best Resource Paper at ACL 2017, and Best Paper Honourable Mention at ACL 2018. His work has been extensively covered in the media, with varying levels of accuracy.

    1
  • Mingxing Tan | AutoML for Efficient Vision Learning

    Online event

    Virtual London Machine Learning Meetup -[masked] @ 18:30

    We would like to invite you to our next Virtual Machine Learning Meetup. Please read the papers below and help us create a vibrant discussion.

    Agenda:
    - 18:25: Virtual doors open
    - 18:30: Talk
    - 19:10: Q&A session
    - 19:30: Close

    *Sponsors*
    https://evolution.ai/ : Machines that Read - Intelligent data extraction from corporate and financial documents.

    * Title: AutoML for Efficient Vision Learning (Mingxing Tan is a researcher at Google Brain)

    * Papers:
    EfficientNet - https://arxiv.org/abs/1905.11946
    EfficientDet - https://arxiv.org/abs/1911.09070
    MnasNet - https://arxiv.org/abs/1807.11626

    Abstract: This talk will focus on a few recent progresses we have made on AutoML, particularly on neural architecture search for efficient convolutional neural networks. We will first discuss the challenges and solutions in designing network architecture search spaces / algorithms / constraints, as well as hyperparamter auto-tuning. Afterwards, we will discuss how to scale up neural networks for better accuracy and efficiency. We will conclude the talk with a few AutoML applications on image classification, detection, segmentation.

    Bio: Mingxing Tan is a researcher at Google Brain, mainly focusing on AutoML research and applications. He has co-authored several popular models including EfficientNet and EfficientDet. He finished his Ph.D. at Peking University and Post-Doc at Cornell University.

    1