🥇Top ML Papers of the Week
The top ML Papers of the Week (May 15 - May 21)
1). Drag Your GAN - an approach for controlling GANs that allows dragging points of the image to precisely reach target points in a user-interactive manner. (paper | tweet)
2). Evidence of Meaning - argues that language models can learn meaning despite being trained only to perform next token prediction on text. (paper | tweet)
3). Med-PaLM 2 - a top-performing LLM for medical question answering; scored up to 86.5% on the MedQA dataset (a new state-of-the-art); approaches or exceeds SoTA across MedMCQA, PubMedQA, and MMLU clinical topics datasets. (paper | tweet)
4). MEGABYTE - a multi-scale decoder architecture enabling end-to-end modeling of sequences of over one million bytes; enables sub-quadratic self-attention and improved parallelism during decoding. (paper | tweet)
5). StructGPT - improves the zero-shot reasoning ability of LLMs over structured data; effective for solving question answering tasks based on structured data. (paper | tweet)
6). TinyStories - uses a synthetic dataset of short stories to train and evaluate LMs that are much smaller than SoTA models but can produce fluent and consistent stories with several paragraphs, and demonstrate reasoning capabilities. (paper | tweet)
7). DoReMi - trains a small proxy model over domains to produce domain weights without knowledge of downstream tasks; it then resamples a dataset with the domain weights and trains a larger model; this enables using a 280M proxy model to train an 8B model (30x larger) more efficiently. (paper | tweet)
8). CodeT5+ - supports a wide range of code understanding and generation tasks and different training methods to improve efficacy and computing efficiency; tested on 20 code-related benchmarks using different settings like zero-shot, fine-tuning, and instruction tuning; achieves SoTA on tasks like code completion, math programming, and text-to-code retrieval tasks. (paper | tweet)
9). Symbol Tuning - an approach to finetune LMs on in-context input-label pairs where natural language labels are replaced by arbitrary symbols; boosts performance on unseen in-context learning tasks and algorithmic reasoning tasks. (paper | tweet)
10). Searching for Needles in a Haystack - shows that PaLM is exposed to over 30 million translation pairs across at least 44 languages; shows that incidental bilingualism connects to the translation capabilities of PaLM. (paper | tweet)