UW/AI2

wellecks {@ | at} uw.edu

Thesis Review Podcast

I am a Postdoctoral Scholar at the University of Washington and AI2, working with Yejin Choi.

Previously, I earned a PhD at New York University, advised by Kyunghyun Cho and Zheng Zhang.

I host the Thesis Review Podcast.

My research focuses on machine learning methods for generating and reasoning with natural language. This includes:

- Developing learning [1, 2, 3, 4], decoding [1, 2, 3, 4], and evaluation [1, 2] methods for neural sequence generation.
- Bridging symbolic structure and neural methods in commonsense [1, 2, 3, 4] and mathematical [1, 2, 3] reasoning.

My long-term goal is enabling high-impact applications in scientific discovery and education through advances in these areas [1, 2, 3].

**[09.2022]**NaturalProver, Quark, and COLD Decoding accepted at NeurIPS 2022!**[09.2022]**Talk at AITP 2022 on NaturalProver [slides].**[07.2022]**Co-organizing*Math-AI: Towards Human-Level Mathematical Reasoning*at Neurips 2022. [webpage]**[06.2022]****Best Paper Award**at NAACL 2022 for our paper on constrained A*-esque decoding!**[05.2022]**New paper on NaturalProver: mathematical proof generation with language models.**[05.2022]**New paper on Quark, an [un]learning algorithm for aligning language models.**[05.2022]**New paper on logically consistent reasoning through Maieutic inference.**[05.2022]**Talks at IFDS and Cohere.ai on constrained text generation [slides].**[04.2022]**Our papers on A* decoding, commonsense knowledge, and continuous prompts accepted at NAACL 2022.**[03.2022]**Guest lecture on neural sequence generation [slides].**[02.2022]**New paper on COLD Decoding: constrained decoding with Langevin dynamics.**[02.2022]**Our paper on generated knowledge prompting accepted at ACL 2022.**[12.2021]**New paper on NeuroLogic A*: decoding with lookahead and logical constraints.**[12.2021]**Held the first MATH-AI for Education workshop at NeurIPS 2021.**[12.2021]**Our paper on generalization in symbolic mathematics accepted to AAAI 2022.**[12.2021]**Mauve received an**Outstanding Paper Award**at NeurIPS 2021.**[12.2021]**Co-organizing a*Workshop on Mathematical Language Processing*at EMNLP 2022. [webpage]**[10.2021]**NaturalProofs accepted for an oral presentation at NeurIPS 2021 [slides].

- Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations

J. Jung, L. Qin,**S. Welleck**, F. Brahman, C. Bhagavatula, R. Le Bras, Y. Choi.

arXiv preprint 2022.

[slides]

- NaturalProver: Grounded Mathematical Proof Generation with Language Models
**S. Welleck**, J. Liu, X. Lu, H. Hajishirzi, Y. Choi.

NeurIPS 2022.

[code][slides] - Quark: Controllable Text Generation with Reinforced [Un]learning

X. Lu,**S. Welleck**, L. Jiang, J. Hessel, L. Qin, P. West, P. Ammanabrolu, Y. Choi.

NeurIPS 2022. - COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics

L. Qin,**S. Welleck**, D. Khasabi, Y. Choi.

NeurIPS 2022.

[code][slides] - NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics

X. Lu,**S. Welleck**, P. West, L. Jiang, J. Kasai, D. Khasabi, R. Le Bras, L. Qin, Y. Yu, R. Zellers, N. Smith, Y. Choi.

NAACL 2022.

[code][slides] - Symbolic Knowledge Distillation: from General Language Models to Commonsense Models

P. West, C. Bhagavatula, J. Hessel, J. Hwang, L. Jiang, R. Le Bras, X. Lu,**S. Welleck**, Y. Choi.

NAACL 2022.

[code] - Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

D. Khasabi, S. Lyu, S. Min, L. Qin, K. Richardson, S. Singh,**S. Welleck**, H. Hajishirzi, T. Khot, A. Sabharway, Y. Choi.

NAACL 2022. - Generated Knowledge Prompting for Commonsense Reasoning

J. Liu, A. Liu, X. Lu,**S. Welleck**, P. West, R. Le Bras, Y. Choi, H. Hajishirzi.

ACL 2022.

[code] - Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics
**S. Welleck**, P. West, J. Cao, Y. Choi.

AAAI 2022.

[code][slides][talk] - Towards Grounded Natural Language Proof Generation
**S. Welleck**, J. Liu, J. Han, Y. Choi.

MathAI4Ed Workshop at NeurIPS 2021 (**Contributed Talk**).

[poster][slides] - NaturalProofs: Mathematical Theorem Proving in Natural Language
**S. Welleck**, J. Liu, R. Le Bras, H. Hajishirzi, Y. Choi, K. Cho.

NeurIPS 2021 Datasets and Benchmarks (**Oral (Top 1%)**).

[data/code][talk] - MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun,**S. Welleck**, Y. Choi, Z. Harchaoui.

NeurIPS 2021 (**Oral, Outstanding Paper Award (Top 0.1%)**).

[code][press] - Divergence Frontiers for Generative Models: Sample Complexity, Quantization Level, and Frontier Integral

L. Liu, K. Pillutla,**S. Welleck**, S. Oh, Y. Choi, Z. Harchaoui.

NeurIPS 2021. - Mode recovery in neural autoregressive sequence modeling

I. Kulikov,**S. Welleck**, K. Cho.

SPNLP 2021.

[code] - Order and Learning in Sequential Neural Structured Prediction
**S. Welleck**.

PhD Thesis, New York University.

[slides] - MLE-guided parameter search for task loss minimization in neural sequence modeling
**S. Welleck**, K. Cho.

AAAI 2021.

[code][poster][talk] - Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
**S. Welleck**, I. Kulikov, J. Kim, R. Pang, K. Cho.

EMNLP 2020.

[code][talk] - A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models

E. Mansimov, A. Wang,**S. Welleck**, K. Cho.

arXiv pre-print 2020.

[code] - Making Inconsistent Dialogue Unlikely with Unlikelihood Training

M. Li, S. Roller, I. Kulikov,**S. Welleck**, Y.L. Boureau, K. Cho, J. Weston.

ACL 2020. - Neural Text Generation with Unlikelihood Training
**S. Welleck**, I. Kulikov, S. Roller, E. Dinan, K. Cho, J. Weston.

ICLR 2020.

[code] - Non-Monotonic Sequential Text Generation
**S. Welleck**, K. Brantley, H. Daume III, K. Cho.

ICML 2019.

[code] [slides] [poster] - Sequential Graph Dependency Parser
**S. Welleck**, K. Cho.

RANLP 2019.

[slides] - Dialogue Natural Language Inference
**S. Welleck**, J. Weston, A. Szlam, K. Cho.

ACL 2019.

[dataset][poster][press] - Loss Functions for Multiset Prediction
**S. Welleck**, Z. Yao, Y. Gai, J. Mao, Z. Zhang, K. Cho.

NeurIPS 2018.

NVIDIA AI Labs Pioneering Research Award 2018.

[poster] - Saliency-based Sequential Image Attention with Multiset Prediction
**S. Welleck**, J. Mao, K. Cho, Z. Zhang.

NeurIPS 2017.

NVIDIA AI Labs Pioneering Research Award 2017.

[poster][press] - Efficient AUC Optimization for Information Ranking Applications
**S. Welleck**.

ECIR 2016.

**Guest Lecture: Neural sequence generation**(DATA 598) [slides]

University of Washington

March 2022.**Deep Learning**(DS-GA 1008)

New York University

Fall 2020**Deep Learning for NLP**

African Masterâ€™s Program in Machine Intelligence

March 2020**Introduction to Machine Learning**(CSCI-UA 0473)

New York University

Spring 2020**NLP with Representation Learning**(DS-GA 1011)

New York University

Fall 2019

**Denoising Diffusion Models**[slides]

July 2022.**Generative Modeling with (W)GAN**[slides]

NYU Shanghai

April 2018.

- Facebook, AI Research Team (FAIR), May. 2019 - Sep. 2019
- Facebook, AI Research Team (FAIR), May. 2018 - Jan. 2019
- Primer AI, Feb. 2016 - Aug. 2016
- IBM, Sep. 2014 - Feb. 2016
- University of Pennsylvania (Computer Science, MSE), May. 2013 - May. 2014
- University of Pennsylvania (Computer Science, BSE), Sep. 2009 - Feb. 2013