Introduction

I'm a Ph.D. student advised by professor Benjamin Van Durme at Johns Hopkins University. I'm interested in natural language processing, especially for semantic parsing and temporal relationship extraction. Before this, I worked as an intern at Knowledge Computing Group, Microsoft Research Asia (MSRA) from 2017.11 to 2018.6, where I was supervised by Chin-Yew Lin. I went to Johns Hopkins University as a visiting researcher from 2018.7 to 2018.10, where I was supervised by Jason Eisner.

Publications

• ICML    Nugget: Neural Agglomerative Embeddings of Text , Guanghui Qin, and Benjamin Van Durme , International Conference on Machine Learning (ICML) , 2023 . [Click to show summary] [Click to hide summary]
  Research Summary

  Embedding text sequences is a widespread requirement in modern language understanding. Existing approaches focus largely on constant-size representations. This is problematic, as the amount of information contained in text often varies with the length of the input. We propose a solution called Nugget, which encodes language into a representation based on a dynamically selected subset of input tokens. These nuggets are learned through tasks like autoencoding and machine translation, and intuitively segment language into meaningful units. We demonstrate Nugget outperforms related approaches in tasks involving semantic comparison. Finally, we illustrate these compact units allow for expanding the contextual window of a language model (LM), suggesting new future LMs that can condition on significantly larger amounts of content. [Click to hide summary]
• EACL    The NLP Task Effectiveness of Long-Range Transformers , Guanghui Qin, Yukun Feng, and Benjamin Van Durme , European Association for Computational Linguistics (EACL) , 2023 . [Click to show summary] [Click to hide summary]
  Research Summary

  Transformer models cannot easily scale to long sequences due to their O(N^2) time and space complexity. This has led to Transformer variants seeking to lower computational complexity, such as Longformer and Performer. While such models have theoretically greater efficiency, their effectiveness on real NLP tasks has not been well studied. We benchmark 7 variants of Transformer models on 5 difficult NLP tasks and 7 datasets. We design experiments to isolate the effect of pretraining and hyperparameter settings, to focus on their capacity for long-range attention. Moreover, we present various methods to investigate attention behaviors to illuminate model details beyond metric scores. We find that the modified attention in long-range transformers has advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. [Click to hide summary]
• NAACL    Learning How to Ask: Querying LMs with Mixtures of Soft Prompts , Guanghui Qin, and Jason M. Eisner , North American Association for Computational Linguistics (NAACL) , 2021 . ( Best Short Paper ) [Click to show summary] [Click to hide summary]
  Research Summary

  Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks, using a fill-in-the-blank paradigm (Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al., 2020). For example, language models retain factual knowledge from their training corpora that can be extracted by asking them to "fill in the blank" in a sentential prompt. However, where does this prompt come from? We explore the idea of learning prompts by gradient descent -- either fine-tuning prompts taken from previous work, or starting from random initialization. Our prompts consist of "soft words," i.e., continuous vectors that are not necessarily word type embeddings from the language model. Furthermore, for each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them. Across multiple English LMs and tasks, our approach hugely outperforms previous methods, showing that the implicit factual knowledge in language models was previously underestimated. Moreover, this knowledge is cheap to elicit: random initialization is nearly as good as informed initialization. [Click to hide summary]
• EACL    Lome: Large ontology multilingual extraction , Patrick Xia, Guanghui Qin, Siddharth Vashishtha, Yunmo Chen, Tongfei Chen, Chandler May, Craig Harman, Kyle Rawlins, Aaron Steven White, and Benjamin Van Durme , European Association for Computational Linguistics (EACL) , 2021 . ( Demo Paper ) [Click to show summary] [Click to hide summary]
  Research Summary

  We present LOME, a system for performing multilingual information extraction. Given a text document as input, our core system identifies spans of textual entity and event mentions with a FrameNet (Baker et al., 1998) parser. It subsequently performs coreference resolution, fine-grained entity typing, and temporal relation prediction between events. By doing so, the system constructs an event and entity focused knowledge graph. We can further apply third-party modules for other types of annotation, like relation extraction. Our (multilingual) first-party modules either outperform or are competitive with the (monolingual) state-of-the-art. We achieve this through the use of multilingual encoders like XLM-R (Conneau et al., 2020) and leveraging multilingual training data. LOME is available as a Docker container on Docker Hub. In addition, a lightweight version of the system is accessible as a web demo. [Click to hide summary]
• EMNLP    Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction , Mahsa Yarmohammadi, Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Craig Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, and Benjamin Van Durme , Empirical Methods in Natural Language Processing (EMNLP) , 2021 . [Click to show summary] [Click to hide summary]
  Research Summary

  Zero-shot cross-lingual information extraction (IE) describes the construction of an IE model for some target language, given existing an- notations exclusively in some other language, typically English. While the advance of pretrained multilingual encoders suggests an easy optimism of "train on English, run on any language", we find through a thorough exploration and extension of techniques that a combination of approaches, both new and old, leads to better performance than any one cross-lingual strategy in particular. We use English-to-Arabic IE as our initial example, with experiments illustrating state of the art in this setting for event extraction, named entity recognition, part-of-speech tagging, and dependency parsing. Finally, we apply data projection to three tasks across eight target languages. [Click to hide summary]
• EMNLP    CopyNext: Explicit Span Copying and Alignment in Sequence to Sequence Models , Abhinav Singh, Patrick Xia, Guanghui Qin, Mahsa Yarmohammadi, and Benjamin Van Durme , Empirical Methods in Natural Language Processing (EMNLP) , 2020 . [Click to show summary] [Click to hide summary]
  Research Summary

  Copy mechanisms are employed in sequence to sequence models (seq2seq) to generate reproductions of words from the input to the output. These frameworks, operating at the lexical type level, fail to provide an explicit alignment that records where each token was copied from. Further, they require contiguous token sequences from the input (spans) to be copied individually. We present a model with an explicit token-level copy operation and extend it to copying entire spans. Our model provides hard alignments between spans in the input and output, allowing for nontraditional applications of seq2seq, like information extraction. We demonstrate the approach on Nested Named Entity Recognition, achieving near state-of-the-art accuracy with an order of magnitude increase in decoding speed. [Click to hide summary]
• ICML    Neural Datalog through time: Informed temporal modeling via logical specification , Hongyuan Mei, Guanghui Qin, Minjie Xu, and Jason M. Eisner , International Conference on Machine Learning (ICML) , 2020 . [Click to show summary] [Click to hide summary]
  Research Summary

  Learning how to predict future events from patterns of past events is difficult when the set of possible event types is large. Training an unrestricted neural model might overfit to spurious patterns. To exploit domain-specific knowledge of how past events might affect an event’s present probability, we propose using a temporal deductive database to track structured facts over time. Rules serve to prove facts from other facts and from past events. Each fact has a time-varying state—a vector computed by a neural net whose topology is determined by the fact’s provenance, including its experience of past events. The possible event types at any time are given by special facts, whose probabilities are neurally modeled alongside their states. In both synthetic and real-world domains, we show that neural probabilistic models derived from concise Datalog programs improve prediction by encoding appropriate domain knowledge in their architecture. [Click to hide summary]
• TACL    Iterative Paraphrastic Augmentation with Discriminative Span Alignment , Ryan Culkin, Edward Hu, Elias Stengel-Eskin, Guanghui Qin, and Benjamin Van Durme , Transactions of the Association for Computational Linguistics (TACL) , 2020 . [Click to show summary] [Click to hide summary]
  Research Summary

  We introduce a novel paraphrastic augmentation strategy based on sentence-level lexically constrained paraphrasing and discriminative span alignment. Our approach allows for the large-scale expansion of existing resources, or the rapid creation of new resources from a small, manually-produced seed corpus. We illustrate our framework on the Berkeley FrameNet Project, a large-scale language understanding effort spanning more than two decades of human labor. Based on roughly four days of collecting training data for the alignment model and approximately one day of parallel compute, we automatically generate 495,300 unique (Frame, Trigger) combinations annotated in context, a roughly 50x expansion atop FrameNet v1.7. [Click to hide summary]
• ICML    Imputing Missing Events in Continuous-Time Event Streams , Hongyuan Mei, Guanghui Qin, and Jason M. Eisner , International Conference on Machine Learning (ICML) , 2019 . [Click to show summary] [Click to hide summary]
  Research Summary

  This work was done during my internship at Johns Hopkins University.

  Problem

  We consider discrete events in continuous time. Events we observe in the world may be caused by other unobserved events. If only some of the events are observed, could we infer the missing events conditioned both the past and future events?

  Approach

  We propose particle smoothing to address this problem. Our work is based on neural Hawkes process, which defines the distribution of the next event using the state of continuous-time LSTM that has read the past history from left to right. Different from particle filtering, when sampling a proposed event, we now use a modified distribution that also considers the state of a second continuous-time LSTM that has read the future observations from right to left.
  Additionally, we introduce an appropriate evaluation loss metric, and then design a consensus decoder that finds a single lowest risk prediction of the missing events in the whole sequence space by combining the sampled particles (instead of picking one of them).
  [Click to hide summary]
• EMNLP    Learning Latent Semantic Annotations for Grounding Natural Language to Structured Data , Guanghui Qin, Jin-Ge Yao, Xuening Wang, and Chin-Yew Lin , Empirical Methods in Natural Language Processing (EMNLP) , 2018 . ( Oral presentation ) [Click to show summary] [Click to hide summary]
  Research Summary

  This work was done during my internship at Microsoft Research Asia.
  

  Problem

  In this work, we mainly focus on the problem of grounded language acquisition. Specifically, given a series of structured data and text pairs, we aim to establish the correspondence between data and text. For example, a sports reporter might write:
  

  The Raptors edged out the Wizards 120 - 116.

  We also have a table recording the statistics of this game. So we could know that the words "Raptors" and "Wizards" are two team names in the table, and the number "120" and "116" are the total points for two teams. More interestingly, people tend to use various phrases to describe information that is inferred from the data. For instance, the phrase "edged out" above implies that the Raptors beat Wizards with a very narrow margin, which could be inferred from the points difference.
  

  Approach

  We design a hidden semi-Markov model to address this problem. Since there's no additional supervision signal other than statistical occurrences of information between data and text, our model is trained in an unsupervised fashion. Also, to address the notorious "garbage collection" issue, we adopt the posterior regularization technique.
  

  Video

  The video presentation (15 min long) for this work is available on Vimeo, which was delivered by me on EMNLP 2018.
  [Click to hide summary]
• EMNLP    Data2Text Studio: Automated Text Generation from Structured Data , Longxu Dou, Guanghui Qin, Jinpeng Wang, Jin-Ge Yao, and Chin-Yew Lin , Empirical Methods in Natural Language Processing (EMNLP) , 2018 . ( Demo paper ) [Click to show summary] [Click to hide summary]
  Research Summary

  This work was done during my internship at Microsoft Research Asia.

  Abstract

  Data2Text Studio is a platform for automated text generation from structured data. It is equipped with a Semi-HMMs model to extract high-quality templates and corresponding trigger conditions from parallel data automatically, which improves the interactivity and interpretability of the generated text. In addition, several easy-to-use tools are provided for developers to edit templates of pre-trained models, and APIs are released for developers to call the pre-trained model to generate texts in third-party applications. We conduct experiments on RotoWire datasets for template extraction and text generation. The results show that our model achieves improvements on both tasks. [Click to hide summary]

Selected Awards

• 2021 Best Short Paper Awardee, NAACL
• 2019 Outstanding Reviewer, EMNLP
• 2014 Silver medalist at the 31st Chinese Physics Olympiad (CPhO), China Physical Society