GiLT: Augmenting Transformer Language Models with Dependency Graphs
GiLT: Augmenting Transformer Language Models with Dependency Graphs was accepted to the ACL 2026 Main Conference.
Authors: Tianyu Huang, Yida Zhao, Chuyan Zhou, Kewei Tu
Overview
- We proposed a novel method to augment Transformer-based syntactic language models with dependency graphs, especially with word-synchronous semantic dependency graphs, which is to model the dependencies and tokens jointly in a unified framework with a dependency-predicting head so that the original word sequences are kept intact.
- Since the use of syntactic information in past works is limited to recursive structures i.e. trees, and graphs are more general structures which we consider have more potential information to be utilized, based on a codebase of Transformer-XL, we implemented the model architecture which operates on the dependency graphs and word sequences jointly.
- As a contributor, I conducted baseline experiments on metrics such as perplexity (PTB 1987-89 WSJ Treebanks), SG and BLiMP tests.