Abstract:

The impressive capabilities of large language models (LMs) are derived from their training data, which is often sourced from the web. However, these datasets also present issues concerning copyright and the lack of consent from data owners. Understanding the degree to which LMs depend on this data, and developing models that adhere to their restriction, remain significant challenges. In this talk, I will first introduce the Open License Corpus (OLC), a new corpus comprising 228 billion tokens of public domain and permissively licensed text. Despite its vast size, models trained solely on the OLC experience performance degradation due to limited domain coverage. We then propose SILO, a new language model that combines a parametric LM with a modifiable nonparametric datastore. This datastore, containing copyrighted books and news, is only accessed during inference. This methodology allows for the use of copyrighted data without directly training on it, enables sentence-level attribution of model outputs to data sources, and provides an opt-out mechanism for data owners. Finally, I will outline our roadmap towards “distributed models”, a set of model components that use data with varying levels of restrictions—such as public domain, copyrighted, or unreleasable data—in different capacities, such as for training, as a datastore, or in other ways.