Parameter-Efficient Tuning as a Response to Corporate Capture in Natural Language Processing

Abstract

Large Language Models (LLM) like ChatGPT have achieved remarkable performance in recent years on an impressive breadth of tasks. The promise of an untapped market has led industry to become significant players in the research field responsible for LLMs, Natural Language Processing (NLP). Industry presence has garnered criticism regarding conflicts of interests and academic independence. Debates on industry involvement in NLP are often intertwined with discourse on the size of state-of-the-art LLMs. As the name would suggest, LLMs are quite large, and it is often unwieldy to perform benchmark-validated research on or with them for institutions with smaller computational budgets such as universities and small businesses. Because benchmarks are highly valued in NLP, partnering with industry and gaining access to their computing resources gives a distinct advantage for publishing. Collaboration with industry becoming integral to staying relevant in NLP research begs the question: At what point does collaboration become corporate capture in the absence of alternatives? The first manuscript chapter of this thesis provides data to help answer this question. We surveyed papers published at a recent NLP conference, EMNLP 2022, to analyze the contributions and artifacts (i.e. datasets, benchmarks, and LLMs) that today's papers base their research on. We find an overwhelming reliance on contributions and artifacts from industry despite their relatively lower publication rates. We posit that when collaboration becomes necessity, the relationship with industry may be better described as corporate capture. Regardless of how we classify the relationship, the conflicts of interest that inevitably arise when researchers from institutions with different priorities come together need to be addressed. Our discussion provides suggestions for how NLP research can better navigate such ethical considerations. Having identified some of the ethical complications NLP researchers face when formulating their research questions, the second manuscript chapter of this thesis provides a case study for how to mitigate their impact from within the research community. We develop a parameter-efficient tuning (PET) method, prefix-propagation, ideal for long-document models. Before the introduction of our method, long document models interfaced poorly with PET. By using prefix-propagation, PET performance on classification tasks equals the less efficient method of fine-tuning. Importantly, PET methods reduce the computational requirements of adopting LLMs by roughly half. By framing our research this way, we intentionally avoided needing to collaborate with industry for their resources and thereby were able to avoid risk of conflicts of interest and corporate capture.