Vis enkel innførsel

dc.contributor.authorGrishina, Anastasiia
dc.contributor.authorHort, Max
dc.contributor.authorMoonen, Leon
dc.date.accessioned2024-04-18T09:01:26Z
dc.date.available2024-04-18T09:01:26Z
dc.date.created2024-01-22T09:51:51Z
dc.date.issued2023
dc.identifier.isbn979-8-4007-0327-0
dc.identifier.urihttps://hdl.handle.net/11250/3127168
dc.description.abstractThe use of modern Natural Language Processing (NLP) techniques has shown to be beneficial for software engineering tasks, such as vulnerability detection and type inference. However, training deep NLP models requires significant computational resources. This paper explores techniques that aim at achieving the best usage of resources and available information in these models. We propose a generic approach, EarlyBIRD, to build composite representations of code from the early layers of a pre-trained transformer model. We empirically investigate the viability of this approach on the CodeBERT model by comparing the performance of 12 strategies for creating composite representations with the standard practice of only using the last encoder layer. Our evaluation on four datasets shows that several early layer combinations yield better performance on defect detection, and some combinations improve multi-class classification. More specifically, we obtain a +2 average improvement of detection accuracy on Devign with only 3 out of 12 layers of CodeBERT and a 3.3x speed-up of fine-tuning. These findings show that early layers can be used to obtain better results using the same resources, as well as to reduce resource usage during fine-tuning and inference.
dc.language.isoeng
dc.relation.ispartofESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
dc.titleThe EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
dc.title.alternativeThe EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification
dc.typeChapter
dc.description.versionpublishedVersion
dc.source.pagenumber895-907
dc.identifier.cristin2231633
cristin.ispublishedtrue
cristin.fulltextoriginal
cristin.qualitycode1


Tilhørende fil(er)

Thumbnail

Denne innførselen finnes i følgende samling(er)

Vis enkel innførsel