PCP-tuning：面向小样本学习的个性化连续提示调优

刘汀; 蔡少填; 陈小军; 章秦

doi:10.13568/j.cnki.651094.651316.2023.09.17.0001

您当前的位置：

首页 >

文章列表页 >

PCP-tuning：面向小样本学习的个性化连续提示调优

更新时间：2026-01-21

- PCP-tuning：面向小样本学习的个性化连续提示调优
- Journal of Xinjiang University (Natural Science Edition in Chinese and English) Vol. 41, Issue 1, Pages: 59-68(2024)
- 作者机构：
  
  深圳大学计算机科学与技术系
- 作者简介：
- 基金信息：
- DOI：10.13568/j.cnki.651094.651316.2023.09.17.0001
  CLC： TP391.1;TP18
- Published：2024
- 稿件说明：
移动端阅览
[1]刘汀,蔡少填,陈小军,等.PCP-tuning：面向小样本学习的个性化连续提示调优[J].新疆大学学报(自然科学版)(中英文),2024,41(01):59-68.
[1]刘汀,蔡少填,陈小军,等.PCP-tuning：面向小样本学习的个性化连续提示调优[J].新疆大学学报(自然科学版)(中英文),2024,41(01):59-68. DOI： 10.13568/j.cnki.651094.651316.2023.09.17.0001.

DOI：10.13568/j.cnki.651094.651316.2023.09.17.0001.

摘要

随着“提示学习”的兴起，预训练语言模型在少样本学习中取得了显著的表现，其中的关键问题是如何为每个训练样本构建合适的提示．近年来研究人员提出了一系列提示构造方法，有的构造离散型的提示，有的构造连续型的提示，但通常都是将一个提示应用到整个数据集上．然而，实验结果表明，很难找到一个能够适用于任务中所有样本的提示．为此，提出了一种用于小样本学习的个性化连续型提示调优方法（PCP-tuning），其目的是根据数据集中每个样本的语义来生成个性化的连续型提示．同时，还提出了两种校准技术来控制生成的连续型提示的分布，以获得更好的下游任务表现．最后在10个基准任务上进行大量实验，证明了新方法的优越性能．

Abstract

Pre-trained language models have achieved remarkable performance in few-shot learning with the rise of “prompt learning”

where the key problem is how to construct a suitable prompt for each example. Sample and prompt will be combined as a new input to language model(LM). A series of prompt construction methods have been proposed recently

some of these methods are for discrete prompt construction

and some focus on continuous prompt construction

both of them normally apply a unified prompt to all examples. However

the results show that it is hard to find a perfect unified prompt that works for all examples in a task

one prompt can only help LM assign the correct class to some samples in the downstream classification task and give the wrong result to others. To this end

we propose a novel personalized continuous prompt tuning(PCP-tuning) method to learn personalized prompts that are tailored to each sample's semantic for few-shot learning. Two calibration techniques are proposed to control the distribution of generated prompts for better prompts. Extensive experimental results on ten benchmark tasks demonstrate the superior performance of our method.

关键词

Keywords

references

艾山·吾买尔，魏文琳，早克热·卡德尔．基于Bi LSTM+Attention的体育领域情感分析研究[J]．新疆大学学报(自然科学版)(中英文)，2020,37(2):142-149.AISHAN W,WEI W L,ZAOKERE K.Sentiment analysis based on Bi LSTM+Attention in sports field[J].Journal of Xinjiang University(Natural Science Edition in Chinese and English),2020,37(2):142-149.(in Chinese)

曾蓉，黄德启，魏霞，等．改进WOA优化LSTM神经网络的短时交通流预测[J]．新疆大学学报(自然科学版)(中英文)，2022,39(2):242-248.ZENG R,HUANG D Q,WEI X,et al.Short-term traffic flow forecast based on modified WOA optimized LSTM neural network[J].Journal of Xinjiang University(Natural Science Edition in Chinese and English),2022,39(2):242-248.(in Chinese)

谭勋，吐尔根·依布拉音，艾山·吾买尔，等．基于相似度计算的维吾尔语词聚类[J]．新疆大学学报(自然科学版)，2012,29(1):104-107.TAN X,TUERGEN Y,AISHAN W,et al.Uygur words clustering based on the similarity calculation[J].Journal of Xinjiang University(Natural Science Edition),2012,29(1):104-107.(in Chinese)

亚力青·阿里玛斯，哈力旦·阿布都热依木，陈洋．基于向量空间模型的维吾尔文文本过滤方法[J]．新疆大学学报(自然科学版)，2015,32(2):221-226.YALIQING A,HALIDAN A,CHEN Y.Uygur text filtering based on vector space model[J].Journal of Xinjiang University(Natural Science Edition),2015,32(2):221-226.(in Chinese)

SUI D B,CHEN Y B,MAO B J,et al.Knowledge guided metric learning for few-shot text classification[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2021:3266-3271．

LIU P F,YUAN W Z,FU J L,et al.Pre-train,prompt,and predict:A systematic survey of prompting methods in natural language processing[J].ACM Computing Surveys,2023,55(9):195．

BROWN T B,MANN B,RYDER N,et al.Language models are few-shot learners[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems.December 6-12,2020,Vancouver,BC,Canada.ACM,2020:1877-1901．

SCHICK T,SCH¨UTZE H.Exploiting cloze-questions for few-shot text classification and natural language inference[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics:Main Volume.Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2021:255-269．

SCHICK T,SCH¨UTZE H.It’s not just size that matters:Small language models are also few-shot learners[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2021:2339-2352．

SHIN T,RAZEGHI Y,LOGAN R L,et al.Auto Prompt:Eliciting knowledge from language models with automatically generated prompts[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2020:4222-4235．

GAO T Y,FISCH A,CHEN D Q.Making pre-trained language models better few-shot learners[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers).Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2021:3816-3830．

HAN X,ZHAO W L,DING N,et al.PTR:Prompt tuning with rules for text classification[J].AI Open,2022,3:182-192．

HU S D,DING N,WANG H D,et al.Knowledgeable prompt-tuning:Incorporating knowledge into prompt verbalizer for text classification[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Dublin,Ireland.Stroudsburg,PA,USA:Association for Computational Linguistics,2022:2225-2240．

ZHONG Z X,FRIEDMAN D,CHEN D Q.Factual probing is[MASK]:Learning vs.learning to recall[C]//Proceedings of the2021 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2021:5017-5033．

LI X L,LIANG P.Prefix-tuning:Optimizing continuous prompts for generation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing(Volume 1:Long Papers).Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2021:4582-4597．

LESTER B,AL-RFOU R,CONSTANT N.The power of scale for parameter-efficient prompt tuning[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.Online and Punta Cana,Dominican Republic.Stroudsburg,PA,USA:Association for Computational Linguistics,2021:3045-3059．

GU Y X,HAN X,LIU Z Y,et al.PPT:Pre-trained prompt tuning for few-shot learning[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers).Dublin,Ireland.Stroudsburg,PA,USA:Association for Computational Linguistics,2022:8410-8423．

CHEN J A,YANG Z C,YANG D Y.Mix Text:Linguistically-informed interpolation of hidden space for semi-supervised text classification[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Online.Stroudsburg,PA,USA:Association for Computational Linguistics,2020:2147-2157．

YU M,GUO X X,YI J F,et al.Diverse few-shot text classification with multiple metrics[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 1 (Long Papers).New Orleans,Louisiana.Stroudsburg,PA,USA:Association for Computational Linguistics,2018:1206-1215．

HAN X,ZHU H,YU P F,et al.Few Rel:A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.Brussels,Belgium.Stroudsburg,PA,USA:Association for Computational Linguistics,2018:4803-4809．

BANSAL T,JHA R,MCCALLUM A.Learning to few-shot learn across diverse natural language classification tasks[C]//Proceedings of the 28th International Conference on Computational Linguistics.Barcelona,Spain(Online).Stroudsburg,PA,USA:International Committee on Computational Linguistics,2020:5108-5123．

CHEN T,KORNBLITH S,NOROUZI M,et al.A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning.ACM,2020:1597-1607．

HE K M,FAN H Q,WU Y X,et al.Momentum contrast for unsupervised visual representation learning[C]//2020 IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR).Seattle,WA,USA.IEEE,2020:9726-9735．

JIANG T,JIAO J,HUANG S H,et al.Prompt BERT:Improving BERT sentence embeddings with prompts[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.Abu Dhabi,United Arab Emirates.Stroudsburg,PA,USA:Association for Computational Linguistics,2022:8826-8837．

WANG A,SINGH A,MICHAEL J,et al.GLUE:A multi-task benchmark and analysis platform for natural language understanding[C]//Proceedings of the 2018 EMNLP Workshop Blackbox NLP:Analyzing and Interpreting Neural Networks for NLP.Brussels,Belgium.Stroudsburg,PA,USA:Association for Computational Linguistics,2018:353-355．

HU M Q,LIU B.Mining and summarizing customer reviews[C]//Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.August 22-25,2004,Seattle,WA,USA.ACM,2004:168-177．

VOORHEES E M,TICE D M.Building a question answering test collection[C]//Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.July 24-28,2000,Athens,Greece.ACM,2000:200-207．

WIEBE J,WILSON T,CARDIE C.Annotating expressions of opinions and emotions in language[J].Language Resources and Evaluation,2005,39(2):165-210．

Views

255

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

文本特征和图结点混合增强的图卷积网络文本分类

Related Author

杨晓奇

刘伍颖

Related Institution

广东外语外贸大学信息科学与技术学院

鲁东大学山东省语言资源开发与应用重点实验室

广东外语外贸大学外国语言学及应用语言学研究中心

⁰