基于λ-主动学习方法的中文微博分词EnhancingMicroblog-oriented.PDF
文本预览下载声明
文章编号:1003-0077 (2011)00-0000-00
基于 -主动学习方法的中文微博分词
张婧, 黄德根, 黄锴宇, 刘壮, 孟祥主
(大连理工大学 计算机科学与技术学院,辽宁省 大连市 116024)
摘要:由于面向中文微博的分词标注语料相对较少,导致基于传统方法和深度学习方法的中文分词系统在
微博语料上的表现效果很差。针对此问题,本文提出一种新的主动学习方法从大规模未标注语料中挑选更
具标注价值的微博分词语料。该方法根据微博语料的特点,在主动学习迭代过程中引入参数 来控制所选
的重复样例的个数,确保了所选样例的多样性;同时,根据样例中字标注结果的不确定性和上下文的多样
性,采用Max 、Avg 和AvgMax 三种策略衡量样例整体的标注价值;此外,用于主动学习的初始分词器除
了使用当前字的上下文作为特征外,还利用字向量自动计算当前字成为停用字的可能性作为模型的特征。
实验使用NLPCC 2015 公开的训练语料和测试语料,结果表明,本文提出的基于主动学习的分词方法,其
F 值较基线系统提高了0.84%~1.49% ,与目前最优的WBA 主动学习方法相比提升效果更加显著。
关键词:中文分词;主动学习;样例多样性;微博语料
中图分类号:TP391 文献标识码:A
Enhancing Microblog-oriented Chinese Word Segmentation with
-Active Learning Method
ZHANG Jing, HUANG Degen, HUANG Kaiyu, LIU Zhuang, MENG Xiangzhu
(School of Computer Science and Technology, Dalian University of Technology, Dalian, Liaoning
116024, China)
Abstract: The manual segmented microblog-oriented corpora are inadequate, which is the reason that the
performance of both the conventional Chinese word segmentation (CWS) systems and the deep learning based
CWS systems is still unsatisfactory. To address this problem, we propose a novel active learning method to
effectively select samples with high annotation value from unlabelled tweets for microblog-oriented CWS.
Considering to the characteristics of microblog data, the parameter is introduced in the procedure of measure
the context diversity of the characters to control the number of the repeatedly selected samples. Furthermore, three
strategies (Max, Avg and AvgMax) are also used to evaluate the overall values of a sample taking advantages of
the uncertainty confidence and the context diversity of the charact
显示全部