基于贝叶斯分类器的主题爬虫研究.pdf

第26卷第 9期 计 算 机 应 用 研 究 V o.l 26 N o. 9 2009年9月 ApplicationR esearch of Computers Sep. 2009 a a­ a, b , , ( 四川大学 a. 计算机学院网络与可信计算研究所; b.信息安全研究所, 成都 610064) : 主题爬虫是实现定题搜索引擎的核心技术提出了基于贝叶斯分类器实现主题爬虫的方法, 介绍了基 于贝叶斯分类器的主题爬虫的系统结构以及系统关键部分的实现, 包括 URL队列爬行历史页面下载以及页 面分析, 并重点介绍了基于贝叶斯分类器的主题相关度算法爬虫使用改进的TF-IDF 算法来提取网页内容的 特征, 并采用贝叶斯分类器计算其主题相关度实验结果表明, 在搜索大量网络资源的情况下,贝叶斯分类器比 PageRank算法更适合用于实现主题爬虫 : 贝叶斯; 分类器; 主题爬虫; 主题相关度 : TP391 : A : 1001-3695( 2009) 09- 3418- 03 doi: 10. 3969 /j. issn. 1001- 3695. 2009. 09. 061 Research on focused craw ler based on Bayes classifier a a­ a, b ZOU Yong-b in , CH EN Xing-shu , WANG W en-xian ( a. N ewt ork T rusted Comp u ting Institu te, Comp u ter Colleg e, b. Institute of Inf orm ation S ecu rity, S ichuan Uni ersity, Ch engdu 610064, Ch i- na) Ab stract: Focused craw ler is the core technology to mi p lement a focused search engine. This paperm ade a research on fo- cused craw ler based on Bayes classifier, introduced the system arch itecture and key mi p lem entation of a focused craw ler based on Bayes classifier. The key parts of the craw ler includeURL queue, craw ling history, page dow nloader and page analyzer. This paper focused on the page rank ing algorithm based on Bayes classifier. U sed an mi provedTF-IDF algorithm to extract the characteristics of the page content, and adopted Bayes classifier to compute the page rank. The expermi ent results show that Bayes classifier is a better choice than PageRank algorithm to mi p lem ent a focused craw ler, under the cond ition of large net- w ork of resources. K ey words: Bayes; classifier; focuse