“Document classification test”版本间的差异

2014年9月9日 (二) 06:15的版本

Problem And Solve

• How to import the sparse data of vsm to weka

Document classification of Sougou data

• DATA

• Data from SougouLab [1],using SogouC.reduced(30M)
• 9-Classes:财经,IT,健康,体育,旅游,教育,招聘,文化,军事
• train and test: train(),test(),dev()

• Text preprocessing

• Segment word using wordlist of 9W.(tencent)
• Remove stop word.stop_wordlist is

• Some Tools

• weka
• scw
• google word2ve
• LDA

• class map

C000007 汽车 C000008 财经 C000010 IT C000013 健康 C000014 体育 C000016 旅游 C000020 教育 C000022 招聘 C000023 文化 C000024 军事

VSM Test

• Data

• dimension:9402

• Method

• document reprenstion: use the tf-idf weight for word weight
• classifier: Native Bayes

• Result

LDA Test

Word2vec Test