Detail
一: originModel=true, coorModel = true, IndependentQueryNorm = false. (Pattern+STD) (1.0 + 1.0) 正确率 0.6697994987468672
MERT (1.8411764705882354, 1.0) 正确率 0.6785714285714286
(Pattern+STD+ANS)(1.0 + 1.0 + 1.0) 正确率 0.2230576441102756
(50) MERT( 7.772017177156982, 3.268445747426725, 0.030124300025176454)正确率0.3916040100250627
(200) MERT(1.8752954510387578, 1.0, 0.003542933407792176)正确率 0.39035087719298245 二: originModel=false, coorModel = true, IndependentQueryNorm = false. (Pattern+STD) (1.0 + 1.0) 正确率 0.6697994987468672
MERT (1.8411764705882354, 1.0) 正确率0.6785714285714286
(Pattern+STD+ANS)(1.0 + 1.0 + 1.0) 正确率 0.22305764411027568
MERT( 7.772017177156982, 3.268445747426725, 0.030124300025176454)正确率0.3916040100250627
三: originModel=false, coorModel = true, IndependentQueryNorm = true. (Pattern+STD) (1.0 + 1.0) 正确率0.6766917293233082
MERT (1.843531468531468, 1.0) 正确率0.6867167919799498
(Pattern+STD+ANS)(1.0 + 1.0 + 1.0) 正确率0.2518796992481203
MERT(4.913845907274932, 2.897752944939135, 0.1545895322080993)正确率0.39974937343358397
四: originModel=false, coorModel = false, IndependentQueryNorm = true. (Pattern+STD) (1.0 + 1.0) 正确率0.568295739348371
MERT (0.44460517358747553 +1.0) 正确率0.606516290726817
(Pattern+STD+ANS)(1.0 + 1.0 + 1.0) 正确率0.21177944862155387
MERT (2.8368058039272563, 6.88885501252716, 0.04340881443310225) 正确率0.517543859649122
五: 在四的基础上。 lucene每次产生候选1000条。并按sq+pattern打分之和按从大到小排名后的200条: decoder_topN=200 param_topN=200 FINAL lambda: {49.363768749479995, 193.56762839161996, 0.8251869637442989} (01LOSS: 0.5971177944862156) 正确率:0.5081453634085213(结果不等于(1-01loss)的原因是:手动排名打乱了候选的顺序)
lucene每次产生候选1000条。并按sq+pattern打分之和按从大到小排名后的200条: decoder_topN=2 param_topN=100 FINAL lambda: {4.698954979956806, 11.119166816361151, 0.07542784920486184} (01LOSS: 0.5018796992481203) 正确率:0.5175438596491229
具体要求:
一 测试coord对分数的影响:PTN+STD
1. 设置 originModel=false, coorModel = false, IndependentQueryNorm = true. 2. ms += multi_score.get(this.lset.question); ms += " "+multi_score.get(this.lset.standarQuestion); ms += " "+multi_score.get("COORD")/2; // System.out.println(ms); 3. 对三个参数进行优化
result:每一条测试集 候选200条 FINAL lambda: {0.4223541925730876, 1.0, 0.020643059822497145} (01LOSS: 0.48120300751879697) correct:0.5112781954887218 result:每一条测试集 候选150条 param_TOP=10 decoder_TOP=2 FINAL lambda: {0.3939081017607732, 1.0, 1.6504305703337059} (01LOSS: 0.3840852130325815) correct:0.12531328320802004
result:每一条测试集 候选150条 param_TOP=20 decoder_TOP=50 FINAL lambda: {0.38099301738613534, 0.9672129506427846, 1.5614477576833157} (01LOSS: 0.3966165413533834) correct:0.12656641604010024
二 测试coord对分数的影响:PTN+STD
1. 设置 originModel=false, coorModel = false, IndependentQueryNorm = true. 2. ms += multi_score.get(this.lset.question) *( multi_score.get("COORD")/2); ms += " "+multi_score.get(this.lset.standarQuestion) * (multi_score.get("COORD")/2); // ms += " "+multi_score.get("COORD")/2; System.out.println(ms); 3. 对两个参数进行优化
result:150条候选。 FINAL lambda: {0.0, 1.0} (01LOSS: 0.993734335839599)
三 测试coord对分数的影响:PTN+STD+AND
1. 设置 originModel=false, coorModel = false, IndependentQueryNorm = true. 2. ms += multi_score.get(this.lset.question) ms += " "+multi_score.get(this.lset.standarQuestion) ms += " "+multi_score.get("COORD")/3; System.out.println(ms); 3. 对三个参数进行优化
result:150条候选 decoder_topN=2 param_topN=10 FINAL lambda: {8.425969384150346, 19.950351209292734, 1.7159789499221048} (01LOSS: 0.48245614035087714) correct:0.4642857142857143
decoder_topN=20 param_topN=50 FINAL lambda: {1.288935458474548, 3.060315117398841, 0.08348361840550714} (01LOSS: 0.4843358395989975) correct:0.5068922305764411
decoder_topN=50 param_topN=100 FINAL lambda: {0.75686983136428, 1.6890088450820864, 0.07790262435138165} (01LOSS: 0.4906015037593985) correct:0.49373433583959897