2013-06-07 - 版本历史

2013年6月7日 (五) 08:40 Wangd

2013-06-07T08:40:33Z

2013年6月7日 (五) 08:40 Wangd

2013-06-07T08:40:03Z

2013年6月7日 (五) 07:51 Wangd

2013-06-07T07:51:51Z

2013年6月7日 (五) 07:42 Wangd

2013-06-07T07:42:49Z

2013年6月7日 (五) 07:40 Wangd

2013-06-07T07:40:49Z

Wangd：以内容“== Data sharing == * LM count files still undelivered! == DNN progress == === Experiments === * sparse DNN: sticky training (retrain the nnet while keeping the spars...”创建新页面

2013-06-07T07:17:25Z

以内容“== Data sharing == * LM count files still undelivered! == DNN progress == === Experiments === * sparse DNN: sticky training (retrain the nnet while keeping the spars...”创建新页面

新页面

== Data sharing ==

* LM count files still undelivered!

== DNN progress ==

=== Experiments ===
* sparse DNN: sticky training (retrain the nnet while keeping the sparsness)

zero small values(test set: 1900), with extremely sparseness:

{| class="wikitable"
!threshold !! 0 !! 0.2 !!0.3 !! 0.4 !! 0.5
|-
|shrinkage% || 0.0 || 66.4 || 81.6 || 0.90 || 0.94
|-
|without sticky: WER || 7.55 || 9.46 || 53.23 || 98.99 || -
|-
|with sticky: WER || 7.55 || 7.56 || 7.87 || 8.81 || 9.87
|-
|}

The conclusion is that with the L2 retrain, the DNN performance is largely called back. The extremely sparse case (th0.3) with sticky training seems quite amazing. This means the network could be sparse. However this is just for the 1900 test. Need test on other sets.

* fixed-point DNN forwarding

According to the fixed-point FST and NN, and the results of the sparse NN, we are working on fast NN decoder which is suitable for embedded device. The work is just started.

=== Tencent exps ===

本周1000小时实验已结束，实验性能如下：

{| class="wikitable"
!!! old baseline !! new baseline !! DNN
|-
|1900 || 8.4 || 6.8 || 4.3
|-
|2044 || 22.4 || 15.7 || 12.7
|-
|online1 || 35.6 ||32.7 || 25.8
|-
|online2 || 29.6 ||27.3 || 22.1
|-
|map || 24.5 || 15.8 || 13.4
|-
|notepad || 16 || 8.1 || 5.6
|-
|general || 36 || 25.1 || 19.3
|-
|speedup || 26.8 || 14 || -
|-
|}

接下来计划：
*6000小时模型训练，dnn模型相关其他技术（序列化dt，alignment，pretrain）

=== GPU & CPU merge ===

#on progress.

== Kaldi/HTK merge ==

* HTK2Kaldi: hold.
* Kaldi2HTK: hold and second priority

The above work is probably not very necessary since Tencent will fully migrate to the hybrid DNN approach, and therefore HTK will be never used.

== Embedded progress ==

*Status:
: check the reference, and change the compiling options
: the large-scale AM training based on the Tencent 400h data is done.
: the random output problem is fixed.

{| class="wikitable"
! Test Set !! #utt !! PS default !! Tencent
|-
| cw || 993 || 8.01(RT: 0.07) || 7.61(RT: 0.40)
|-
| hfc || 986 || 6.69(RT: 0.07) || 5.48(RT: 0.40)
|-
| zz || 984 || 12.73(RT: 0.07) || 5.91(RT: 0.40)
|-
|}

*To be done
:# large scale parallel training.
:# NN based engine(dynamic and static).

←上一版本		2013年6月7日 (五) 08:40的版本
第64行：		第64行：
	:# NN based engine(dynamic and static).		:# NN based engine(dynamic and static).
	:# Semi-continuous model with the Tencent data		:# Semi-continuous model with the Tencent data
		+	:# Debug on external an ARM board.

←上一版本		2013年6月7日 (五) 07:51的版本
第36行：		第36行：
	#on progress.		#on progress.

		+	== RNN LM progress ==
		+	*Initial work started. 100M data with a 10k vocabulary obtained a perplexity 180.
		+	*More exploration continuous.

	== Embedded progress ==		== Embedded progress ==

@@ 第47行： / 第47行： @@
 ! Sys !! WER !! RT
 |-
-| PS model       || 8.81  || 0.07
+| SP model       || 8.81  || 0.07
 |-
 | Tencent tone   || 6.33  || 0.40

@@ 第30行： / 第30行： @@
 === Tencent exps ===
-*6000小时模型训
+*6000h model training, could be finished on 25th approximately.
 === GPU & CPU merge ===

@@ 第21行： / 第21行： @@
 |}
-The conclusion is that with the L2 retrain, the DNN performance is largely called back. The extremely sparse case (th0.3) with sticky training seems quite amazing. This means the network could be sparse. However this is just for the 1900 test. Need test on other sets.
+Conclusion: The extremely sparse network can largely pertain the performance of DNN. The structure seems more important than the parameter tuning based on the structure.
 * fixed-point DNN forwarding
-According to the fixed-point FST and NN, and the results of the sparse NN, we are working on fast NN decoder which is suitable for embedded device. The work is just started.
+#working on migrating the Atlas lib to ARM.
 === Tencent exps ===
-本周1000小时实验已结束，实验性能如下：
+*6000小时模型训
-−
-{| class="wikitable"
-!!!  old baseline !!   new baseline !!   DNN
-|-
-|1900  ||  8.4 || 6.8 || 4.3
-|-
-|2044  ||  22.4  || 15.7  || 12.7
-|-
-|online1 || 35.6 ||32.7 || 25.8
-|-
-|online2 || 29.6 ||27.3 || 22.1
-|-
-|map || 24.5 || 15.8 || 13.4
-|-
-|notepad || 16 || 8.1 || 5.6
-|-
-|general || 36 || 25.1 || 19.3
-|-
-|speedup || 26.8 || 14 || -
-|-
-|}
-−
-接下来计划：
-*6000小时模型训练，dnn模型相关其他技术（序列化dt，alignment，pretrain）
 === GPU & CPU merge ===
@@ 第59行： / 第36行： @@
 #on progress.
 == Embedded progress ==
@@ 第71行： / 第41行： @@
 *Status:
 : check the reference, and change the compiling options
-: the large-scale AM training based on the Tencent 400h data is done.
+: the large-scale AM training based on the Tencent 400h data is done, continuous HMM.
-: the random output problem is fixed.
 {| class="wikitable"
-! Test Set !! #utt !! PS default !! Tencent
+! Sys !! WER !! RT
 |-
-|  cw  || 993 || 8.01(RT: 0.07) || 7.61(RT: 0.40)
+| PS model       || 8.81  || 0.07
 |-
-|  hfc || 986 || 6.69(RT: 0.07) ||  5.48(RT: 0.40)
+| Tencent tone   || 6.33  || 0.40
 |-
-|  zz  || 984 || 12.73(RT: 0.07) || 5.91(RT: 0.40)
+| Tencent notone || 5.04  || 0.31
 |-
 |}
 *To be done
 :# large scale parallel training.
 :# NN based engine(dynamic and static).