最佳实践：深度学习用于自然语言处理（二）

待字闺中 · 公众号 · 程序员 · 2017-08-21 06:23

正文

（continue）

Attention

Attention is most commonly used in sequence-to-sequence models to attend to encoder states, but can also be used in any sequence model to look back at past states. Using attention, we obtain a context vector

$c i = \sum j a i j s j a i = softmax (f a t t (h i, s j)) " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

The attention function $f a t t (h i, s j) " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ fatt(hi,sj) calculates an unnormalized alignment score between the current hidden state $h i " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ hi and the previous hidden state $s j " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

Wa are learned attention parameters. Analogously, we can also use matrices $W 1 " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ W2 to learn separate transformations for $h i " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

Additive and multiplicative attention are similar in complexity, although multiplicative attention is faster and more space-efficient in practice as it can be implemented more efficiently using matrix multiplication. Both variants perform similar for small dimensionality $d h " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ dh of the decoder states, but additive attention performs better for larger dimensions. One way to mitigate this is to scale $f a t t (h i, s j) " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

Attention cannot only be used to attend to encoder or previous hidden states, but also to obtain a distribution over other features, such as the word embeddings of a text as used for reading comprehension (Kadlec et al., 2017) [37]. However, attention is not directly applicable to classification tasks that do not require additional information, such as sentiment analysis. In such models, the final hidden state of an LSTM or an aggregation function such as max pooling or averaging is often used to obtain a sentence representation.

Self-attention Without any additional information, however, we can still extract relevant aspects from the sentence by allowing it to attend to itself using self-attention (Lin et al., 2017) [18]. Self-attention, also called intra-attention has been used successfully in a variety of tasks including reading comprehension (Cheng et al., 2016) [38], textual entailment (Parikh et al., 2016) [39], and abstractive summarization (Paulus et al., 2017) [40].

We can simplify additive attention to compute the unnormalized alignment score for each hidden state $h i " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

In matrix form, for hidden states $H = h 1, \dots, h n " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ H=h1,…,hn we can calculate the attention vector $a " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ a and the final sentence representation $c " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

Rather than only extracting one vector, we can perform several hops of attention by using a matrix $V a " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ va, which allows us to extract an attention matrix $A " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

Key-value attention Finally, key-value attention (Daniluk et al., 2017) [19] is a recent attention variant that separates form from function by keeping separate vectors for the attention calculation. It has also been found useful for different document modelling tasks (Liu & Lapata, 2017) [41]. Specifically, key-value attention splits each hidden vector $h i " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ [ki;vi]=hi. The keys are used for calculating the attention distribution $a i " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

$a i = softmax (v a ⊤ tanh (W 1 [k i - L; \dots; k i - 1] + (W 2 k i) 1 ⊤)) " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

L is the length of the attention window and $1 " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$ 1 is a vector of ones. The values are then used to obtain the context representation $c i " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

ci is used together with the current value $v i " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

The optimization algorithm and scheme is often one of the parts of the model that is used as-is and treated as a black-box. Sometimes, even slight changes to the algorithm, e.g. reducing the $β 2 " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

Optimization algorithm Adam (Kingma & Ba, 2015) [21] is one of the most popular and widely used optimization algorithms and often the go-to optimizer for NLP researchers. It is often thought that Adam clearly outperforms vanilla stochastic gradient descent (SGD). However, while it converges much faster than SGD, it has been observed that SGD with learning rate annealing slightly outperforms Adam (Wu et al., 2016). Recent work furthermore shows that SGD with properly tuned momentum outperforms Adam (Zhang et al., 2017) [42].

Optimization scheme While Adam internally tunes the learning rate for every parameter (Ruder, 2016) [22], we can explicitly use SGD-style annealing with Adam. In particular, we can perform learning rate annealing with restarts: We set a learning rate and train the model until convergence. We then halve the learning rate and restart by loading the previous best model. In Adam's case, this causes the optimizer to forget its per-parameter learning rates and start fresh. Denkowski & Neubig (2017) [23] show that Adam with 2 restarts and learning rate annealing is faster and performs better than SGD with annealing.

Ensembling is an important way to ensure that results are still reliable if the diversity of the evaluated models increases (Denkowski & Neubig, 2017). While ensembling different checkpoints of a model has been shown to be effective (Jean et al., 2015; Sennrich et al., 2016) [51, 52], it comes at the cost of model diversity. Cyclical learning rates can help to mitigate this effect (Huang et al., 2017) [53]. However, if resources are available, we prefer to ensemble multiple independently trained models to maximize model diversity.

Rather than pre-defining or using off-the-shelf hyperparameters, simply tuning the hyperparameters of our model can yield significant improvements over baselines. Recent advances in Bayesian Optimization have made it an ideal tool for the black-box optimization of hyperparameters in neural networks (Snoek et al., 2012) [56] and far more efficient than the widely used grid search. Automatic tuning of hyperparameters of an LSTM has led to state-of-the-art results in language modeling, outperforming models that are far more complex (Melis et al., 2017).

Learning the initial state We generally initialize the initial LSTM states with a $0 " role="presentation" style="font-variant: inherit; font-stretch: inherit; line-height: normal; font-family: inherit; vertical-align: baseline; border-width: 0px; border-style: initial; border-color: initial; display: inline; word-spacing: normal; word-wrap: normal; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px;">$

Gradient norm clipping One way to decrease the risk of exploding gradients is to clip their maximum value (Mikolov, 2012) [57]. This, however, does not improve performance consistently (Reimers & Gurevych, 2017). Rather than clipping each gradient independently, clipping the global norm of the gradient (Pascanu et al., 2013) [58] yields more significant improvements (a Tensorflow implementation can be found here).

最佳实践：深度学习用于自然语言处理（二）

正文

（continue）

Attention

Optimization

Ensembling

Hyperparameter optimization

LSTM tricks