介紹了一種基于低階矩自適應(yīng)估計(jì)的隨機(jī)目標(biāo)函數(shù)一階梯度優(yōu)化算法Adam。該方法易于實(shí)現(xiàn)，計(jì)算效率高，對(duì)內(nèi)存的要求少，對(duì)梯度的對(duì)角重新縮放是不變的，并且非常適合于數(shù)據(jù)和/或參數(shù)很大的問(wèn)題。該方法也適用于非平穩(wěn)目標(biāo)和具有非常嘈雜和/或稀疏梯度的問(wèn)題。超參數(shù)有直觀的解釋，通常需要很少的調(diào)整。本文討論了一些與相關(guān)算法的聯(lián)系，Adam 正是在這些算法上受到啟發(fā)。我們還分析了算法的理論收斂性，并給出了與在線凸優(yōu)化框架下的最優(yōu)結(jié)果相當(dāng)?shù)氖諗窟z憾界。實(shí)驗(yàn)結(jié)果表明，該方法在實(shí)際應(yīng)用中效果良好，與其它隨機(jī)優(yōu)化方法相比具有一定的優(yōu)越性。最后，我們討論了AdaMax，一個(gè)基于無(wú)窮范數(shù)的Adam變體。

Stochastic gradient-based optimization is of core practical importance in many fields of science and engineering. Many problems in these fields can be cast as the optimization of some scalar parameterized objective function requiring maximization or minimization with respect to its parameters. If the function is differentiable w.r.t. its parameters, gradient descent is a relatively efficient optimization method, since the computation of first-order partial derivatives w.r.t. all the parameters is of the same computational complexity as just evaluating the function. Often, objective functions are stochastic. For example, many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking gradient steps w.r.t. individual subfunctions, i.e. stochastic gradient descent (SGD) or ascent. SGD proved itself as an efficient and effective optimization method that was central in many machine learning success stories, such as recent advances in deep learning (Deng et al., 2013; Krizhevsky et al., 2012; Hinton & Salakhutdinov, 2006; Hinton et al., 2012a; Graves et al., 2013). Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization. For all such noisy objectives, efficient stochastic optimization techniques are required. The focus of this paper is on the optimization of stochastic objectives with high-dimensional parameters spaces. In these cases, higher-order optimization methods are ill-suited, and discussion in this paper will be restricted to first-order methods.

基于隨機(jī)梯度的優(yōu)化方法在許多科學(xué)和工程領(lǐng)域具有核心的實(shí)際意義。這些領(lǐng)域中的許多問(wèn)題都可以轉(zhuǎn)化為對(duì)某個(gè)標(biāo)量參數(shù)化目標(biāo)函數(shù)的優(yōu)化，該函數(shù)的參數(shù)需要最大化或最小化。如果函數(shù)的參數(shù)是可微的，梯度下降法是一種比較有效的優(yōu)化方法，因?yàn)橐浑A偏導(dǎo)數(shù)的計(jì)算與函數(shù)的求值具有相同的計(jì)算復(fù)雜度。通常，目標(biāo)函數(shù)是隨機(jī)的。例如，許多目標(biāo)函數(shù)由在不同數(shù)據(jù)子樣本上求值的子函數(shù)和組成;在這種情況下，優(yōu)化可以通過(guò)采取梯度步驟w.r.t.單獨(dú)的子函數(shù)，即隨機(jī)梯度下降(SGD)或上升來(lái)提高效率。SGD證明了自己是一種高效和有效的優(yōu)化方法，這在許多機(jī)器學(xué)習(xí)成功的故事中都是核心，比如最近在深度學(xué)習(xí)方面的進(jìn)展(Deng et al.， 2013;Krizhevsky等，2012;Hinton & Salakhutdinov, 2006;Hinton等，2012a;Graves 等人，2013)。目標(biāo)也可能有數(shù)據(jù)子采樣之外的其他噪音源，如dropout (Hinton et al.， 2012b)正則化。對(duì)于所有這些有噪聲的目標(biāo)，都需要有效的隨機(jī)優(yōu)化技術(shù)。本文主要研究具有高維參數(shù)空間的隨機(jī)目標(biāo)的優(yōu)化問(wèn)題。在這種情況下，高階優(yōu)化方法是不合適的，本文的討論將局限于一階方法。