题目/Title：HW-ADAM: FPGA-Based Accelerator for Adaptive Moment Estimation
作者/Author：
Weiyi Zhang, Liting Niu, Debing Zhang, Guangqi Wang, Fasih Ud Din Farrukh, Chun Zhang
期刊/Journal：ELECTRONICS
年份/Issue Date：2023Jan.
卷（期）及页码/Volume（No.）&pages：Vol.12, No.2, pp.1-17
摘要/Abstract：
The selection of the optimizer is critical for convergence in the field of on-chip training. As one second moment optimizer, adaptive moment estimation (ADAM) shows a significant advantage compared with non-moment optimizers such as stochastic gradient descent (SGD) and first-moment optimizers such as Momentum. However, ADAM is hard to implement on hardware due to the computationally intensive operations, including square, root extraction, and division. This work proposed Hardware-ADAM (HW-ADAM), an efficient fixed-point accelerator for ADAM highlighting hardware-oriented mathematical optimizations. HW-ADAM has two designs: Efficient-ADAM (E-ADAM) unit reduced the hardware resource consumption by around 90% compared with the related work. E-ADAM achieved a throughput of 2.89 MUOP/s (Million Updating Operation per Second), which is 2.8× of the original ADAM. Fast-ADAM (F-ADAM) unit reduced 91.5% flip-flops, 65.7% look-up tables, and 50% DSPs compared with the related work. The F-ADAM unit achieved a throughput of 16.7 MUOP/s, which is 16.4× of the original ADAM.