(344e) Attention-Based Recurrent Neural Network for Multi-Step-Ahead Prediction

The advances in smart manufacturing, Industrial Internet of Things (IIOT), and data storage technologies have facilitated industries to collect, review, and utilize numerous types and huge amounts of data that could significantly improve plant operation and production of high quality products. The data can be used for different types of analytics including descriptive, diagnostic, and predictive analytics. However, process data are high dimensional, noisy, and dynamic with complex interaction among system variables, which makes data analytics very challenging. Recurrent neural networks (RNNs), a type of model for deep learning, can address some challenging issues effectively.

Motivated by [1, 2], we introduce an attention-based RNN for multi-step-ahead prediction that can have applications in model predictive control, fault diagnosis, etc. This model consists of an RNN that encodes a sequence of input time series data into a new representation (called context vector) and another RNN that decodes the representation into output target sequence. An attention model integrated to the encoder-decoder RNN model allows the network to focus on parts of the input sequence that are relevant to predicting the target sequence. The attention model is jointly trained with all other components of the model. By having a deep architecture, the model can learn a very complex dynamic system, and it is robust to noise. In order to show the effectiveness of the proposed approach, we perform a comparative study on the problem of catalyst activation prediction, by using Nonlinear AutoRegressive with eXogeneous (NARX) predictor, and conventional machine learning techniques such as Support Vector Regression (SVR).

[1] Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014).

[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceeding of the International Conference on Learning Representations (ICLR 2015).

_________________________________________________________________

* All correspondence should be addressed to Prof. Yinlun Huang (Phone: 313-577-3771; E-mail: yhuang@wayne.edu).