Approaches Based on Markovian Architectural Bias in Recurrent Neural Networks

Recent studies show that state-space dynamics of randomly initialized recurrent neural network (RNN) has interesting and potentially useful properties even without training. More precisely, when initializing RNN with small weights, recurrent unit activities reflect history of inputs presented to the network according to the Markovian scheme. This property of RNN is called Markovian architectural bias. Our work focuses on various techniques that make use of architectural bias. The first technique is based on the substitution of RNN output layer with prediction model, resulting in capabilities to exploit interesting state representation. The second approach, known as echo state networks (ESNs), is based on large untrained randomly interconnected hidden layer, which serves as reservoir of interesting behavior. We have investigated both approaches and their combination and performed simulations to demonstrate their usefulness.

• Type: Conference
• Year: 2004
• Where: SOFSEM
• Authors: Matej Makula, Michal Cernanský, Lubica Benusková
• Keywords: Architectural Bias, Markovian Architectural Bias, Recurrent Neural Network, SOFSEM 2004, Theoretical Computer Science