Time Series: Recurrent Neural Networks

Recurrent Neural Networks are neural networks with a bidirectional flow architecture, instead of the commonly used feed-forward architecture. This means that each node output will impact its neighbors, regardless of whether they are forward or behind it. RNN’s are a popular choice for time series, because they consider the impacts of time data in both directions, which makes it easy to extrapolate trends and predict into the future.

Common Applications

Common Industries

  • Agriculture

  • Tech

  • Finance

  • Any Industry Using NLP

Common Problem Types

  • Language Recognition (textual, speech or otherwise)

  • Time Series Predictions Into Future

  • Language Translation

  • Robotic Control

A Brief History

While RNN’s have their research roots in statistical mechanics in the 1920’s, Hopfield Nets developed in the 70’s and 80’s, despite a general lack of interest in neural networks (the so called "AI winter"). Hopfield Nets were seen as allowing for associative memory structures, remembering the outputs of all or many nodes- a contrast to the feed forward architecture, which didn’t need to remember what previous nodes' outputs were, just the current and previous layer nodes. The late 1990’s came around and along with other neural network R&D, LSTM (Long Short-Term Memory) networks were developed; its success was partially due to its ability to handle much larger sequences of time-steps, along with the development of computing power since the 1970’s. LSTM is used heavily in NLP, where it is only recently being challenged by transformer models (such as the well known ChatGPT: "Chat Generative Pre-trained Transformer"). RNN’s are still heavily used wherever there is time series data, as well as NLP.

Code Examples

All of the code examples are written in Python, unless otherwise noted.

Containers

These are code examples in the form of Jupyter notebooks running in a container that come with all the data, libraries, and code you’ll need to run it. Click here to learn why you should be using containers, along with how to do so. 
#pull container, only needs to be run once
docker pull ghcr.io/thedatamine/starter-guides:time-series-rnn

#run container
docker run -p 8888:8888 -it ghcr.io/thedatamine/starter-guides:time-series-rnn

Need help implementing any of this code? Feel free to reach out to datamine-help@purdue.edu and we can help!

Resources

All resources are chosen by Data Mine staff to be of decent quality, and most if not all content is free.

Websites

Recurrent Neural Networks (IBM)

Recurrent Neural Networks Cheatsheet (Stanford)

Recurrent Neural Networks (Towards Data Science)

What is a Recurrent Neural Network? (MathWorks)

Videos

Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) (~25 minutes)

What is Recurrent Neural Network (RNN)? (~16 minutes)

Recurrent Neural Networks : Data Science Concepts (~27 minutes)

What is LSTM (Long Short Term Memory)? (~8 minutes)

Books

Introduction to Statistical Learning (ISL)

Also known as the "machine learning bible", this book is very well known and highly recommended and very clear, and has numerous code projects attached with each chapter. See Chapter 10.5.

Recurrent neural networks: from simple to gated architectures (2022)

Gentle introduction for those new to neural networks, but robust in its technical explanation for those who want to go further.

Recurrent neural networks: concepts and applications (2023)

Great for visual learners; great introduction and overview of the applications of RNN’s that goes deep for those interested.

Articles

Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements (1972)

Seminal paper on RNN’s from the 1970’s that advanced RNN’s and culminated in Hopfield Networks.