RNNS are useful in working with any data where order matters. Examples include:
This blog will provide a little background, describe the ways you can create RNNs in SAS Viya, and provide example code.
RNNs are called recurrent because the network feeds back on itself. RNNs are built with Long Short-Term Memory (LSTM) models or gated recurrent units (GRUs) One advantage of LSTM models is that they can “remember” dependencies for long periods of time.
This process is explained in detail in an excellent 30-minute video by Brandon Rohrer. If you have time, watch the video. If not, the next 3 images provide a quick synopsis.
A recurrent neural network starts with new information which runs through a neural network. Then a “squashing” function (activation function, link function) is used to map the output of a neuron to a limited range. I don’t know who first coined the term squashing function, but I love it. Very descriptive. You are essentially taking any real number and squasing it down to a specific limited range. Commonly used squashing functions are:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Both of these functions are used in the full RNN LSTM process. Hyperbolic tangent is commonly used to squash the output from a neural network to a number from -1 to 1. This number from -1 to 1 is now your prediction and can also be fed back into the neural network again as shown below.
A recurrent neural network uses long short-term memory process to make decision in multiple steps. The essential function for making these decisions (selection, forgetting, and ignoring) is a gate. Mathematically, the gate is simple a multiplication function where the output of the previous step is multiplied by a number. Multiplication times zero is a closed gate; that output is stopped right there. Multiplication by 1 is a fully open gate; the output advances fully intact. You can also have a halfway open gate by multiplying by 0.5, and so on.
An RNN with LSTM has:
This is illustrated conceptually below:
RNNs in SAS Viya
SAS Viya has two CAS action sets that allow you to build, train, score, and export RNNs.
The deepRNN action set is designed specifically for RNNs. It uses one of two loss functions:
The deepRNN action set includes 3 actions as shown below.
The deepLearn action set is highly flexible and lets you build many types of neural networks. With respect to RNNS, the deepLearn action set supports both long short-term memory (LSTM) and gated recurrent unit (GRU) RNNs.
RNN Coding in Python Using SWAT to Run CAS Actions
To build your own RNN layer by layer using the deepLearn action set, you would first start a session and load the action set.
sess = swat.CAS('sas-cas-server-default-bin', portnumber, 'gatedemoXXX', 'lnxsas', caslib="casuser") sess.loadactionset('deepLearn')
Then you would build an LSTM model (here the model is named classifer)
sess.buildmodel(model=dict(name='classifier', replace=True ), type='RNN' )
Next add an input layer
sess.addlayer(model='classifier', name='data', layer=dict(type='input') )
Add your RNN layers
sess.addlayer(model='classifier', name='rnn11', srclayers=['data'], layer=dict(type='recurrent', n=50, init='xavier', rnnType='LSTM', act='TANH', outputType='samelength', reverse=False ) ) sess.addlayer(model='classifier', name='rnn21', srclayers=['rnn11'], layer=dict(type='recurrent', n=50, init='xavier', rnnType='LSTM', act='TANH', outputType='samelength', reverse=False ) ) sess.addlayer(model='classifier', name='rnn31', srclayers=['rnn21'], layer=dict(type='recurrent', n=50, init='xavier', rnnType='LSTM', act='TANH', outputType='encoding', reverse=False ) )
Add your output layer; use a softmax activation function.
sess.addlayer(model='classifier', name='outlayer', srclayers=['fc1'], layer=dict(type='output', act='SOFTMAX' ) )
(The above code is from SAS Viya documentation )
Recall that the softmax function takes a vector of scores and transforms it to a vector of values between 0 and 1 that sum to 1.
RNNs on GPUs
SAS supports using GPUs for recurrent neural networks to reduce processing time, but keep in mind that the underlying algorithm is slightly different when run on GPUs than when on CPUs
There are a number of requirements, including:
Dilated RNNs Available via SAS Viya dlModelZoo
SAS Viya dlModelZoo provides a variety of predefined PyTorch models that can be used OOTB, including dilated RNN. Dilated Rnns are designed to help address RNN training issues, such as complex dependencies, vanishing gradients, and exploding gradients. Dilated RNNs are also easily parallelized, and may require fewer parameters.
For More Information
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.