This post is the third, and last, post in this series focused on attention and large language models. Although this post can be read as a standalone post, it may be helpful to review the first and second parts of this series. This post will focus on explaining how the transformer architecture is the basis for modern LLMs. Decoder based LLMs, such as the GPT models, and encoder-based models, such as the BERT family of models are both based on the transformer architecture.   From Transformers to LLMs   The transformer model is one of the most influential architectures proposed. Thanks to advancements in computing, basing the model off attention, and the ability to massively parallelize computations, the door to building language models way more powerful than anything seen before is now open. The original transformer architecture largely focused on machine translation; however, by applying modifications to the layers appearing after the decoder output, the architecture can be applied to a variety of different tasks. As a result, the transformer architecture is the basis for modern large language models including the family of GPT models, BERT, LLaMA, Gemini, and many others.   A lot of the differences between LLMs are due to the volumes of data used, the number of layers, the number of parameters, training times, and the training objectives. LLMs architectures are similar with regards to the type of layers and connections that we see. Understanding the transformer architecture can help you understand how modern LLMs work.   Key LLM Ideas   There are two key ideas that gave rise to modern LLMs, pre-training and fine tuning. Most deep learning methods require substantial amounts of manually labeled data, which restricts their applicability in many domains that do not have much annotated data. Prior research showed that models that can leverage linguistic information from unlabeled data could be used as an alternative to gathering more annotated data. The goal of one of the first transformer based large language models, the Generative Pre-trained Transformer, also known as GPT, was to learn a universal representation that transfers to a wide range of tasks with little adaptation.   These models are effectively semi-supervised because they are pre-trained with a large language modeling objective on unlabeled data, and then they could be fine-tuned on a supervised objective with minor, or no, architectural modifications. Fine tuning does assume that a labeled data set consisting of input tokens and a label are used. Most of the work on language models prior to the development of the transformer architecture generally required major architectural changes to apply a language model to a different supervised task. The core idea in the development of modern LLMs is that with enough pre-training these models can be easily applied to variety of different tasks.   GPT-1 and BERT   Two models that are directly based on the original transformer architecture are the first GPT model and the BERT model. Both models were published a few months apart in 2018 and were two of the first transformer-based large language models. It is worth noting that at the time, both models achieved state of the art performance on a variety of benchmark data sets.​     ​Figure 1. GPT-1 (Left) and BERT (Right) Architectures.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In terms of their architectural design, both models are noticeably simpler than transformers. This is because GPT models are decoder-only models, while BERT, and its variations, are encoder-only models. The GPT decoder is even simpler than the transformer decoder because it does not include the second multi-head attention layer. ​   Comparing the model architectures of these two models side by side, it might be a bit difficult to immediately spot any differences aside from the positional embedding. Let's discuss these two models in a bit more detail.​   GPT-1   GPT stands for generative pre trained transformer [2]. These models are generative, meaning that they produce sequences of tokens. The decoder in the transformer architecture is what made the transformer autoregressive. Since GPT is a decoder-only model, this model is also an autoregressive model. In this case autoregressive means that the prediction for the current token is a function of the previous tokens.   Pre-trained indicates that the model was trained using a large corpus of text on a standard language modeling objective, similar to what was used in the original transformer. The objective being, given the previous tokens in a sequence, predict the next token.   Lastly the T stands for transformer. In summary, the GPT model is a decoder-only model based off the transformer architecture which replaces recurrences with multi-head attention.   GPT-1 Architecture     Figure 2. Detailed GPT-1 Architecture.   The architecture visualization may make it seem as if these are small models, but it is worth noting that there are 12 layers in this decoder only transformer. Expanding that out a little bit, this implies 12 multi-head attention layers. The rest of the architecture hyperparameters are not much different from what we saw in the original transformer. The embedding dimensions is 768, but instead of using 8 attention heads per multi-head attention layer, GPT uses 12. All in all, this model has a grand total of 117 million learnable parameters.   The last few layers after the decoder output can be changed so that it can adapt to a variety of different tasks. By default, the output of this model is the probability of a particular token given the inputs. A softmax layer could also be applied to the output to use the model for classification.   GPT-1 Pre-Training   The BookCorpus dataset was used for pre-training. It contains 7,000 unpublished fiction books from various genres. This dataset was chosen because it contains long passages that could help the model learn dependencies between distant information.   The pre-training was performed with no specific task. The goal was to create a high-capacity language model that could be tuned without having to make major architectural changes. If the model was pre-trained enough with a large volume of data, that may help the model understand some of the semantics of language. Afterwards, the model could be fine-tuned with just a few epochs for a specific task.   The rationale for this approach is that all language tasks require some knowledge of language in the first place. If you think of an RNN, the initial weight updates would effectively allow the model to learn some of the semantics of language necessary before the model is able to perform the specific supervised task. This is something that we see often in our daily lives. As an example, before a child is able to play a sport, the child must learn to walk, develop finer motor skills, understand language, and understand the concept of rules before they're able to play a sport. Similarly, the goal of pre-training is for the model to learn the semantics of language before they're then trained to perform a supervised task.   Figure 3. Pre-Training Objective.   The pre training objective is simply a standard language modeling objective of maximizing likelihood. The objective is to maximize the probability of correctly predicting the next token in the sequence using a neural network with parameters theta. The model was trained using ADAM optimization.   Fine-Tuning   Given GPT’s pre-training, the model can be tuned to generate different types of outputs by passing the decoder output to a softmax layer, then training the model for a few epochs. In various experiments, researchers found that 3 epochs worth of training is sufficient in most cases. Generally speaking, the hyper-parameter settings from the unsupervised pre-training phase are used in fine-tuning tasks. Another clever approach to avoid having to customize the model architecture too much was to restructure the inputs for fine tuning tasks.   Input Transformations   The inputs to the model can be restructured in specific ways to construct sequences that can be used for fine tuning. Text inputs can be transformed to handle a variety of different tasks such as classification, textual entailment, similarity, question and answering, etc.     Figure 4. Input Restructuring for Classification.   For simple tasks like classification, not much needs to be done outside of adding a linear layer followed by a softmax. For other tasks such as similarity, and question and answering, sequences of text can be concatenated by using a delimiter token in between two sequences of text. The delimiter acts as a breaking point for the sequences.     Figure 5. Input Restructuring for Similarity Tasks.   In similarity tasks where sentences are compared, the sentences have no inherent ordering associated with them. Since there is no ordering, the sequences can be concatenated twice but the ordering of the texts can be flipped in the concatenations. These sequences can then be passed through the decoder independently and then added element wise prior to being fed to one final linear layer and softmax.     Figure 6. Input Restructuring for Q&A.   A question and answering task requires a document, question, and a set of possible answers, to be passed through the decoder. The document context and question is concatenated, with each possible answer adding a delimiter token in between. Each of the sequences are processed independently through the decoder. The softmax layer produces an output distribution of possible answers.     Figure 7. Fine-Tuning Objective.   The fine-tuning modeling objective is also changed since the parameters need to be adapted to the new supervised task. The likelihood of the conditional probability of a label, y, given the sequence of input tokens is maximized. Research also found that adding an auxiliary objective to fine tune the model was useful. This helps the model achieve better generalization and accelerates convergence.   These are just a few examples of ways in which inputs can be re-structured to fine tune these models. In smaller/older LLMs, fine-tuning can still be a viable approach to improve model performance for certain downstream tasks. That being said, it is worth noting that modern LLMs, such as GPT-3, GPT-4, Gemini, and more are not open source, which makes it harder to have this level of flexibility with how they can be fine tuned. Even if they were open source, they are so large that fine tuning is infeasible without access to massive amounts of resources. Fine-tuning these modern LLMs is often reserved for publishers of these models and institutions with the time, resources, and data to be able to do so. As a result, modern LLMs are often improved for specific tasks via different techniques such as prompting and retrieval augmented generation systems. Both of these topics are outside the scope of this post but will be explored in a future ones.   BERT   The other LLM that we will briefly discuss is BERT, which stands for “Bidirectional Encoder Representations from Transformers” [3]. The B in BERT stands for bidirectional, this comes from the fact that the language modeling objective isn't just predicting the next word in the sequence. In fact, the pre-training objective is to predict a random word in the sequence using all the other words in the sequence. So maybe if we have the sentence "The gray dog wagged her tail", we could try to predict the word "dog" using all other words in the sequence. ​   The ER in BERT stand for "Encoder Representations". BERT is an encoder-only model, and its objective is to extract information from text inputs. This extracted information can then be used to accomplish different tasks, like text classification or sentiment analysis, by performing minor adaptations to the architecture. Unlike GPT, BERT is not an autoregressive model that generates text. This is part of the reason why the language modeling objective can attend to words in both directions. Because it is not trying to predict future words in a sequence, the entire sequence of words can be used to extract useful information. ​   The T in BERT also stands for "Transformer“ because it uses the transformer architecture proposed in the "Attention is All you Need" [1] paper. Although there are differences in the training approach, the model architecture for BERT’s encoder is not too different from the architecture of the original transformer.   BERT Encoding   Figure 8. BERT Architecture.   BERTs architecture also consists of multiple stacked encoders that process the embedded text to create transformed representations of the inputs. Like GPT, BERT is also pre-trained on a language modeling objective and can be subsequently fine-tuned to perform different tasks. BERT is pre-trained using the BookCorpus data set with two training objectives, language modeling and next sentence prediction. BERT's parameters can be fine-tuned end to end by plugging task-specific inputs and targets into the model. Compared to pre-training, fine tuning is relatively inexpensive. Although encoder-only models cannot be used to generate text by default, the encodings created by the model can be used for tasks like text classification, question answering (like multiple-choice questions), and name entity recognition. It should be noted that the encodings can be passed to decoder layers, which can be used to generate sequential outputs. ​   The encoding blocks consist of a combination of multi-head attention layers and fully connected layers. These are connected with residual and batch normalization layers. Attention layers enable the model to learn long-range relationships between sequential elements, and the rest of the encoder block is designed to efficiently extract this information and then convert it into a useful form (a form that is useful in predicting the target). ​   BERT Pre-Training   The pre-training objective for many decoder-based models, such as GPT, is to predict the probability of the next token in a sequence. This is useful in decoder-based models because we want to generate syntactically correct text. The BERT language modeling task is an important part of its pre-training because it enables the model to learn what words mean and how they are used in text. When a person reads or listens, we have access to the entire sequence to extract information to generate some sort of response. This is similar to what BERT models attempt to do during pre-training by accessing an entire sequence and exploring the interrelatedness of all words in said sequence. ​     Figure 9. BERT Language Modeling Task.   To enable this pre-training, we select a random token from each sentence in the training data and replace it with a placeholder token (called the [MASK] token). The models is trained to predict the replaced token (via supervised training). The problem with this approach is that the model will never actually see real text input data with the special [MASK] token, so when preparing the training data for pre-training, we include the [MASK] token in 80% of the training data. For the rest of the training data, we leave the [MASK] token out, replacing the token of interest with a random token 10% of the time, and leaving the true token in the training data 10% of time. This prepares the model to see data that does not include the [MASK] token.​   Many language tasks such as question answering and natural language inference are based on understanding the relationships between two sentences. These relationships are not things directly captured by language modeling nor by the decoder pre-training employed in the GPT-1 model. This is why the next sentence prediction task is an important part of pre-training the BERT model. Without it, the model would not have a good understanding of how sentences are composed together.     Figure 10. Next Sentence Prediction Pre-Training.   The model is trained to predict whether sentence A is followed in the corpus by sentence B. This is a simple binary classification task, where the model predicts whether a pair of sentences A and B should have the label [IsNext], meaning that sentence B directly follows sentence A, or the label [NotNext], meaning sentence B is either a random sentence unrelated to sentence A or sentences are missing between them. The data is prepared such that 50% of the training pair have the label [IsNext] and 50% have the label [NotNext].   In this example, the first pair of sentences appear in order in Shakespeare's Romeo and Juliet, but the second pair of sentences are not contiguous in the text. Instead, they are pulled randomly from the sentences in the play.​   BERT and GPT-1 Differences   Now that we have discussed both GPT-1 and BERT, let's come back and discuss some of the differences.   Although both models are based off the transformer architecture, they are fundamentally different. GPT models are decoder only models whereas BERT and its variations are encoder only. GPT-1 was trained with a language modeling objective that is unidirectional whereas BERT is bidirectional. This is a major difference between the design of both models, because in GPT the prediction for a given position can only depend on the outputs that came prior to it. Whereas in BERT everything in the sequence attends to itself without any restriction, the modeling objective randomly masks certain words, and every other token in a sequence is used to predict the masked word. These differences make GPT better for tasks that require syntactically accurate text generation including dialogue, creative writing, and more. Whereas BERT is better suited for tasks like sentiment analysis, language understanding, and named entity recognition.   There are also other architectural differences, such as the positional embeddings, the number of layers, hyperparameters, and optimization settings. In the original BERT paper, one BERT model was built using an architecture that closely matched GPT‘s, and one that was a larger version of BERT. The GPT-1 model had 117 million parameters, while the BERT model had a total of 345 million parameters.   BERT Variations   The first BERT model was published by Google in 2018 and set state-of-the-art performance on a variety of language tasks. The model was quickly integrated to Google to improve their search functionality. Unlike the family of GPT models, there isn’t a family of ever-growing BERT models published by Google. Many of the variations of BERT have been published by different institutions considering that BERT is open source. A lot of models based off BERT have focused on ways to improve its efficiency and reduce its size so that it can be more easily fine-tuned and integrated into existing applications.   One such variation of BERT is RoBERTa which was published in 2019 [4]. This model was constructed during a replication study of BERTs pre-training to carefully measure the impact of hyperparameters and training data size. The study found that BERT was very undertrained, and its performance could be improved via better hyperparameter tuning. The RoBERTa model didn't do much in terms of architectural changes or introduce many new techniques relative to BERT. When evaluated on the same data sets that BERT was, it did achieve state-of-the-art results at the time.   Another variation is ALBERT, which was published in late 2019, early 2020 [5]. The goal of ALBERT was to function as a lite BERT model. As model sizes increases, more GPU (or TPU) memory is required and training times become longer. It used techniques like factorized embedding parametrization, cross-layer parameter sharing, and an additional loss to reduce the total number of learnable parameters. These changes allow the model to remain smaller than BERT even while using larger feed forward components in the architecture of the network.   The last variation that we will discuss is DistilBERT, which was published in 2020 [6]. This model reduced the size of BERT by 40% while retaining 97% of it’s language understanding and was 60% faster. The model achieved this by using a technique called knowledge distillation in which a compact model (also known as student) is trained to reproduce the behavior of a larger model (the teacher). This model added a distillation loss to the training objective such that the final loss was a linear combination of the masked language modeling loss and the distillation loss. Additional architectural changes that included removing the token type embeddings, removing the pooler, and reducing the number of layers by 2 helped the model achieve this level of performance.   Other notable works on encoder-based transformers exists, however these are beyond the scope of this post.   GPT Timeline   The GPT family of models is published by open AI in partnership with Microsoft. The architectures for GPT-1 and GPT-2 are overall very similar. The major difference in the training of these two models is that GPT-2 was trained on a much larger volume of data and the number of learnable parameters in GPT-2 is also substantially larger. GPT-1 was trained on a dataset containing about 3.3 billion tokens, whereas GPT 2 was trained on a much larger and diverse dataset containing text from over 45 million links and approximately 10 billion tokens after cleaning the text. GPT-2 contains a larger vocabulary, a larger context size so that it can process even more information, and even more layers. Work on GPT- 2 showed a lot of promise in zero shot learning, which is effectively applying the model on a task to which it wasn't trained on. The increase in parameter size showed that the model was good on several tasks on which it wasn't explicitly trained on, such as question answering. The massive increase in performance and state-of-the-art results on multiple tasks, as well as other external publications, led OpenAI researchers to continue building larger models with even more diverse datasets.   GPT-3 was truly the game changer for Large Language Models. It contains approximately 175 billion parameters and was trained on roughly 300 billion tokens. The GPT-3 and GPT-4 family of models are closed source models (now common practice by most of the major developers of LLMs) so most of the known information pertaining to these models comes directly from OpenAI but in some cases, some of the model information can be derived. It is estimated that GPT-4 contains a total of 1.7 trillion weights and, as of early 2024, it is currently unknown what the size of the training dataset was. The sheer size of these models give it extremely impressive performance even on tasks for which it was not trained on. These models with just clever prompting can outperform the best encoder-based models on tasks in which encoders are meant to shine due to the complexity and size of the language model.   Conclusion   The introduction of the original transformer in 2018 feels like ages ago given the rapid pace of progress in the field. Nowadays, there are new transformer-based models, applications, products, etc. being released every day. Although there have been many new techniques and ways to improve LLMs, understanding the original transformer and the concepts behind it is still valuable since it is still the backbone to modern LLMs. Hopefully this series as posts has broadened your knowledge on how transformer based models work, are trained, and how they’re developed.   References   Attention is All You Need Improving Language Understanding by Generative Pre-Training BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding RoBERTa: A Robustly Optimized BERT Pre-Training Approach  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Neural Machine Translation with a Transformer and Keras Introduction to Attention, Transformers, and Large Language Models: Part 1 Introduction to Attention, Transformers, and Large Language Models: Part 2     Find more articles from SAS Global Enablement and Learning here. ... View more

This post is a continuation of the first post in this series. In the previous post we introduced the attention mechanism and the transformer architecture. The focus of this second post will be to break down the different components that make up the transformer, which were introduced in the previous post. It should be noted that some topics will not be discussed in as much detail since they are common deep learning topics. These include skip layers, residual layers, feed forward layers, and linear layers. The ultimate objective of this series of posts is to provide enough detail and background to get machine learning/deep learning developers comfortable implementing and using LLMs.   Input and Output Embeddings   The process of turning text into embedded vectors is the first step in training transformer based models and using them to generate new predictions. For this reason, the first component of the transformer that we will discuss are the input embeddings in the encoder and the decoder. The Multi-Head Attention (MHA) layers, feed forward layers, linear layers, and even the embeddings themselves, have learnable parameters associated with them that are learned during training. Those learnable parameters are what fundamentally make transformers neural networks. The reason I highlight this is because, just like neural networks, transformers only work with numeric data. The process of turning text into embedded vectors is the first step in training transformer based models and using them during to generate new predictions.     Figure 1. Input and Output Embeddings with Positional Encondings.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Tokenization, Embedding, and Vectorization   Language models, transformers included, have a vocabulary size which generally represent the words that the language model knows. Each word in the vocabulary has a specific number, known as a token, that can be used to represent it. The vocabulary size is something that is determined early in the modeling process, and it is usually tied to the text that will be used to train and assess the model. There are also tokens for punctuation marks, special tokens such as the START and END tokens which indicate the beginning and end of a sentence, and tokens for unknown words that are not part of the known vocabulary. There are many techniques used in language models to tokenize text, however many of those techniques go beyond the scope of this post.     Figure 2. Text Tokenization.   When the transformer receives input text it is turned into tokens which are used to represent the words in the input. In figure 2, we can see the tokens (except for the [START] and [END] tokens) that represent the input sequence in a BERT model. These tokens can then be used to find the embeddings corresponding to each of those words.     Figure 3. Token Vectorization and Embedding Matrix (Right).   You can think of tokens as keys that can be used in a look up table to find the vector representation of the words in the input. The vectors that correspond to words are known as embeddings, and the matrix that houses all those vectors is known as the embedding matrix (shown in figure 3). The dimensionality of the embeddings will differ depending on the model being used. and it is usually specified by the modeler early in the modeling process. The original transformer embeddings output vectors with a dimensionality of 512.   What truly makes embeddings powerful is the fact that there are learnable parameters associated with the embedding matrix. These learnable parameters are adjusted during training such that they learn the relationships between words in a sequence. The original transformer used the same embedding matrix in the encoder and decoder.   Positional Encodings   After the words in an input have been vectorized, both the encoder and the decoder in the original transformer apply positional encoding to the embedded vectors. The positional encoding encodes the position of a word in a sequence.     Figure 4. Positional Encoding Layers.   In the sentence “The wolf ate the lamb”, 'The' is the first word, 'wolf' is the second word, etc. The order in which those words appear in the sentence matters. “The wolf ate the lamb” is not the same sentence as “The lamb ate the wolf”, a single positional swap can change the entire meaning of a sentence. For this reason, the positional encoding encodes the location of where words appear in a sequence. Without this, the order of the inputs is lost which would have very detrimental effects on the model.   In a transformer, positional encodings replace convolutions and recurrences. It is hypothesized that this function allows the model to easily learn to attend to tokens in a sequence via their relative positions. Positional encodings are designed such that they have the same dimensionality as the embeddings, in other words, they can simply be added to the embedded vectors. This makes this implementation of positional encodings very computationally efficient.     Figure 5. Sinusoidal Positional Encodings.   Sinusoids provide a way to accomplish positional encoding that is based on the Fourier transform. A sine and cosine function of different frequencies (illustrated in figure 5) are used to encode the positions of words in a sequence. The functions takes the position of the word, the element in the embedded vector to which information is being encoded to, and the model embedding dimensions as arguments. Stated more clearly, the encoding function output (given the previously stated input) is added to the corresponding element in the vector. The sine and cosine function are used for even and odd words respectively.   It should also be noted that the equations in figure 5 are not the only way to encode the position of text in a sequence. Another positional embedding alternative that was explored in the original transformer are learned embeddings. Learned embeddings yielded similar performance to sinusoidal positional encodings. The authors of the original transformer chose sinusoidal positional encoding because it does not require our inputs to have a fixed length. The sinusoidal embeddings allow the model to extrapolate to sequence lengths longer than the ones encountered during training, whereas learned embeddings require the inputs to have a fixed length. Learned embeddings generally require that additional information be added to the input tokens to indicate their position in a sequence. Other language models, such as GPT-1 and GPT-2, use learned positional embeddings.   Global Self-Attention Layer   As we stated in the previous post, each Multi-Head Attention (MHA) layer in the transformer is slightly different. As such it may not be uncommon to see these layers being referred to by a specific name. The first MHA layer in the encoder is sometimes called the global self-attention layer. The global self-attention layer allows information to flow in both directions because it performs self-attention without masking. Meaning that later words in a sequence can attend to words earlier in the sequence and vice versa. This ultimately allows encoders to truly capture the interrelatedness between all words in a sequence. Think about a scenario in which you are translating a sentence. Generally speaking, you listen to the entire sentence or a certain number of words before starting to translate. The order of the words in a sentence matters and they vary from language to language. If you start to translate everything word by word, your translation may be incorrect in the target language. Let’s say you’re trying to translate the Japanese phrase “Oishii desu!” which translates to “It is delicious!”. If you were to translate it word by word in the order in which the Japanese sentence appears, you’d produce the Yoda-like sentence “Delicious it is!” which is not the best way to translate that sentence.   Let's look at an example of what occurs within the global self-attention layer. As we saw earlier, the input to the encoder is a sequence of text. After that sequence of text gets tokenized, vectorized, and the positional encoding is applied, we end up with a matrix that represents the input sequence. One axis corresponds to the sequence of words and the other axis corresponds to the embedding dimension.                      Figure 6. Input Dimensions (Left) with Example (Right).   Let’s say that the input sequence is “this is an input!”. That would mean that the sequence length is a total of 5, and the embedding dimension is a hyperparameter determined by the modeler. In the original transformer the embedding dimension is 768, but for the sake of this example, let’s say it’s 3.     Figure 7. Self-Attention Matrices.   Now that we have our matrix of inputs, we can move forward and start to compute self-attention. In self-attention the query, key, and value, are all the same, which means this matrix is equal to Q, K, and V. Although technically we have everything that we need to compute self-attention, remember that within transformers, attention appears in the multi-head attention layers. Multi-head attention layers are important because they introduce learnable parameters (or weights) into the attention computation. Without these weights we would just be re-averaging vectors as this matrix moves through the transformer. The query, key, and value, have their own set of learnable parameters that will be used to modify them. The weight matrices have a dimension of embedding by embedding so that they can generate an output of the same dimension as their input.     Figure 8. Generating Weighted Embeddings.   It is important to note that the operation illustrated in figure 9 will be performed for Q, K, and V, and each of these will have its own separate set of weight matrices. It is important to note that After the query, key, and value are multiplied by their weight matrices, we can then compute self-attention. Remember that in the multi-head attention layer, attention is computed once for each attention head and each attention head will have its own set of weights for Q, K, and V. In the original transformer each MHA layer had a total of eight attention heads, however, for the sake of this example we are going to focus on a single attention head. Once we have multiplied query, key, and value by their weights we can then compute self-attention.     Figure 9. Visualization of the Self-Attention Computation.   The query matrix is multiplied by the transpose of the key matrix and this product is divided by the square root of the dimension of the keys, which is not displayed here. Once we have scaled the product of these two matrices, we can apply a softmax function to it. The resulting matrix is then multiplied by the value matrix. If you have kept track of the dimensions of all of the matrices, you will know that the matrix we output after we perform all these operations will have the same dimension as the original input matrix. All of the heads are then concatenated and passed to one final linear layer where it is multiplied by one final weight matrix. The weight matrix has a dimension of embedding by embedding so that we can output a matrix of the same dimension as the original input to the encoder.     Figure 10. Attention Head Output Visualization.   The operations that occur in the other MHA layers in the transformer are very similar to the steps that we have already outlined. When we discuss the other MHA layers, we will focus on highlighting the differences between those steps and these.   Causal Self-Attention Layer   Next let’s look at the first MHA layer in the decoder. This layer is sometimes called the causal self-attention layer.     Figure 11. Causal Self-Attention Layer.   As we have stated previously, transformers are autoregressive models which means that they generate one token at a time and feed that output back to the input. Unlike the global self-attention layer, every word in a sequence does not attend to every other word in the sequence. In the encoder block the model tries to learn the interrelatedness between all the words. The decoder requires a different approach, if every word could attend to every other word in the sequence the model would not learn to predict tokens in a way that makes syntactical sense, it would just output a grab bag of words. That means that we need to ensure that during training, we don’t look at future words in a sequence. Leftward information flow in the decoder needs to be prevented to preserve the decoders auto regressive property. This is achieved by applying a mask that effectively hides future words in a sequence during training.   Teacher Forcing and Masking   Now you might be wondering, if the decoder is autoregressive how is it that it is more parallelizable than RNNs? Part of why this is possible is a training technique called teacher forcing. Teacher forcing allows the true value of an input to be passed to the next time step regardless of the model's output at the current time step. Let's say that we are training a transformer to translate Spanish text to English text, and in this training iteration the sequence that we're trying to translate is “El lobo se comió el cordero” which translates to “The wolf ate the lamb”. In this scenario we have the input which is the Spanish sequence, and the target which is the English sequence. Let's say we're trying to predict the third token in the sequence, ate. Regardless of whether we correctly predicted the tokens for “The wolf” in the previous two iterations, when trying to predict the third token we pass the correct tokens (The wolf) as inputs to try to produce the third token (ate) in the sequence. Additionally, thanks to the way attention is computed, each word doesn't need to be processed sequentially. So if we are trying to predict the third token, we can process the first two tokens simultaneously. Thanks to teacher forcing we do not need to run the model sequentially, the outputs at different sequence locations can be computed in parallel.   Masking is another major component in training the decoder portion of the transformer. It is implemented in the causal self attention layer as part of the scaled dot product attention calculation.   Figure 12. Attention Computation with Masking Inside Causal Self Attention Layer.   The mask hides values that an input shouldn't see depending on which token in the sequence we're trying to predict. It sets all values in the softmax input that correspond to illegal connections  (think future words the model should not be currently seeing) equal to negative infinity. The ability to mask values in combination with teacher forcing allows us to train language models on much larger sequence than we were previously able to and as a result, generate language models larger than ever before.   Cross Attention Layer   The last MHA layer that we find in the transformer performs cross attention. This layer is important in the transformer architecture because it connects the encoder output to the decoder. Before discussing the cross attention layer in more detail, it is important to highlight another key difference between the encoder and the decoder. That difference is that the decoder receives the outputs shifted right, this can be seen at the bottom of figure 13.   Figure 13. Cross Attention Layer.   The first token that is generated in the decoder is always a special [START] token, so before we try to predict the first token in the sequence, we consult the encoder output. Using the same machine translation example we used earlier, "El lobo se comió el cordero" translating to "The wolf ate the lamb", prior to translating the first word in the sequence, we consult the encoder output first. During inference we always start by predicting the [START] token, consulting the encoder output, then feeding the output back to the decoder to predict the first token in the sequence. The attention that we compute here determines how relevant the source sequence is in the computation of the output. It is important to note that there are multiple layers in the decoder and every single cross attention layer that appears in the decoder will receive the encoder output. For example, in the original transformer there are 6 cross attention layers, all six of those layers will receive the encoder output.   In the cross-attention layer the encoder output is set equal to both the key and the value. The decoder predictions, whether it is only the [START] token or a sequence of tokens, will act as the query in the cross-attention computation. Padding may need to be applied so that the dimensionality matches between the encoder output and the decoder output.     Figure 14. Encoder Output Matrix.   The cross-attention computation is very similar to the ones that we saw in the causal self-attention layer. Remember that the query, key, and value matrices are multiplied by their weight matrices prior to getting to this point. The masking component is not applied in the cross-attention layer, because the output of the previous multi-head attention layer did include masking and that is the matrix that is used as the query here.   Residual, Linear, and SoftMax Layers   Throughout the transformer we have residual layers and layer normalization at the output of each multi-head attention layer and each feed forward layer. Residual connections and layer normalization are common techniques in deep learning so we will not do a deep exploration of these topics at this time. However, it is important to point out that these techniques help to prevent the loss of information and improve the overall generalizability of the transformer.     Figure 15. Residual Layers with Layer Normalization.   The feed forward blocks in the encoder and decoder contains two fully connected hidden layers, the first hidden layer in the block contains a user-defined number of hidden units while the second layer contains a number of hidden units that matches the dimensionality of the embeddings.     Figure 16. Feedforward Layers.   This allows the output of the feed-forward block to match the dimensionality of its input. It will output a matrix with dimension of sequence by hidden unit size. Since the hidden unit size is equal to the embedding dimensions, both the input matrix to the feed forward layer and its output matrix are of the same size. Just like with residual layers and layer normalization, fully connected layers are commonly used in deep learning so we will not be exploring this topic in much detail. The output of the decoder layers is passed through one final linear layer and a softmax activation to produce the output probabilities for a given token, in other words, a prediction for the next word in the sequence.     Figure 17. Decoder Output.   During inference, the output of the softmax would be fed as an input to the decoder to try to predict the next token in a sequence. This process continues recursively until the decoder produces the final token, the [END] token.   Conclusion   In this post we’ve elaborated upon some of the topics discussed in the previous post of this series. We’ve now explored the individual components of the transformer model in more depth and discussed how text moves through the transformer architecture. In the next post of this series we’ll discuss two early LLMs, GPT and BERT and how they’re different from the transformer model.   References   Attention is All You Need Serial order: a parallel distributed processing approach Long Short-Term Memory Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Neural Machine Translation with a Transformer and Keras Introduction to Attention, Transformers, and Large Language Models: Part 1     Find more articles from SAS Global Enablement and Learning here. ... View more

Transformer based models are the backbones of modern large language models. Unlike language models based off recurrent neural networks which operate via recursions, transformer-based models rely entirely on attention. The objective of this post is to introduce the attention mechanism and introduce readers to the transformer model. This lays the groundwork for future posts focused on large language models. The ultimate objective of this series of posts is to provide enough detail and background to get machine learning/deep learning developers comfortable implementing and using LLMs.   Introduction   Language models are used for a variety of tasks including speech recognition, machine translation, text generation, entity recognition, sentiment analysis, and many more applications. Before the first paper on the transformer model was published in 2018 [1], most language models were using recurrent neural networks [2](RNNs). It should be mentioned that earlier language models were used prior to RNNs, such as n-gram models, but those are beyond the scope of this post.   History of Recurrent Neural Networks (RNN)   Early works in language modeling can be traced all the way back to the early 1900s. Although this work was not useful in modern applications, it was influential in the development of these models. Substantial progress in language modeling came with the advent of the RNN in the 1980s and it's variations in the 1990s given the wide availability of computational resources that sprung up during these periods. A major step forward was taken in 1997 when the Long Short-Term Memory (LSTM) network paper was published [3]. At the time, this model set performance records across many natural language tasks.   For roughly two decades recurrent neural networks and their variations (such as LSTM and GRU [4]) were the basis of the most powerful language models. These models can be used to process sequential data such as time series data and text data. The order in which words appear is very important to the construction of language, therefore any model that is used for natural language tasks must take sequences into account. RNNs were, and in some cases still are, used for text classification, machine translation, and entity recognition.   Limitations of RNNs   In the years since there have been many developments that have improved the power of these models. Although these models achieved amazing results, the creation of these models preclude the wide availability of parallel computing and extensive computational resources. RNN's require computations using previous elements in a sequence for them to generate outputs later in a sequence. The inherent sequential nature of these models limits the degree to which they can be parallelized.   Relating signals from two arbitrary positions becomes prohibitively difficult. As sequence lengths increase, it becomes increasingly harder for a model to learn dependencies between distant positions. To make the problem worse, the computational complexity can in some cases grow linearly or logarithmically.   Lastly, these issues limit the length of sequences that RNNs can handle. Limiting the length of sequences that RNNs can process limits the kinds of data that we can use to train models, the applications for which you can use these models, and ultimately limits how deep of an understanding of language a model could have.   Attention is All You Need   In 2017, a paper was published by researchers at Google and the University of Toronto titled “Attention is All You Need”[1]. In this paper the authors introduced a model called the transformer. This model leverages a technique called attention and it was introduced in the model in the form of multi-head attention layers. Attention is inspired by the idea of how human brains deal with massive amounts of audio and visual input.   The transformer relies entirely on attention without the need for recurrences or convolutions like what are used in RNNs and CNNs, respectively. One of the key motivations for using attention without recurrences is that it can address the shortcomings that are found in sequential models, such as RNNs. Attention does not require text data to be processed sequentially. This means that these models are much more parallelizable. They achieve this while reducing the overall number of sequential operations required to generate an output and reduce computational complexity. One last key advantage of attention-based models is that they can learn dependencies between distant positions. This allows them to process and learn from longer sequences than RNNs are capable of.   Figure 1. Scaled Dot Product Attention.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In figure 1 we can see a visualization of how we can compute the type of attention that is implemented in Transformers, scaled dot product attention. Attention layers are fundamentally weighted mean reductions. They take three vectors as inputs. These are called the query, key, and value vectors. These vector names were inspired by data retrieval systems. One way that we can think about attention is as mapping a query and a key-value pair to an output.   The process begins by taking a query and a key, generally in matrix form, and then performing matrix multiplication between the query and the transposed key matrix. We then scale these products and optionally apply a mask to this output. The mask is useful because it can hide words in a sequence. Masking can be used to prevent language models from looking at future words in a sequence during training. The resulting matrix is then passed to a softmax function. The softmax output is then multiplied with the value matrix. Equation 1. Scaled Dot Product Attention.   We can see the formula for scaled dot product attention above. We divide the product of the query and key matrices by the square root of the dimension of the keys. This is used as a scaling factor because for keys with a high dimensionality, the dot products grow large in magnitude. This pushes the softmax into regions where it has very small gradients.   Mathematically looking at attention, it may be difficult to get an intuitive grasp with regards to what attention is computing. An example that can be used to grain some intuition about attention, is by thinking about a sentence like, "The quick brown fox jumps over the lazy dog." In a given sentence we often place our attention on a specific word (or words) while the rest of the sentence generally adds structure and context to what you’re paying attention to.     Figure 2. Attention Sentence Example.   What’s being paid attention to in the sentence is the query, in this case we'll choose "the lazy dog". This is what we'll calculate attention for. The key and value are the words we evaluating in relation to the query. In other words, how relevant are those words to the query. In a conversation there is always a subject or topic of interest, which is what we focus our attention on. In this example you think of the topic of interest as the lazy dog. Maybe we're having a conversation about how lazy our friend's dog is. The next few sentences or paragraphs in the conversation may revolve around this lazy dog. And to reinforce the idea that the dog is lazy, we give an example of something that happened. A quick fox jumped over this dog and , presumably, this dog had no reaction. In this case, quick brown fox and jumps over are the key and value. They're the words that help to communicate this idea that the dog is lazy.   Researchers often have an idea on how to improve language models based off of real world observations, in this case it was this idea of attention. It is important to remember however that many of these ideas may not be how things work in the brain. These are just ideas that researchers try to model mathematically to see if they yield better results in the tasks at hand, in this case, language modeling.   Self-Attention   Although we can calculate attention using different queries, keys, and values, another variation of attention which we find in transformer models is called self-attention. Self-attention is a special case of attention in which the query, key and value are all the same. This means that everything in a sequence will ultimately attend to itself. This technique could be useful in a situation where you're trying to understand the meaning of a given input sequence and how each of the values in that sequence are related to one another.   As an example, think of a translator translating a sentence from one language to another. They generally read or listen to the entire sentence that needs to be translated, process all the words to determine their meanings, and only then do they start to produce a grammatically correct translation that gets the actual meaning across.  Take the Japanese sentence "oiishi desu", which translates to "it is tasty". "Oiishi" means "tasty" and "desu" means  "it is". If the translator were to translate each word as they hear them, the sentence would be translated to "Tasty it is". This may get the meaning across, but it is grammatically incorrect and may be confusing. In some cases, this approach of translating each word one at a time may yield something with an entirely different meaning. An example of this is the sentence "The cat sat on the mat". By just swapping two words you can form the sentence "The mat sat on the cat". This has an entirely different meaning. This is why processing all words in a sentence allows a translator to understand the true meaning of what's being said and also allows them to produce the grammatically correct sentence. Everything prior to producing the translation is effectively what self-attention helps transformer based models do. It can be useful in situations where we need to find how every word in a sequence relates to others.   Multi-Head Attention   Attention isn't only calculated once, in fact, it is calculated many times within a transformer. In transformers, attention is used in what are called multi-head attention (MHA) blocks. Within these blocks attention is computed multiple times in what are called heads; more specifically, attention is computed once per head. It’s important to note that we don’t just compute attention between Q, K, and V, multiple times. That would not accomplish anything since we would just be repeatedly re-averaging vectors. MHA blocks pass Q, K, and V through their own set of linear layers, which is where we start to introduce learnable parameters and effectively turn transformers into neural networks.     Figure 3. Multi-Head Attention.   Each attention head (denoted as h in figure 3) contains a different set of learnable parameters for all three of Q, K and, V. This makes it so that each attention head effectively computes attention using different values for all three of these vectors. The output of each attention head is then concatenated and passed through one final linear layer to generate the MHA layer output. Since each attention head has different weights for all three of our vectors, you can think of multi-head attention as a method that allows the model to jointly attend to information from different representation subspaces at different positions. That means that as we train our model each attention head will capture different relationships between these three inputs.     Equation 2. Multi-Head Attention.   The equation for multi-head attention is straightforward given that we know how to calculate attention. All we need to do is concatenate the outputs of each attention head into a single matrix and then multiply that by a matrix of learnable parameters which represents the final linear layer in the MHA block. The way we compute an individual attention head is by computing attention on the products of weighted query, key, and value matrices. The number of attention heads is a hyperparameter that can be chosen by the modeler which represents how many times Q, K, and V are projected.   Ultimately, you can think of multi-head attention as a method that allows the model to jointly attend to information from different representation subspaces at different positions. That means that as we train our model using some data set, each attention head will capture and focus on different relationships between these three inputs. ​   Transformer   Now that we understand what attention and multi-head attention is, we can start to discuss the overall architecture of the transformer. The transformer consists of two sections, an encoder and a decoder.   Figure 3. Transformer Architecture.   The encoder takes an input sequence, learns the interrelatedness of all the words in the sequence, and outputs a representation of that. The decoder consults the encoder output to generate its own output, auto-regressively, one token at a time. The decoder output is the transformers final output. We can see that within the transformer the multi-head attention layers appears in both the encoder and the decoder. In very simple terms, the learnable parameters along with how queries, keys, and values are combined within the transformer is how these models fundamentally learn the meaning and interrelatedness between words. There’s a handful of other things that make these models work, those will be discussed in parts 2 and 3 of this series of blogs.   Let's look at the encoder, the decoder, and all the other components that make transformers work in more detail.   Transformer Encoder Figure 4. Transformer Encoder.   The encoder shown in Figure 4 receives the user input, passes it through an input embedding, applies positional encoding and then passes the input through a stack of N identical layers. Each layer has two sub layers: a multi-head attention layer and a feed forward layer. You can think of the feed forward layer as what you would find in a classic neural network. Both the multi-head attention layer and the feed forward layer are followed by a residual layer with layer normalization. In the "Attention is All You Need" paper [1], this stack is repeated a total of 6 times.   The objective of the encoder is to learn a representation and the interrelatedness between all the words in the sequence. Every word in the sequence pays attention to every other word in the sequence. If this sounds familiar it's because the MHA layer in the encoder performs self-attention. We're going to cover the details of what occurs in each of those layers in the next post.   Transformer Decoder   Figure 5. Transformer Decoder.   The architecture for the decoder does not look much different than the encoder architecture. The key difference is that the decoder applies masking, it contains a second multi-head attention layer, and the user inputs are not passed into the decoder. The decoder actually consults the encoder output in order to generate its own output. Just like in the encoder, the stack is repeated a total of 6 times in the "Attention is All You Need" paper [1].   The objective of the decoder is to predict words one at a time. Unlike the encoder where the model tries to learn the interrelatedness between all words in a sequence, the decoder must only consult outputs for previous time steps (i.e. previous words) when computing a future output. For this reason, the decoder performs masked multi-head attention in the first MHA layer. The masking in this first MHA layer is used to prevent future words in a sequence from being used in the attention computation during training. When the decoder is being used to generate predictions, it does not know what the future word that it needs to predict is. A decoder trained on data where it is allowed to see the future would not perform well on new data where this information is not available to it. ​   Conclusion   In this post we’ve introduced the attention mechanism, self-attention, multi-head attention and done a quick overview of the transformer architecture. Multi-head attention layers are one of the main mechanisms through which transformer models learn the meaning and interrelatedness of words. You may be struggling to wrap your head around some of these concepts and how they play a part in the transformer model, and more broadly, LLMs. In the next post we will spend more time diving a bit deeper into the different components of the transformer and breaking down how information flows through it.   References   Attention is All You Need Serial order: a parallel distributed processing approach Long Short-Term Memory Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Neural Machine Translation with a Transformer and Keras     Find more articles from SAS Global Enablement and Learning here. ... View more

Semantic segmentation is a technique in the realm of computer vision that allows us to predict the contents of an image at a pixel level. These algorithms play a crucial role in a wide range of applications, including autonomous vehicles, object detection, medical imaging, and more. The goal of this article is to provide an introduction to segmentation for those unfamiliar with the topic but more importantly, to highlight and show users how to use a new CAS action set for deep learning recently introduced by R&D, the Model Zoo action set.   Introduction to Semantic Segmentation   Before we talk about the Model Zoo action set and show an example into how we can use it to build segmentation models let’s dive deeper into some of the details of segmentation models.   To build a deep learning algorithm or more broadly, machine learning algorithm, we need a training dataset. For segmentation models the training dataset consists of images, which act as the inputs, and their ground truth annotations (often referred to as masks), which act as the target.    Figure 1. Raw Image (Left) and Corresponding Mask (Right).   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The masks needed to train a segmentation model can be generated in different ways, including using tools such as CVAT (Computer Vision Annotation Tool) where users can manually annotate a set of images and assign classes to the segmented regions of the image. The annotated images can be exported in different formats, but the format used in this blog is one in which an image has each pixel value correspond to a class. This exported image assigns a unique color to each class which means that each class has a unique set of pixel values.   Figure 2. Annotating Images using CVAT.   Ultimately trained segmentation models are used to generate segmentation masks. The output masks will assign each pixel to the class the model found had the highest predicted probability. As stated before, these masks can be used in a wide array of applications and in a variety of ways. This data comes from a project I worked on with Data4Good for the UNC Center for Galapagos Studies where due to the size, quality, and distribution of the data, I built a segmentation model that can identify turtles in an image and segment them out. Doing this will remove the amount of noise in an image since turtles can appear in all kinds of different background (e.g. boats, sand, underwater, in front of someone, etc.) which can play a negative role when matching turtles to one another. The predicted masks were used to alter the original image such that only the turtle remains (background removed). The new altered images were then used to identify individual turtles from a list of ~600 different turtles.   Now that we’ve covered segmentation more broadly, in the next section we will choose a model architecture to perform segmentation and dive a little deeper into those details.   U-Net Model Architecture   In order to build a segmentation model, a suitable convolutional neural network (CNN) based architecture is chosen or designed to capture relevant features and learn the intricate relationships between pixels. Among various architectures used for semantic segmentation, the U-Net architecture has been one of the most popular options for years due to its performance and intuitive architecture. The U-Net model contains a U-shape architecture (hence the name!) with skip layer connections that combines both high resolution and low-resolution feature maps.     Figure 3. U-Net Model Architecture.   The first half of the U-Net model, or down sampling half, does the following:   Passes an input image through two, back-to-back 3x3 convolutional layers with ReLU activation. Feature maps generated from the second convolutional layer output is passed through a single 2x2 max pooling layer. Steps 1 and 2 are repeated three more times for a total of four down sampling steps. The feature map generated by the last max pooling layer is then passed through a final set of two, back-to-back 3x3 convolutional layers with ReLU activation.   The second half of the U-Net, or the up-sampling half, starts at the bottom of the U-Net after we perform our fifth set of 3x3 convolutions. The process looks like down sampling except we’re going in the opposite direction:   Start the up-sampling portion by performing a 2x2 up-convolution, which is also known as transposed convolution, on the feature map generated after the last convolutional layer. Take the feature map output from the convolutional layers from the other side of the U-Net (since the U is symmetrical) then copy and crop the feature map to match the dimensions of our up-sampled feature map. Concatenate our previously down-sampled feature map with our new up-sampled feature map. Pass the concatenated feature maps through a set of two back-to-back 3x3 convolutional layers with ReLU activation. Repeat steps 1 through 4 three more times for a total of four up sampling steps. Pass the final up-sampled feature map through a final set of two back-to-back three by three convolutional layers with ReLU activation. pass the last feature map through a 1x1 convolution layer to make the final feature map.   This process allows us to combine both high dimensional features and low dimensional features, which appears to be a key contributor in the performance of these models in segmentation tasks. The network can also be modified such that the output matches our input image dimensions. This way the masks can be overlaid over the input image if necessary. One final key distinction from this model architecture and other common CNNs is that this architecture does not have fully connected layers at the end. Instead, the output feature map of a U-Net model is directly passed to a pixel wise softmax link function, which generates the pixel wise class probabilities for each pixel in the resulting image.   Introduction to Model Zoo   Now that we have a short background in semantic segmentation and the U-Net architecture we can now start to discuss model zoo. Model Zoo is a CAS action set for deep learning based on the PyTorch programming framework. The action set has a backend that calls the PyTorch C++ API and a front end that is responsible for initiating actions, terminating actions, reading CAS tables, writing CAS tables, and communicating with the backend portion. Aside from the architectural differences, Model Zoo takes a different approach to deep learning than the traditional deepLearn CAS action set.   The process to build a neural network that works well on specific types of data can be a very time-consuming process that does not always yield the best results. For this reason, many users choose to use neural network architectures that have been published by reputable research groups that perform very well on specific tasks. These groundbreaking neural network architectures are generally published alongside an open-source implementation that oftentimes make their way to some of the most popular open-source deep learning libraries such as TensorFlow and PyTorch. For those reasons, Model Zoo currently puts a focus on using pre-defined model architectures, which is how many users use deep learning in practice. All that being said, Model Zoo also allows users to define custom architectures using PyTorch, a machine learning framework for deep learning built for Python and C++. There is a lot that can be said about defining custom models using PyTorch for Model Zoo, but that is beyond the scope of this blog. Readers of this blog should note that Model Zoo was released in the 2022.09 LTS release of Viya, so there is a still a lot of development being done on Model Zoo and readers can expect a lot more functionality to be added to the action set over the next few years.   Now that we’ve given some background about Model Zoo, let’s discuss how we can use Model Zoo to train a U-Net segmentation model.   Model Zoo U-Net Segmentation Example   This demonstration will be performed using Python; however, users should be aware that they can also use Model Zoo in CASL. The first thing we need to do, like with most Python program, is import the necessary packages:   import os import yaml import swat import dlpy import numpy as np import pandas as pd from glob import glob   This may end up looking like a typical list of packages when using Model Zoo. Below is a table briefly describing the use of each package in this demonstration:   Package Use os Required to save files to disk yaml Used to ensure that the YAML files used by Model Zoo are accurate and complete. swat Used to call the necessary CAS action sets, including the connection object and the model zoo actions. dlpy Used to perform deep learning tasks using a keras like syntax. This can also be used to train model zoo models using the MZModel class, however, to make this more adaptable to CASL users we will not use this approach. glob Used to create a list of files stored in a directory. numpy Used for array manipulation, including images. pandas Used to create dataframes/tables that can be saved in different formats. Table 1. Python Package Description.   Now that we have imported the package, we can establish a connection to CAS using the SWAT’s CAS class and load the necessary action sets. In this demonstration I will load the dlModelzoo action set, the image action set to be able to explore images saved in CAS, and the sampling action set to partition tables.   conn = swat.CAS("connection information") conn.loadactionset("dlModelzoo") conn.loadactionset("image") conn.loadactionset("sampling")   Now that we have gone through some of the basic bookkeeping, we can start getting more into the details of the requirements to build segmentation models. We require a table that contains two columns, the location of the image files complete with the extension, and the mask files complete with the extensions as well. Note that these paths should be relative to the CAS library that you’re planning on using. There are a variety of ways for a user to create this table such as creating a pandas data frame then saving as a CSV, using SAS programming then exporting as a CSV, creating it in excel, creating it using the DLPy package, etc. Once you have the CSV you can then load it into memory using the load table action from the table action set. Below is a sample of what your csv can look like:   Figure 4. Sample of CSV Containing Image Paths and Mask Paths.   Something different between model zoo and the deepLearn CAS action set is that we need to create a YAML file that is used by the actions to specify the model and/or data set specific parameters. Although this YAML file may seem a little bit strange, it gives users the flexibility to have a fixed location where they can quickly make changes to the model such as image transformations, input size changes, number of target classes in the chosen algorithm, etc. It is possible to have all the information within a single YAML file, however for the sake of readability I generally prefer to split it up into two YAML files; one for training and one for scoring. One suggestion that I would like to give users when creating the YAML file, is to use a text editor such as notepad++. The main reason for this is that the YAML file is sensitive to spacing, and the sub sections are tab delimited. Below is the YAML file that was used to train the model as well as a table explaining the options used in the YAML file:   documents = """ sas: dlx: train: label: "galapagos_unet" dataset: type: "Segmentation" organization: "CoCo" # enum[IMAGENET, OPENIMAGE, COCO] preProcessing: - modelInput: label: input_tensor1 imageTransformation: resize: type: TO_FIX_DIM size: 256 256 target_size: 256 256 imgStdType: STD model: type: "TORCHNATIVE" # enum [TORCHSCRIPT, TORCHNATIVE] name: "SAS_TORCH_UNET" # ignored when type is "TORCHSCRIPT" caslib: "mycl" classNumber: 2 inputs: - label: input_tensor1 size: - 0 outputs: - label: output_tensor1 size: - 0 """   Train and Score Description label Used by the train and scoring actions to call a specific section of the YAML file dataset Specifies the dataset type the action receives and the type determines how the data is read in and processed. dataset.type Supported dataset types are Univariate, ObjDetect, Segmentation, and Autoencoder. preProcessing Specifies the pre-processing (augmentation) of the input data as well as the target data. preProcessing.modelInput The action looks for the matching label in the model.inputs section to find the stream of input to apply the pre-processing to. modelInput.label The label gives the inputs a name and indicates the inputs (data) that the pre-processing will be applied to. Used by the actions to find the stream of inputs to apply the pre-processing to. modelInput.imageTransformation Allows users to apply transformations to the image, such as resizing, color transformations, random transformations, etc. imageTransformation.resize Used to resize the inputs to a specific height and width imageTransformation.imgStdType Normalizes the image pixel values such that each value falls in a range of [0, 1] Table 2. Train and Score YAML Options.   There are options such as dataset.organization that are just for the user to have a little bit more information that are not used by the Model Zoo CAS actions. Your YAML files should start off with SAS, DLX, and then the subsection after DLX can either be train or score depending on whether you will use this YAML file to train the algorithm or score using the algorithm. Most of the options specified under the training subsection are explained in the table above, so I will not be explaining those options. In this demonstration we’re using the segmentation data set type and two options to preprocess the input dataset. The first preprocessing technique used was to resize the images to fixed dimensions of 256 by 256 pixels for both the input images and the target images. The other transformation that was applied to the inputs was a standardization of the image pixels. Image pixels range from values between 0 and 255, so to ensure the weight updates are not too drastic in scale and to have information flow better through the network, regularization techniques such as this one are common. Note that there are many preprocessing techniques available in model zoo, however, in this example I chose not to apply any outside of these two. The second important subsection that we need to specify is the model subsection and here's where we can give more details about what kind of model we ultimately want to build. The table below explains some of the options that were used:   Model Options Descriptions type Currently supports two types of models, TorchScript and TorchNative. TorchNative are models that are pre-written in C++ and built into the action library. TorchNative is used for pre-built models such as YOLO and UNet. TorchScript is used to import custom models written in PyTorch. name Used only for TorchNative models to specify what type of model the user wants to use. caslib CasLib where model weights can be loaded and saved to. classNumber Number of classes in our targets. inputs Can be used to apply changes and transformations to the input and target tensors. inputs.label & outputs.label Specifies the name of the tensor to be modified. inputs.size & outputs.size Can be used to reshape the specified tensor to a given size. A value of 0 means no reshaping, if reshaping, then values need to be specified as: - Channel - Height - Width              Table 3. Model YAML Options.   Underneath the model subsection the first thing that we need to specify is what type of model we want to build. TorchNative corresponds to the pre-built models, while TorchScript correspond to user defined models using PyTorch. Afterwards you need to specify the name of the model that you want to build, which in this case SAS_TORCH_UNET. The ModelZoo documentation shows which models are currently available to use within the action set, with presumably more on the way in the future. Another necessary option when building a segmentation model is the number of pixel classes. The objective of the model in this case is to distinguish between the background and the turtles, so in this case my class number is going to be two, one class corresponding to background and one class corresponding responding to a turtle. We do not need to resize the input or output tensors, so we leave the size option as 0. That completes the YAML file, so next we can check to see whether it is syntactically accurate by using the YAML package along with the following line of code:   for data in yaml.load_all(documents): print(data)   If the YAML file is syntactically correct the output will be a JSON string that displays all of the user specified options. Keep in mind that this doesn't ensure that you won't have any errors whenever the YAML file is used to train or score the model, this only ensures that the YAML file is syntactically correct. In other words that every single option is delimited and spaced correctly.   Although users can build their own YAML file from scratch, the file can also be generated automatically via the DLPy package and modified as needed. The code below will create an object called unet that has the yaml file saved as part of an attribute that can accessed with the documents_train attribute.   from dlpy import mzmodel # Defines the UNet Model unet = mzmodel.MZModel(conn, model_type = "TorchNative", model_name = "unet", num_classes = 2, dataset_type = "segmentation") # Defines transformations for the UNet Model unet.add_image_transformation(image_resize_type = "RETAIN_ASPECTRATIO", image_size = 256, target_size = 256) # Displays model information unet.documents_train   Now that the YAML file is complete and syntactically correct, we can move forward with training the segmentation model. The way that we're going to do that is by using the dlmztrain action from the Model Zoo action set. Below is the dlmztrain action as well as a table with a description of the options being used in the action:   train_action = conn.dlmztrain(loglevel = "DEBUG", table = dict(name = "part_metadata", where = "_PartInd_ = 1"), inputs = "image_path", targets = "label_path", ngpus = 1, validationTable = dict(name = "part_metadata", where = "_PartInd_ = 0"), modelOut = dict(name = "galapagos_unet", replace = True), checkpointBest = True, outputIndexMap = dict(name = "galapagos_outputindex", replace = True), optimizer=dict(loss='cross_entropy', mode=dict(type='synchronous', syncFreq=1,), algorithm=dict( learningRate=dict(value = 0.0002, scalefactor=1e-3), method='ADAM', weight_decay=0.0005 ), batchSize = 10, seed=12345, maxEpochs=50), learningRateScheduler = dict(policy = "STEP", stepsize = 10, gamma = 0.5), extraoptions=dict(yaml = documents, label='galapagos_unet'))   dlmzTrain Action Options Description loglevel Reporting level for progress messages sent to the client. DEBUG allows users to see more information related to the training process. table CAS Table that is used to store the input data for the training step in training a deep learning model. inputs The input variables for the training task. Currently, we support image data as input, the input column can either be a string of an image path or a binary of the image data. targets The target variables for the training task. ngpus Used together with HyperParameter Tuning, specifies the number of GPU's to use across the entire grid. The GPU's with the lowest amount of memory currently allocated will be chosen. This option is mutually exclusive with the gpu option. validationTable The CAS table that contains the validation dataset. Used to assess model weights after each epoch. modelOut The CAS table used to store the trained model and model weights. checkPointBest Specifies whether to save the model weights that performed best on validation or the model produced in the final epoch. outputIndexMap The output CAS table containing a mapping from nominal class type to numeric values. The Index table of nominal class type to numeric values built in the training process is written to the CAS table at the end of the action. optimizer Key component of training any type of neural network. Currently supports a variety of optimization algorithms including SGD, Adagrad, ADAM, and ADAMW. tuner Specifies settings for hyper parameter tuning learningRateScheduler Used to specify a learning rate policy, such as a fixed learning rate or one that’s modified after n number of steps. extraOptions Used to specify the YAML file that should be read when training or scoring model as well as which section to read by using the label option.                          Table 4. dlmzTrain Action Options.   The first option I would like to highlight is the log level option. There are a variety of values that it could take. Personally, I like to set the log level equal to debug so that I get more messages about the training process. In the table option we specify the name of the table that we imported from a CSV that contains the names of the images and their corresponding masks. We use the inputs option to specify the name of the column that contains the names of the images, and the targets option to specify the name of the column that contains the images of the masks. If we are using a machine with GPU capabilities, we can use the GPU equals option and specify the number corresponding to the GPU that we want to use.   Our model weights are going to be stored in a CAS table called galapagos_unet and the weights that will be stored are the weights associated with the model that had the best performance thanks to the checkPointBest option. The optimizer parameter is very important because this is the process that we take to optimize the model weights. The loss function is used to specify how the loss will be calculated. Loss functions are very closely related to the distribution of the target and it's number of classes , as well as what type of model you may be trying to build. The algorithm option allows us to specify the details regarding the algorithm we want to use such as the learning rate, momentum, optimization technique, weight decay and more. Other options that I have specified in the optimizer option include the batch size, which is the number of images that I want to use for every iteration of the optimization algorithm, the seed to make the optimization reproducible, and the max epochs which specifies how many iterations of the optimization I want to go through. The learning rate scheduler can be used to specify whether we want a fixed learning rate for the optimization, or whether we want the learning rate to be changed throughout the optimization. In this demo, the learning rate is adjusted after every 10 epochs. Lastly, we use extraoptions to specify the name of the YAML file, and which section of the YAML file should be read to train the model by using the label option.   Figure 5. Sample of dlmzTrain Action Output.   Keep in mind that the output will differ depending on the log level, a level of debug gives a lot of information regarding the training process. I can also see information about the model such as the name of the model, the kind and number of layers in the model, how the loss function progresses, as well as the misclassification error and the mean intersection over union. At the conclusion of the training process, we get a reason as to why the optimization stopped and a message stating that the action completed successfully.   Now that the model is trained, we can move to start scoring with the model. To score using our model, we need instructions in a YAML file once again. The good thing is that the instructions for this YAML file look very similar to the ones in the training YAML file.   score_doc = """ sas: dlx: score: label: "galapagos_unet_score" dataset: type: "Segmentation" organization: "CoCo" # enum[IMAGENET, OPENIMAGE, COCO] preProcessing: - modelInput: label: input_tensor1 imageTransformation: resize: type: TO_FIX_DIM size: 256 256 target_size: 256 256 imgStdType: STD model: type: "TORCHNATIVE" # enum [TORCHSCRIPT, TORCHNATIVE] name: "SAS_TORCH_UNET" # ignored when type is "TORCHSCRIPT" caslib: "mycl" classNumber: 2 inputs: - label: input_tensor1 size: - 0 outputs: - label: output_tensor1 size: - 0 """   The main differences between this YAML file and the one that we use for training, is that the third sub option is score instead of train, and the label is now “galapagos_unet_score” so that this YAML file has a different name. Outside of this change everything remains the same, and as a matter of fact, in most cases the training YAML file can be the same as the scoring YAML file. When using DLPy to build your UNet models, DLPy makes no distinction between these two files. With a complete YAML file we can now use the dlmzscore action. Below we can see the code as well as a table that provides a brief description for all the options that were used within the action:   score_action = conn.dlmzscore(modelTable = "galapagos_unet", table = dict(name = "part_metadata", where = "_PartInd_ = 0"), inputs = "image_path", targets = "label_path", batchsize = 10, gpu = {0}, loglevel = "DEBUG", tableout = dict(name = "unet_output", replace = True), extraoptions = dict(yaml = score_doc, label = "galapagos_unet_score"))   dlmzScore Action Options Description loglevel Reporting level for progress messages sent to the client. DEBUG allows users to see more information related to the training process. modelTable The CAS table containing the model and model weights. This parameter is optional. When it is specified, the table stores a binary blob containing the model and weights in Pytorch format; when it is not specified, you can specify the model table file path relative to a CAS table path in the YAML file in the extraOptions parameter. table The input CAS table that is used to store the input data for scoring a Deep Learning model. inputs The input variables for the scoring task. Currently, we support image data as input, the input column can either be a string of an image path or a binary of the image data. targets The target variables for the scoring task. batchsize Number of images to score each epoch. GPU Specifies the GPU to use when scoring. tableOut Output table used to store the output generated from scoring. extraOptions Used to specify the YAML file that should be read when training or scoring model as well as which section to read by using the label option.                  Table 5. dlmzScore Action Options.   Similar to the dlmztrain action, the amount of information that we get as part of the output, is going to depend on the log level value. Since the log level is set to debug once again, we get as much information as possible regarding the scoring process. Once again we see information regarding our model such as the number of layers, the kinds of layers, the number of parameters, and how the loss, MCE, and mean IoU, changed as we scored our batch of validation images. The tableout option dictates what the name of the output table that contains the predicted images is going to be. The table generated by the score action is the unet_output.       Figure 6. Sample of dlmzScore Action Output.   If we want to save these images we can use the image action set, and from the image action set use the saveImages action. Simply exploring these images is going to show that the output appears like just a plain black image, but the reason for this is because these images are outputting values of zeros and one, a pixel value of 0 corresponds to background, while a pixel value of 1 corresponds to a turtle. In order to fully visualize the mask, a little bit of additional pre-processing was performed to visualize the output.   Figure 7. Image Input (Left), Raw Mask Prediction (Middle), and Processed Mask (Right).   Visualizing these images we can see that this model performed pretty well, which is also corroborated by the mean IoU on the validation dataset. It should be noted that the purpose of the blog is to act as an introduction into segmentation models, Model Zoo and it's syntax, and as such, not much time was spent fine tuning this model. I hope that this blog has allowed you to have a better understanding of segmentation models and hopefully provided you with enough context to get started with the action set!   Additional Resources   ModelZoo Action Set ModelZoo Model Reference Using YAML Specified Options Image Action Set Table Action Set     Find more articles from SAS Global Enablement and Learning here. ... View more

Have you ever been in a situation where you need a dataset to try or showcase a new feature, present information externally or to other internal divisions, protect personally identifiable information before using data elsewhere, etc. I recently found myself in this situation, and anyone that's ever been in this situation knows that finding a good data set (especially if it needs to be legally approved) is a lot harder than it sounds. Given SAS's push towards generative A.I., I decided to use use a model that falls under the "generative A.I" umbrella, generative adversarial networks (also known as GANs) to tackle this task!  GANs can learn the joint probability distribution of a data set with different variable types to generate synthetic data that contains a distribution similar to the original dataset. In this post we’ll dive a deeper into the science behind GANs and will conclude the post with a demonstration on how to use the tabularGanTrain action to build GANs that allow you to generate synthetic data.   Introduction to Generative Adversarial Networks   Generative Adversarial Networks (GANs) were first introduced by Ian Goodfellow and his colleagues in their paper "Generative Adversarial Nets" in 2014. The idea they introduced in this paper is what can we accomplish if we build two models that actively compete against one another? One of these models would be the generator, which captures the data distribution. The second model would be the discriminator (also known as critic) that estimates the probability of whether a sample came from the training data or the generator. The analogy the authors use to explain the model in layman's terms is that the generative model is analogous to a team of counterfeiters trying to produce fake currency and use it without detection. The discriminative model is analogous to the police trying to detect the counterfeit money.   As one can probably imagine, the objectives of these models are counter to one another. As a result, the training procedure for the generator is to maximize the probability of a discriminator making a mistake. On the other hand, the training procedure for the discriminator is to minimize the mistakes when comparing samples produced by the generator and the real distribution. Goodfellow et al proposed the use of fully connected networks for both the generator and the discriminator since this would allow the entire system to be parallelized and the networks can be trained with back propagation. Newer advancements in GANs modify the architecture of the system so that they use different types of networks and techniques. Two such advancements are pertinent to this post, the Conditional Tabular GAN (CTGAN) and the Correlation-Capturing GAN (CorGAN).   CTGAN   GANs can be used to generate a wide variety of data, such as image data, video data, audio data, and tabular data. Each type of data comes with its unique set of challenges, tabular data being no exception. To address the unique challenges posed by generating synthetic tabular data, the Conditional Tabular GAN (CTGAN) and the Correlation-Capturing GAN (CorGAN) were published in 2019 and 2020 and respectively. SAS combines concepts from both models to create the Correlation-Preserving Conditional Tabular GAN (CPCTGAN) model. This model is used in the tabularGanTrain action which I will showcase later in the post.   Some of the key challenges postured by Xu et al (CTGAN authors) in generating synthetic tabular data include the need to simultaneously model discrete and continuous columns, multi-model non-Gaussian values within continuous columns, learning from sparse one hot encoded vectors and imbalances in categorical columns. The CTGAN model introduces several novel techniques to address those issues, including augmenting the training produce with mode-specific normalization, architectural changes, a conditional generator to address data imbalances and training by sampling. In this post we will touch on these topics, but we will not address them exhaustively. For more information, please refer to the paper and GitHub [2][3].   Mode Specific Normalization   Discrete values can naturally be represented as one hot encoded vectors, but what about numeric columns with arbitrary distributions? Mode specific normalization helps CTGAN deal with columns that have complicated distribution. This method allows CTGAN to process the columns in the training dataset independently. The method can best be explained in three steps:   For each continuous column, a variational gaussian mixture model is used to estimate the number of modes and fit a gaussian mixture. For each value in the continuous columns, compute the probability of each value coming from each node. One mode is sampled from the given probability density and the sample mode is then used to normalize the value.     Figure 1. Mode Specific Normalization [1].   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The representation of a row then becomes the concatenation of continuous and discrete columns.   Conditional Generator and Training by Sampling   In a traditional GAN, vectors sampled from a standard multivariate normal distribution are fed into the model, however, this does not account for imbalances in the levels of categorical columns. Additionally, if the training data is randomly sampled, rows belonging to minor categories will be underrepresented. The goal is to resample in a way that all categories from discrete attributes are sampled evenly (but not necessarily uniform) during the training process and to recover the real data distribution during testing (not resampled). The solution consists of three key elements:   Conditional Vector: The conditional vector is introduced to indicate the condition that a discrete column takes on a specific value. The conditional vector is generated by concatenating various masks that represent one hot encoded vectors in discrete columns. e.g. We have two discrete columns, D1 = {1, 2, 3} and D2 = {1,2}. To represent the condition D2 = 1, we produce two mask vectors. These vectors are m1 = [0, 0, 0], which represents D1 in this case (no values from D1 selected), and m2 = [1, 0], which represents the fact that we want the first value in D2. Concatenating these two gives us the conditional vector [0, 0, 0, 1, 0].   Generator Loss: During training the conditional generator is free to produce any set of one-hot discrete vectors. The mechanism proposed to enforce the conditional generator to produce synthetic copies of the discrete columns is to add a penalty to the cross entropy loss function such that as the training advances, the generator learns to make copies of the one hot discrete vectors that match the training sets distributions.   Training by Sampling: The output produced by the conditional generator must be assessed by the discriminator. What the discriminator will do is that it is going to estimate the distance between the learned conditional distribution and the real data conditional distribution.     Figure 2. CTGAN model/training by sampling method [1].     CTGAN Network Structure   The last major changes that allow the CTGAN models to work are the network structures of both the generator and the discriminator. Since columns in a row do not have any local structure, fully connected networks are used in both the generator and discriminator to attempt to capture all possible correlations between the columns. Furthermore, both models are trained using WGAN loss with a gradient penalty and ADAM optimization.     Figure 3. Generator Network Architecture.         Figure 4. Critic Network Architecture.     Symbol Meaning ⨁ Concatenation ReLU Rectified Linear Unit activation function BN Batch Normalization FC Fully Connected layer tanh Hyperbolic Tangent activation function Gumbel Gumbel Softmax link function Leaky Leaky ReLU   Table 1. Legend for symbols used in architecture equations.   There’s a lot more that could be said regarding CTGAN and a lot of detail was left out of this post for the sake of brevity. For more information, please refer to the paper and GitHub in the references section [2][3].   CorGAN   CorGAN follows a lot of the ideas introduced by Goodfellow and the overall architecture is similar to CTGAN but there are some key differences:   A denoising autoencoder is trained and the generator output is passed through the decoder to generate the synthetic sample. Unlike CTGAN, the input to the generator is only random noise and does not contain any real samples. This is true for both training and scoring.   The published autoencoder followed a very simple architecture where the encoder is simply a fully connected layer with a 128-dimensional output and tanh actication. The decoder also consists of a fully connected layer where the outputs are passed to a sigmoid function and the output dimensions match the training sample dimensions.   Just like CTGAN, a real sample and synthetic sample are then passed to the discriminator during training so that it’s able to learn whether an observation is real or synthetic. Due to the fact that we spent significant amount of time discussing CTGAN we will not be going in too much detail with regards to the CorGAN model/paper. For more information please refer to the paper and GitHub in the references section [4][5]   CPCTGAN   As stated earlier CPCTGAN, implements techniques from both the CTGAN and CorGAN models to generate synthetic tabular data. CPCTGAN combines the data transformation, conditional generator and training by sampling mechanisms from CTGAN. From CorGAN it adopts the mechanism to use a pre-trained autoencoder to then apply the decoder to the generator during model training.   SAS’ architecture for the autoencoder is also a lot more complex than the one published in the CorGAN GitHub repo for their proposed model.       Figure 5. CPCTGAN Autoencoder Architecture [6].   Since the autoencoder is also applied to the model architecture, overall it looks different than the previous two architectures.     Figure 6. CPCTGAN Entire Model Architecture [6].   Outside of the combination of those two models, there aren’t many additional differences between CPCTGAN and the other models.   tabularGanTrain Demonstration   This demonstration will be performed using Python; however, users should be aware that they can also use CASL and SAS Studio to perform everything in this demo. The first thing we need to do, like with most Python programs, is import the necessary packages:   # imports necessary packages import os import swat import pandas as pd   Package Use os Required to save files to disk swat Used to call the necessary CAS action sets, including the connection object and the model zoo actions. pandas Used to create dataframes/tables that can be saved in different formats.   Table 2. Python Package Description.   Now that we have imported the packages, we can then establish a connection to CAS using the SWAT’s CAS class and load the necessary action sets. In this demonstration I will load the dataPreprocess action set to impute missing values, the generativeAdversarialNet action set which allows us to train StyleGANs and CPCTGAN models, the percentile action set to assess models, and the sampling action set to partition data sets.   # Creates the connection object conn = swat.CAS("connection information") # Imports necessary action sets conn.loadactionset("dataPreprocess") conn.loadactionset("decisionTree") conn.loadactionset("generativeAdversarialNet") conn.loadactionset("percentile") conn.loadactionset("sampling")   To use the tabularGanTrain action all you need is the data set that you synthetically reacreate. In this demo I will use the home equity data set which is one commonly used around SAS. The link to download the dataset is listed in the reference. The data set contains information about customers such as their debt to income ratio, why they're requesting a home equity loan, their job category, if they've ever defaulted on a loan, etc. The data set has a target variable which states whether an individual defaulted on the loan or not.     Figure 7. HMEQ Sample.   As we can see in figure 7, there are a few data quality issues that need to be addressed. The dataset is mildly preprocessed, so there's not too much that we need to do. I performed simple imputation to replace missing values for the numeric and categorical columns. Additionally there are a few numeric variables with low cardinality that could be changed into categorical variables. Some categorical variables such as NINQ, DEROG, DELINQ, and CLNO were turned into binary categorical variables so that they simply indicate whether these events happen as opposed to the number of times that they occurred. The key reason for this is because these variables generally were very low cardinality with the majority of values taken being either 0 or 1. For that reason all categories that were greater than one were simply replaced with 1. The transformations aren't shown, but below you will see the necessary action in order for you to perform the imputation.   # Performs simple imputation on the continuous variables conn.dataPreprocess.impute( table = "hmeq", methodContinuous = 'MEDIAN', methodNominal = 'MODE', inputs = inputs, copyAllVars = True, casOut = dict(name = "hmeq", replace = True) )   Now that we have imputed our missing values we can proceed with generating the synthetic data by using the tabularGanTrain action. The options for the action include options for the optimization, number of samples that we want to generate, options for the gaussian mixture model, etc. In table 3 you will find the descriptions for the options that were used for this example.   # Creates the synthetic data conn.tabularGanTrain(table = dict(name = "hmeq2", vars = inputs + ["BAD"]), nominals = nominals, gpu = {"useGPU":True, "device":0}, optimizerAe = {"method":'ADAM',"numEpochs":20}, optimizerGan = {"method":'ADAM',"numEpochs":50}, miniBatchSize = 500, seed = 12345, scoreSeed = 1234, numSamples = 5960, saveState = {"name":"cpctStore", "replace":True}, casOut = {"name":"synth_data", "replace":True} )   The task requires that we specify which variables are our inputs, and out of those inputs we need to specify which variables are categorical. The variables that we specify both in the vars option and nominals options are the ones that we are going to synthetically generate. In this demonstration, all of the variables (numeric and categorical) were synthetically generated. A total of 5,960 samples (number of rows in HMEQ) were synthetically generated. ADAM optimization was used for both the autoencoder and GAN, the Beta1, Beta2, alpha and learning rate were not changed. Each epoch of training uses 500 rows of data. There are a lot of settings that could be changed, but the results given these minimal changes were satisfactory.   Option/Parameter Description table Name of the input table. Also used to specify the list of the inputs to be synthetically generated. nominals Specifies the name of the input variables that are nominal. gpu Specifies whether to use a GPU (if available) and which GPU (if multiple are available). optimizerAe Specifies the optimization settings for the autoencoder used in the model. optimizerGan Specifies the optimization settings for the GAN. miniBatchSize Number of observations used in each training iteration. seed Seed to use for the random number generator for training. scoreseed Seed used for the random number generator for scoring. numSamples Number of samples that will be synthetically generated. saveState Specifies the table in which to save the model state for model scoring. casOut Specifies the output CAS table in which to store the generated tabular data from the trained model   Table 3. tabularGanTrain options and parameters.   Model Assessment   Note that the action set has no way to accept a validation set. This makes sense considering there is no way to "how good" the synthetic data is. A simple way to start to assess the model is by  generating summary statistics for all of our columns.     Figure 8. Real Data Summary Statistics.     Figure 9. Synthetic Data Summary Statistics.   As we can see the Summary statistics for both data sets look similar with some columns more than others (more on these observations later). There are multiple ways that we could potentially assess this data, one way we could assess the continuous variables could be by performing two sample t-tests to compare the means (comparing one real column mean to it's corresponding synthetic column mean). With regards to the categorical variables, we could perform tests of associations to measure whether there is a difference or not between the distributions of the levels. Although there are many ways to assess this model based on the use case, in this post the synthetic data set will be assessed in one of the ways proposed by Torfi and Fox in their CorGAN paper. This method is to train a predictive model using the real data, train a separate predictive model using the synthetic data, then assess both models against the real validation data set. Once the model predictions are generated, their performance can then be compared. Note that the model trained using synthetic data does not get any real data, even the target variable is synthetically generated. This assessment approach could be useful in a situation where synthetic data is being used in a machine learning context, although not every use case may benefit from assessing models using this approach.   In this post I will train gradient boosting models to assess the difference in performance between a model trained using the real data and one trained using the synthetic data. Both models were made using the same exact hyperparameter settings and the same training and validation partition sizes. Below you will find the code that was used to train both models:   # Creates a list of the dataset that will be used to create the models datasets = ["hmeq", "synth_data"] # Splits the original and synthetic datasets into training and validation for data in datasets: conn.sampling.srs( table = data, samppct = 70, seed = 919, partind = True, output = dict(casOut = dict(name = data, replace = True), copyVars = 'ALL') ) # Trains a gradient boosting model using the original and synthetic datasets for data in datasets: model_name = "gboost_" + data conn.decisionTree.gbtreeTrain( table = dict(name = data, where = '_PartInd_ = 1'), target = target, inputs = inputs, nominals = nominals, m = 8, nBins = 100, nTree = 100, casOut = dict(name = model_name, replace = True) ) # Scores both of the gradient boosting models for data in datasets: model_name = "gboost_" + data output_tbl = "scored_" + data assess_tbl = "assessed_" + data conn.decisionTree.gbtreeScore( table = dict(name = "hmeq", where = '_PartInd_ = 0'), model = model_name, casout = dict(name=output_tbl,replace=True), copyVars = target, encodename = True, assessonerow = True ) conn.percentile.assess( table = output_tbl, inputs = "P_BAD1", casout = dict(name=assess_tbl,replace=True), response = target, event = "1" )   The final set of assessed tables contain information that can be used to construct ROC curves and lift curves. For this post, I decided to use accuracy and misclassification error to assess these models since the only goal here is binary classification. Choosing a probability cutoff of 0.5 the following accuracy and misclassification results were obtained:     Figure 10. Real Data vs. Synthetic Data Gradient Boosting Assessment (No Pre-processessing).   As we can see the gboost model trained with the real data has an accuracy that's approximately 14% better than the model trained using synthetic data. Overall for a model that was trained with purely synthetic data, the performance isn't bad. A small confession is that the results in figure 10 are for models that were trained before the continuous variables were standardized.   Observations and Tips   Observing the results generated by the GAN made me realize a few things that a user needs to be aware of:   The GAN does not recognize limits with numeric columns. E.G. In the real world, a borrower cannot receive a negative loan. Therefore, if a column has a lower limit of 0 (or an upper limit), the GAN will not be able to identify this and will generate values that exceed the limits. The GAN will generate floating point values for continuous integer columns. Columns such as the number of times a borrower has gone delinquent on their loan will generally be represented with integer values and the number of unique values are often very low. As such it may be better to treat these columns as categorical. The key reason for these suggestions is because the GAN will generate values that may exceed the upper and or lower limits (as outlined in point 1) and two it will produce floating point values which may require the user to have to perform post-processing that will involve rounding/interpreting the numbers. Performance seems to be worse on categorical columns, especially columns containing rare levels (a known issue addressed in [2] and [4]). For this reason it may be advantageous to bin categorical columns to reduce the overall amount of levels (especially rare levels). Other techniques such as weight of evidence encoding which can reduce the cardinality of your variables and convert them into numeric values may also improve the performance on categorical variables. The GANs work seem to work much better on numeric columns with high means. It could be a function of the data considering that the variables in the data set with lower means also tended to have lower limits and lower cardinalities.   With all those observations in mind and out of the way, I wanted to perform an additional test and this is when I decided to normalize the numeric columns prior to generating the synthetic data. The main reason behind this approach was to make it so that our continuous columns fall along a normal distribution and can now generate better values (also negative outputs are not explicitly invalid). Better standardization techniques probably exist, but this was a quick one I could implement to test my observations. A downside of this approach is that it makes the model less interpretable, which could make this a harder decision to make in certain scenarios. After training the GAN and GBoost models using this new pre-processed data, there was an immediate increase in the performance of the model.     Figure 11. Real Data vs. Synthetic Data Gradient Boosting Assessment (No Pre-processessing).   There are a few other ways that one could potentially improve this model. One of them is that there really isn't a limit with regards to how many synthetic observations the user could generate. For the sake of this demonstration both the real and synthetic training data set had the same amount of observations but it's possible that the sample used wasn't enough to completely represent the joint probability distribution of the data set. I took some steps to address some of the issues associated with the cardinality and distribution of certain variables, and I am sure that there are more ways to potentially preprocess these columns that could yield better performance. Different normalization techniques for the continuous columns is something that could also yield better results.   In conclusion, this technique can prove to be very useful whenever we would like to generate data with a similar distribution to protect personally identifiable information (PII), to present information to external organizations or other internal divisions, teaching courses, or anything in between. This technique could also be used as a data augmentation technique to enhance smaller training data sets and potentially improve the performance on validation. I'm sure there are many other ways someone could use this, and if you have more suggestions, I would love to know!   References: Generative Adversarial Nets (2014) Modeling Tabular Data using Conditional GAN (2019) CTGAN GitHub Repository CorGAN: Correlation-Capturing Convolutional Generative Adversarial Networks for Generating Synthetic Healthcare Records (2020) CorGAN GitHub Repository SAS Documentation - Generative Adversarial Network Action Set Details Download HMEQ Dataset   Find more articles from SAS Global Enablement and Learning here. ... View more

If computer vision sounds interesting to you, then there is no better place to get started than image classification. Image classification is one of the fundamental techniques in computer vision, which enables machines to identify and categorize objects within images and, in many ways, is the basis for computer vision. Image classification algorithms are used in many industries including manufacturing, healthcare, security, retail, and many more. In this post I will introduce image classification using the model zoo action set. The goal of this post is to provide individuals new to computer vision with a basic understanding of how these algorithms work, and for individuals familiar with image classification, how to get started with the model zoo action set which is a new SAS action set for deep learning.   Image Classification Introduction   Image classification is ultimately a pattern recognition task where machines learn from a dataset of labeled images to predict the class (or label) of unseen images. Broadly speaking image classification algorithms are a subset of machine learning algorithms, which means that we need a training dataset to build the model. The training data set consists of images and the label associated with those images is the target. An example of what an image classification algorithm does in deployment is if it sees an image of dog (assuming it was trained on a dataset that enables it to classify dogs) it will generate a label for said image.   Working with Image Data   The data required to create an image classification model are going to be images, and the labels associated with the images. There's a variety of different ways SAS can read the images and their corresponding labels. One way is by placing all images belonging to a given class inside of a folder, where the folder name is the label associated with the images within said folder. SAS can then be look inside the folder and create a table containing the image data. Another way is to provide SAS with a csv containing both the image location and their labels, during model training and validation model zoo takes care of the rest. Both methods are very similar, it's a matter of convenience which technique is used.     Figure 1. CSV Containing Image Paths and Labels. Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post.   We now know how SAS can access our image data, but how are they processed and where is the data contained in the images? In traditional supervised machine learning we generally work with structured data, which usually comes in the form of rows and columns. In images the data we work with are the pixel values that when combined come together to generate said image. If you consider a row of data from a table as 1-dimensional (with n number of inputs), then you can probably start to imagine that images should contain information in at least two dimensions since they are a collection of pixel values arranged in a specific way. Regardless of the type of image we may end up working with, their pixel values will range from 0 to 255. In a grayscale (black and white) image, a pixel value of 0 represents black and a value of 255 represents white. Everything in between is some combination (gray) of the two. Color images are more complex because we must combine different colors to get our desired color. In the case of color images, we generally achieve this by combining red, blue, and green pixel values. Whenever we’re working with images we call each two-dimensional array corresponding to a specific color, a channel.   Figure 2. Grayscale and RGB Channels Example.   Both grayscale images and RGB images have pixel values ranging from 0 to 255, where a value of 0 corresponds to the absence of that color, and a value of 255 corresponds to the highest intensity of said color. In color images when we stack the RGB channels on top of each other we generate more complex colors since we combine the red, green, and blue channels to varying degrees. Now that we know the information that is contained in images, let's move forward and talk about how we can extract this information.   Fully Connected Neural Networks   At the heart of image classification lies the concept of feature extraction. This process involves capturing distinctive characteristics from images that differentiate one class from another. These include things like the edges of objects, textures, colors, etc. In a sense, this is similar to how we process visual information. We can see the shape of objects, silhouettes, color, lighting, and interpret what we’re seeing.   Although other algorithms have worked for image classification in the past, modern day computer vision applications use neural networks as their backbone. It should also be noted that there are different types of neural networks. Traditional neural networks, like the ones used for traditional machine learning tasks, are generally not the type of network used for computer vision. In these traditional neural networks, often called fully connected networks, all of our inputs are passed to every single hidden unit (also called neurons), where each hidden unit is a nonlinear function that transforms the inputs to learn complex nonlinear functions.     Figure 3. Single Layer Fully Connected Network with 3 Hidden Units and Logit Link Function(left). Decision Boundary (Right).   Fully connected neural networks can have multiple fully connected layers that ultimately result in either a shallow or deep neural network. After the final fully connected layer, the outputs of neurons are combined to generate some output that matches the distribution of the target.      Figure 4. Generalized Fully Connected Neural Network Diagram.   Although the neural networks that we use for computer vision are different than fully connected neural networks, a lot of the concepts are the same. There are multiple reasons why fully connected neural networks are not used for computer vision applications. When training a network, instead of having an observation correspond to a row from a table, now a single observation is an entire image. Consider a case where we have 32 x 32 pixel images (approximately the size of a thumbnail on a forum) although small, that corresponds to 1024 pixels per image. In other words if you flatten out the image that would be like a row with 1024 columns! From the get-go, this makes our fully connected neural networks very computationally expensive.  Another issue as well is that pixel values are all related to one another. Imagine an image of a dog, realistically you could look at small regions of the image (parts of the dog such as ears, nose, eyes, tongue, paws, etc.) and classify the image as a dog without seeing the entire thing. So in a sense, pixels generally have some form of relatedness to pixels in their surrounding regions. Fully connected neural networks are not great at extracting this type of relatedness. For those reasons and more outside the scope of this blog, instead of using fully connected neural networks for image classification we instead use convolutional neural networks (CNNs).   Convolutional Neural Networks (CNNs)   CNNs, inspired by the visual processing of the human brain, have become the backbone of computer vision. These architectures are designed to learn hierarchical features from images, starting from simple features like edges to more complex ones. These networks are not just multiple layers of fully connected neurons, instead they incorporate a sequence of techniques that help the network extract information from images. Although there are many techniques that are used in modern day CNNs, the key techniques introduced in convolutional neural networks that helped them take a leap forward in the realm of computer vision are convolutional layers and pooling layers.     Figure 5. Convolutional Neural Network Layers.   In the image above we illustrate a very general architecture for a CNN. Because we no longer have only fully connected layers, we can create neural networks architectures that will look very different and work well on different tasks. Almost all convolutional neural networks will contain convolutional layers and generally some type of pooling layer. Before we introduce the architecture that we're going to be using, we need to develop a basic understanding of convolutional and pooling layers.   Convolutional Layers   Convolutional layers are generally used after the input layer and throughout different parts of the CNN. The convolutional layers use what are called filters (also called kernels), which help capture edges and contain learnable parameters that help them extract key pieces of information given a training dataset.  In a fully connected neural network, the learnable parameters are contained within the neurons where we have one learnable parameter per input. In the output layer of fully connected neural networks we also have one learnable parameter for each output. In convolutional neural networks the filters contain the learnable parameters which it learns through multiple iterations of an optimization process using the training data set (similar to fully connected networks). What makes these filters special is that they help us address the issues fully connected neural network have when trying to perform image classification. The parameters in a filter are shared for multiple regions of an image/input, and the filters are applied for a region of the image. Filters make it so that we have a computationally efficient solution to processing images; they capture some of the interrelatedness of pixels in a region, and add invariance to the network. Each filter is going to generate an output called a feature map. The way that a filter generates the feature map is by computing a product between the parameters weights and a small region of pixels. This is illustrated in the figure below:  Figure 6. Cross Correlation Operation.   The operation is formally known as the cross-correlation operation but might sometimes be referred to as a convolution without the kernel flipping. The feature map then gets an activation function applied to it, just like neurons in fully connected networks. Filters also have hyper parameters associated with them, such as their width, height, and stride (how many rows/columns they move at a time). Convolutional layers are just one part of the story, the other layer type that makes convolutional neural networks very powerful are pooling layers.   Pooling Layers   Pooling layers help to summarize regions of incoming information, help to increase the invariance of CNNs, and make the networks more computationally efficient. Pooling layers are summary operations, with the most common pooling functions being max, average, and min pooling. There are other kinds of pooling layers, but that is outside the scope of this blog.     Figure 7. Common Pooling Layers.   Because filters apply a summary function to a given region, they are generally combined with a larger stride value in order to down-sample information. Pulling layers, as opposed to convolutional layers, are generally used more freely because they do not contain learnable parameters and therefore do not increase the computational requirements of networks too much. One of the theoretical motivations for pooling layers is that every single pixel value in an image does not need to be analyzed in order to get the main piece of information from a given region. Similar to how you can see a portion of an image, say the ear of a dog, and you can correctly deduce that the full image should be a dog.   Fully Connected Layers   Convolutional neural networks also include fully connected layers. They tend to appear at the end of the networks before we get to the output layers. Fully connected layers work in the same way as they did in fully connected neural networks. Before using these in a CNN we generally flatten the inputs to these layers such that they’re all contained in a single vector that can then be passed to every hidden unit in the fully connected layer (like a row in a table). Fully connected layers is where CNNs get very computationally expensive, and for this reason we tend to use pooling layers prior to this point to reduce the dimensionality of the incoming information. Just like with fully connected neural networks, the output of the fully connected layers gets passed to an output layer where a link function is used to compute the class probabilities. In the case of multiclass classification, the link function tends to be the softmax function which generates the probability of an observation belonging to each of the classes. Whichever class has the highest probability is the class that the observation will be assigned to.   LeNet5   We could spend more time talking about different techniques used in CNN's, different layers, different architectures, etc., however the goal of this blog is to give users a basic understanding of image classification. Not much else is needed outside of convolutional layers, and pooling layers in order to get a working model for image classification. To illustrate this point I will use a convolutional neural network known as LeNet-5.  LeNet-5 was introduced by Yann LeCun and others in 1998 and was one of the earliest successful CNN architectures. It featured alternating convolutional and pooling layers, culminating in fully connected layers for classification. While simplistic by today's standards, LeNet-5 laid the groundwork for future advancements in CNNs.     Figure 8. LeNet-5 Architecture.   SAS allows us to build our own custom CNNs, which for the dataset chosen in this blog, building a CNN from scratch and getting good results is very feasible. However, I will opt to use this well known architecture, LeNet-5, because this mirrors how computer vision users commonly train computer vision models. Commonly a popular architecture that is known to perform well on a given task is used as opposed to constantly building new models from scratch.   Introduction to Model Zoo   Now that we have a background in computer vision and the LeNet-5 architecture, we can now start to discuss model zoo. Model Zoo is a CAS action set for deep learning based on the PyTorch programming framework available since the 2022.09 LTS release of Viya. The action set has a backend that calls the PyTorch C++ API and a front end that is responsible for initiating actions, terminating actions, reading CAS tables, writing CAS tables, and communicating with the backend portion. Aside from the architectural differences, Model Zoo takes a different approach to deep learning than the traditional deepLearn CAS action set. The process to build a neural network that works well on specific types of data can be a very time-consuming process that does not always yield the best results. For this reason, many users choose to use neural network architectures that have been published by reputable research groups that perform very well on specific tasks. These groundbreaking neural network architectures are generally published alongside an open-source implementation that oftentimes make their way to some of the most popular open-source deep learning libraries such as TensorFlow and PyTorch. For those reasons, Model Zoo currently puts a focus on using pre-defined model architectures as opposed to building your models directly in Model Zoo, which is how many users use deep learning in practice.   Model Zoo is a great tool for open source developers that have access to a SAS Viya deployment. Model Zoo can be used via Python or R through the SWAT package. The SWAT package acts as an API that allows you to submit CAS actions from Python or R as well as integrate these into classes, functions, conditional statements, etc. Additionally, if you're a deep learning developer familiar with PyTorch you can continue to use it to develop your models. Models created using PyTorch can be saved as torchscript files, wrapped in a few additional classes, then trained using CAS. The key advantage that this gives users is the ability to automatically have these models parallelized by CAS, and if GPUs are available you can leverage those as well. There is a lot that can be said about defining custom models using PyTorch then importing them into Model Zoo, but that is beyond the scope of this blog. Model Zoo is going to be one of the key SAS tools for deep learning over the coming years; so as a result, readers can expect a lot more functionality to be added to the action set over the coming years.   Model Zoo LeNet-5 Example   This demonstration will be performed using Python; however, users should be aware that they can also use Model Zoo in R or CASL. The first thing we need to do, like with most Python program, is import the necessary packages:   import os import swat import yaml import pandas as pd   This may end up looking like a typical list of packages when using Model Zoo. Below is a table briefly describing the use of each package in this demonstration:   Package Use os Required to save files to disk yaml Used to ensure that the YAML files used by Model Zoo are accurate and complete. swat Used to call the necessary CAS action sets, including the connection object and the model zoo actions. pandas Used to create dataframes/tables that can be saved in different formats.   Table 1. Python Package Description.   Now that we have imported the package, we can then establish a connection to CAS using the SWAT’s CAS class and load the necessary action sets. In this demonstration I will load the dlModelzoo action set, the image action set to be able to explore images saved in CAS, and the sampling action set to partition tables.   conn = swat.CAS("connection information") conn.loadactionset("dlModelzoo") conn.loadactionset("image") conn.loadactionset("sampling")   Now that we have done some of the basic setup, we can start getting into the details of how to build your image classification models. We require a table that contains the images and their associated labels or a CSV that contains the image path (including image name, extension, and path relative to a CAS library). For this demonstration, the CSV shown in figure 1 was read into memory using the read_csv method from the connection object.   # Read CSV into memory conn.read_csv('mnist_fashion_labels.csv', casout=dict(name='mnist', caslib='mycl', replace=True))   Something different between model zoo and the deepLearn CAS action set is that we need to create a YAML file that is used by the actions to specify the model and/or data set specific parameters. Although this YAML file may seem strange, it gives users the flexibility to have a fixed location where they can quickly make changes to the model such as image transformations, input size changes, number of target classes in the chosen algorithm, etc. It is possible to have all the information within a single YAML file, however for the sake of readability I generally prefer to split it up into two YAML files; one for training and one for scoring. One suggestion that I would like to give users when creating the YAML file, is to use a text editor such as VSCode or notepad++. The main reason for this is that the YAML file is sensitive to spacing, and the sub sections are tab delimited. Below is the YAML file that was used to train the model as well as a table explaining the options used in the YAML file:   documents = """ sas: dlx: train: label: "lenet5_mnist" dataset: type: "UNIVARIATE" organization: "CoCo" preProcessing: - modelInput: label: input_tensor1 imageTransformation: resize: type: TO_FIX_DIM size: 32 32 imgStdType: STD model: type: "TORCHNATIVE" name: "SAS_TORCH_LENET" classNumber: 10 caslib: "mycl" inputs: - label: input_tensor1 size: - 0 outputs: - label: output_tensor1 size: - 0 """   Train and Score Description label Used by the train and scoring actions to call a specific section of the YAML file dataset Specifies the dataset type the action receives and the type determines how the data is read in and processed. dataset.type Supported dataset types are Univariate, ObjDetect, Segmentation, and Autoencoder. preProcessing Specifies the pre-processing (augmentation) of the input data as well as the target data. preProcessing.modelInput The action looks for the matching label in the model.inputs section to find the stream of input to apply the pre-processing to. modelInput.label The label gives the inputs a name and indicates the inputs (data) that the pre-processing will be applied to. Used by the actions to find the stream of inputs to apply the pre-processing to. modelInput.imageTransformation Allows users to apply transformations to the image, such as resizing, color transformations, random transformations, etc. imageTransformation.resize Used to resize the inputs to a specific height and width imageTransformation.imgStdType Normalizes the image pixel values such that each value falls in a range of [0, 1]   Table 2. Train and Score YAML Options.   There are options such as dataset.organization that are just for the user to have a little bit more information that are not used by the Model Zoo CAS actions. Your YAML files should start off with SAS, DLX, and then the subsection after DLX can either be train or score depending on whether you will use this YAML file to train the algorithm or score using the algorithm. Most of the options specified under the training subsection are explained in the table above, so I will not be explaining those options.   Dataset and Pre-Processing   Before we go any further, we first need to discuss the dataset that we’re going to be using in this demonstration and try to understand it a bit more. We’re going to use a classic dataset in computer vision, the MNIST Fashion dataset. This full dataset contains 60,000 images split into 50,000 images for training and 10,000 belonging to validation. For the sake of our demo we're using a smaller version of this dataset containing only 10,000 images for training and 5,000 for validation. The dataset consists of 10 classes of grayscale images, where the classes are different articles of clothing such as shoes, shirts, skirts, etc. In this dataset all the images are PNG files of the same size, 28 x 28. This makes this dataset very popular for learning image classification since the overall computational requirements to process data of this size isn’t massive. This dataset doesn’t need too much in terms of pre-processing. The only transformation that we’re applying to the inputs is a standardization of the image pixels. Image pixels range from values between 0 and 255, so to ensure the weight updates/values are not too drastic in scale and for information to flow better through the network, standardization techniques such as this one are common. Note that there are many preprocessing techniques available in model zoo, however, in this example I chose not to apply any outside of this one. The second important subsection that we need to specify is the model subsection, here is where we can give model zoo more details about what kind of model we want to build. The table below explains some of the options that were used:   Model Options Descriptions type Currently supports two types of models, TorchScript and TorchNative. TorchNative are models that are pre-written in C++ and built into the action library. TorchNative is used for pre-built models such as YOLO and UNet. TorchScript is used to import custom models written in PyTorch. name Used only for TorchNative models to specify what type of model the user wants to use. caslib CasLib where model weights can be loaded and saved to. classNumber Number of classes in our targets. inputs Can be used to apply changes and transformations to the input and target tensors. inputs.label & outputs.label Specifies the name of the tensor to be modified. inputs.size & outputs.size Can be used to reshape the specified tensor to a given size. A value of 0 means no reshaping, if reshaping, then values need to be specified as: - Channel - Height - Width   Table 3. Model YAML Options.   Underneath the model subsection the first thing that we need to specify is what type of model we want to build. TorchNative corresponds to the pre-built models, while TorchScript correspond to user defined models using PyTorch. Afterwards you need to specify the name of the model that you want to build, which in this case SAS_TORCH_LENET. The ModelZoo documentation shows which models are currently available to use within the action set, with more on the way in the future. Another necessary option when building a classification model is the number of classes. The objective of the model in this case is to distinguish between the 10 different image types, so in this case my class number is going to be 10, one corresponding to different articles of clothing. We do not need to resize the input or output tensors, so we leave the size option as 0. That completes the YAML file, so next we can check to see whether it is syntactically accurate by using the YAML package along with the following lines of code:   for data in yaml.load_all(documents): print(data)   If the YAML file is syntactically correct the output will be a JSON string that displays all the user specified options. Keep in mind that this doesn't ensure that you won't have any errors whenever the YAML file is used to train or score the model, this only ensures that the YAML file is syntactically correct. In other words that every single option is delimited and spaced correctly.   Now that the YAML file is complete and syntactically correct, we can move forward with training the image classification model. The way that we're going to do that is by using the dlmztrain action from the Model Zoo action set. Below is the dlmztrain action as well as a table with a description of the options being used in the action:   train_action = conn.dlmztrain(loglevel = "DEBUG", table = dict(name = "mnist", where = "_PartInd_ = 1"), validationtable = dict(name = "mnist", where = "_PartInd_ = 0"), inputs = "_path_", targets = "xlabels", ngpus = 1, indexvariables = "labels", modelOut = dict(name = "LeNet5", replace = True), outputIndexMap = dict(name = "LeNet5_outputindex", replace = True), dropLast=True, optimizer=dict(loss='cross_entropy', mode=dict(type='synchronous', syncFreq=1,), algorithm=dict( learningRate=0.002, momentum=0.5, method='sgd', weight_decay=0.0005 ), batchSize = 32, seed=12345, maxEpochs=50), options=dict(yaml = documents, label="lenet5_mnist"))   dlmzTrain Action Options Description loglevel Reporting level for progress messages sent to the client. DEBUG allows users to see more information related to the training process. table CAS Table that is used to store the input data for the training step in training a deep learning model. inputs The input variables for the training task. Currently, we support image data as input, the input column can either be a string of an image path or a binary of the image data. targets The target variables for the training task. ngpus Used together with HyperParameter Tuning, specifies the number of GPU's to use across the entire grid. The GPU's with the lowest amount of memory currently allocated will be chosen. This option is mutually exclusive with the gpu option. validationTable The CAS table that contains the validation dataset. Used to assess model weights after each epoch. modelOut The CAS table used to store the trained model and model weights. checkPointBest Specifies whether to save the model weights that performed best on validation or the model produced in the final epoch. outputIndexMap The output CAS table containing a mapping from nominal class type to numeric values. The Index table of nominal class type to numeric values built in the training process is written to the CAS table at the end of the action. optimizer Key component of training any type of neural network. Currently supports a variety of optimization algorithms including SGD, Adagrad, ADAM, and ADAMW. tuner Specifies settings for hyper parameter tuning learningRateScheduler Used to specify a learning rate policy, such as a fixed learning rate or one that’s modified after n number of steps. extraOptions Used to specify the YAML file that should be read when training or scoring model as well as which section to read by using the label option.   Table 4. dlmzTrain Action Options.   The first option I would like to highlight is the log level option. There are a variety of values that it could take. Personally, I like to set the log level equal to debug so that I get more messages about the training process. In the table option we specify the name of the CAS table that we created by using the loadimages action. The CAS table contains the names of the images, the location in memory of the images themselves, and their corresponding labels. We use the inputs option to specify the name of the column that contains the images, and the targets option to specify the name of the column that contains the labels associated with the image. If we are using a machine with GPU capabilities, we can use the GPU equals option and specify the number corresponding to the GPU that we want to use.   Our model weights are going to be stored in a CAS table called LeNet5 and the weights that will be stored are the weights associated with the iteration that yielded the best performance on validation thanks to the checkPointBest option. The optimizer parameter is very important because this is the process that perform to optimize the model weights. The loss function is used to specify how the loss will be calculated. Loss functions are very closely related to the distribution of the target and it's number of classes, as well as what type of model you may be trying to build. The algorithm option allows us to specify the details regarding the algorithm we want to use such as the learning rate, momentum, optimization technique, weight decay and more. Other options that I have specified in the optimizer option include the batch size, which is the number of images that I want to use for every iteration of the optimization algorithm, the seed to make the optimization reproducible, and the max epochs which specifies how many iterations of the optimization I want to go through. The learning rate scheduler can be used to specify whether we want a fixed learning rate for the optimization, or whether we want the learning rate to be changed throughout the optimization. In this demo, the learning rate is adjusted after every 10 epochs. Lastly, we use extraoptions to specify the name of the YAML file, and which section of the YAML file should be read to train the model by using the label option.      Figure 9. dlmzTrain Action Output.   Keep in mind that the output will differ depending on the log level, a level of debug gives a lot of information regarding the training process. I can also see information about the model such as the name of the model, the kind and number of layers in the model, how the loss function progresses, as well as the misclassification error. At the conclusion of the training process, we get a reason as to why the optimization stopped and a message stating that the action completed successfully. Now that the model is trained, we can move to start scoring with the model. To score using our model, we need instructions in a YAML file once again. The good thing is that the instructions for this YAML file look very similar to the ones in the training YAML file. # Specifies the YAML file to score using the model   score_doc = """ sas: dlx: score: label: "lenet5_mnist" dataset: type: "UNIVARIATE" organization: "CoCo" # enum[IMAGENET, OPENIMAGE, COCO] preProcessing: - modelInput: label: input_tensor1 imageTransformation: resize: type: TO_FIX_DIM size: 32 32 model: type: "TORCHNATIVE" # enum [TORCHSCRIPT, TORCHNATIVE] name: "SAS_TORCH_LENET" # ignored when type is "TORCHSCRIPT" classNumber: 10 caslib: "mycl" inputs: - label: input_tensor1 size: - 0 outputs: - label: output_tensor1 size: - 0 """   The main differences between this YAML file and the one that we use for training, is that the third sub option is score instead of train, outside of this change everything remains the same. With a complete YAML file we can now use the dlmzscore action. Below we can see the code as well as a table that provides a brief description for all the options that were used within the action:    # Scores the validation datset score_action = conn.dlmzscore(modelTable = "LeNet5", table = dict(name = "mnist", where = "_PartInd_ = 0"), inputs = "_path_", targets = "xlabels", batchsize = 16, gpu = {0}, loglevel = "DEBUG", tableout = dict(name = "lenet_output", replace = True), options = dict(yaml = score_doc, label = "lenet5_mnist"))   dlmzScore Action Options Description loglevel Reporting level for progress messages sent to the client. DEBUG allows users to see more information related to the training process. modelTable The CAS table containing the model and model weights. This parameter is optional. When it is specified, the table stores a binary blob containing the model and weights in Pytorch format; when it is not specified, you can specify the model table file path relative to a CAS table path in the YAML file in the extraOptions parameter. table The input CAS table that is used to store the input data for scoring a Deep Learning model. inputs The input variables for the scoring task. Currently, we support image data as input, the input column can either be a string of an image path or a binary of the image data. targets The target variables for the scoring task. batchsize Number of images to score each epoch. GPU Specifies the GPU to use when scoring. tableOut Output table used to store the output generated from scoring. extraOptions Used to specify the YAML file that should be read when training or scoring model as well as which section to read by using the label option.   Table 5. dlmzScore Action Options.   Similar to the dlmztrain action, the amount of information that we get as part of the output, is going to depend on the log level value. Since the log level is once again set to debug, we get as much information as possible regarding the scoring process. Just like before, we see information regarding our model such as the number of layers, the kinds of layers, the number of parameters, etc.. We also see how the loss and MCE changed as we scored our batch of validation images. The tableout option specifies the name of the output table that will contain the predicted labels for the validation images, the table generated by the score action was named lenet_output.     Figure 10. dlmzScore Action Output.   The output table is overall very simple as it contains just the label associated with the validation image, however it can be modified to contain more information such as the image for which we are generating the prediction for.      Figure 11. First 5 Rows of Validation Images with True Label (left) and First 5 Predicted Labels (right).   Your image classification may be a part of a much larger process that involves other moving parts but realistically these predictions can be used to take real world actions. There are a lot of things that could be used to improve this model like various image augmentation techniques, shuffling the images, changes to the optimization settings, etc. However, the purpose of this blog was to introduce image classification to those unfamiliar with it and to introduce modelzoo to existing deep learning practitioners.   Additional Resources: Model Zoo Action Set LeNet Model Reference Using YAML Specified Options SWAT GitHub Documentation Fashion MNIST Dataset   Find more articles from SAS Global Enablement and Learning here. ... View more