This information was a bit hidden in the documentation, so I can see how this would be confusing!
The concatenation layer will combine the source layers along the "last dimension" (Here is a reference from DLPy, a Python package that SAS has released to make it easier to work with the SAS deep learning tools: https://sassoftware.github.io/python-dlpy/generated/dlpy.layers.Concat.html; as a side note, the DLPy package calls the SAS Viya CAS actions behind the scenes so the functionality of the Concat layer in DLPy is the same as the Concat layer you would use from PROC CAS in the SAS Deep Learning course).
In this case by "last dimension" the documentation means that it will only concatenate the source layers along the "depth" axis (the 3rd axis). There is a note in the SAS Deep Learning course notes on page 2-43:
Note: Concatenation layers require that the incoming feature maps be the same size. It is a common practice to down-sample larger feature maps when combining feature maps of varying sizes. To down-sample, simply increase the stride value to a value greater than one.
Given that the feature maps have 3 dimensions (width, height, and depth) this statement might be slightly unclear. To clarify, the concatenation layer requires that all of the incoming feature maps have the same values for the width and height, and this is because it will combine all of the incoming feature maps along the depth dimension.
A little toy example, with dimensions specified as (width, height, depth):
Layer1 outputs a feature map with dimensions (4, 4, 3)
Layer2 outputs a feature map with dimensions (4, 4, 1)
Layer3 outputs a feature map with dimensions (4, 4, 1)
Layer4 outputs a feature map with dimensions (5, 5, 3)
A concat layer would be able to combine Layer1, Layer2, and Layer3 to output a feature map of dimension (4, 4, 5), combing the feature maps along the depth axis.
If we tried to include Layer4 in the concat layer we would get an error related to a dimension mismatch. This could be solved by passing Layer4 through a convolutional or pooling layer to resize the feature map to (4, 4, 3) so that the width and height match the rest of the layers input to the concat layer.
This is a great question and I think others will have the same question and find the topic here useful. Thank you for posting this!
-Ari
