Error when fitting a CNN model using SAS-DLPY and SAS Viya

[JianShen]

I'm trying to fit an image classification model that I created in Python. I made the model using the SAS-DLPY package in Python. All the steps leading up to the fit is going well (loading images, shuffling the data, creating patches, and creating a simple CNN model). 

Yet, the moment I try to fit the model, I run into 2 situations:

Situation 1:

dc_model.fit(data = train,
             mini_batch_size = 32,
             max_epochs = 5,
             lr = 0.001,
             log_level = 2)

Using the fit above, everything works fine. I get a fitted model that I can check the performance for (e.g., misclassification rate and loss). 

Situation 2:

dc_model.fit(data = train,
             mini_batch_size = 32,
             max_epochs = 7,
             lr = 0.001,
             log_level = 2)

However, using the fit in situation 2 (with the only difference being max epochs = 7 instead of 5), I get the error below:

Traceback (most recent call last):
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\swat\cas\rest\connection.py", line 434, in invoke
    self._results = json.loads(a2u(res, 'utf-8'), strict=False)
  File "C:\Python\lib\json\__init__.py", line 367, in loads
    return cls(**kw).decode(s)
  File "C:\Python\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python\lib\json\decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-9003bd2a9e43>", line 5, in <module>
    log_level = 2)
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\dlpy\model.py", line 281, in fit
    model_weights=dict(replace=True, **self.model_weights.to_table_params()))
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\dlpy\model.py", line 421, in train
    rt = self._retrieve_('deeplearn.dltrain', message_level='note', **parameters)
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\dlpy\network.py", line 247, in _retrieve_
    return self.conn.retrieve(_name_, _messagelevel=message_level, **kwargs)
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\swat\cas\connection.py", line 1692, in retrieve
    signature = self._invoke_with_signature(a2n(_name_), **kwargs)
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\swat\cas\connection.py", line 1171, in _invoke_with_signature
    self._invoke_without_signature(_name_, **kwargs)
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\swat\cas\connection.py", line 945, in _invoke_without_signature
    errorcheck(self._sw_connection.invoke(a2n(_name_), kwargs),
  File "C:\Users\WenkuoChen\PycharmProjects\NVWA Pilot\venv\lib\site-packages\swat\cas\rest\connection.py", line 441, in invoke
    raise SWATError(str(exc))
swat.exceptions.SWATError: Expecting value: line 1 column 1 (char 0)

I have no idea what's causing this, since it only seems to occur when I increase the model complexity/demands on system resources. For example,if I increase the amount of training images by increasing the amount of patches created, I get the same error.

The SAS Viya environment that I'm using has no GPU's, and a low amount of CPU's (8 I've been told). I was wondering if it's possible that the lack of GPU's (and low amount of CPU's) might be the cause? Or is there something else that I'm missing?

I've included my full script. The images that I'm using are JPG images, with names like dog.1.jpg. The Python interface that I use is Pycharm (version 2018.3.1), with SAS-DLPY version 1.0.2 and Python version 3.6.

Hopefuly, someone can enlighten me on this error!

[joeFurbee]

Hi @JianShen,

I ran you error past a couple of engineers. They don't believe it's an issue with DLPy, but an issue with connection loss and not returning anything. They are still looking into it. In the meantime, can you try a couple of other parameter changes?

What occurs when you set mini_batch_size = 8 (mini_batch_size is important as it sets the required memory independent of how many images in a table)? What occurs when you set max_epochs = 6?

Thanks,

Joe

[JianShen]

Hi @joeFurbee,

Thank you for the quick reply!

Based on your suggestions, I've tested the following fit-settings:

dc_model.fit(data = train,
             mini_batch_size = 32,
             max_epochs = 6,
             lr = 0.001,
             log_level = 2)

Using max_epochs = 6 and mini_batch_size = 32, the fit runs without any errors.

dc_model.fit(data = train,
             mini_batch_size = 8,
             max_epochs = 6,
             lr = 0.001,
             log_level = 2)

The fit also runs without any errors using max_epochs = 6 and mini_batch_size = 8.

So far, everything seems to point towards the possibility that the SAS Viya-environment I'm using is unable to handle things beyond a certain load, thereby resulting in the error (that's the feeling I'm getting at least).

Thank you,

Jian