System error in scoring phase with Gradient Boosting model

[albe]

Hi everybody, I'm quite stuck with the following issue with Enterprise Miner 13.2.

I've got two similar dataset (around 200.000 obs and 300 variables), one for training purpose and the other for testing (same variables obviously, just the target is missing in the test dataset).

Long story short, I built and compared 5 different models: I can use any of them for scoring, except the "best" one, a gradient boosting model. After a few minutes attempting to score just a tiny sample of the test dataset (100 observations) I get this message: "System error - Program halted in server workspace method".

I've got another simpler GB model (fewer iterations, smaller depth) that I can use successfully to score.

Any suggestion?

Thanks!

[M_Maldonado]

Hi Albe,

Below a quick thing to try. I have another couple of ideas depending how complex your diagram is.

If this is urgent, please also submit a Tech support form to confirm what's going on.

Troubleshoot

What happens if you add just a Score node to the "Best" Boosting model. Just Boosting->Score, do not connect your scoring data, does that run OK? If yes, you can use the optimized score code to give it a try to manually score your test data.

How Complex is your diagram? can you share a screenshot? If there are no transformations or new variables in that flow, then you might use the EMFlowscore that your gradient boosting node has already produced.

Start with these two ideas and let's figure out what's happening with that node of yours.

Thanks,

Miguel

[JasonXin]

Albe,

First, thanks  for using SAS. My name is Jason Xin, analytics solution architect focused on financial services.

I was helping another customer when I saw your post. I searched SAS.com's technical support and found this support Note that I think is relevant to your error message.

http://support.sas.com/kb/15/720.html

"Problem Note 15720: Enterprise Miner nodes fail when Server Initialization code contains a syntax error"

If you can share with us what your log says, that may be very helpful. Or have you tried the Optimized code? Thanks.

Best Regards

Jason Xin

[albe]

Miguel, Jason,

thanks for the prompt responses.

I tried just the boosting -> score without the scoring dataset, still the same error.

And the diagram is really easy: training data, partitioning, models, model compare and then score.

I have yet to try the EMFlowscore.

As for the problem note 15720, I checked View->Project Log for strings like "error", "not valid" and such, no luck.

Jason, you were asking for the Project Log or some other log?

By the way, as the simpler models run with no problem whatsoever, could it be something related to memory allocation or model "size"?

[M_Maldonado]

Hi Albe,

Since your diagram is simple and your "Best" gradient boosting node is running OK, let's try to use the score code manually.

First, identify the ID of your gradient boosting node (click on it, and on top of their properties find the Node ID e.g. Boost3). Do the same for your workspace, click anywhere blank on your diagram and find the ID (e.g. EMWS1).

Next, locate the physical path of your workspace. One way to do it: click on the explorer icon, select Show Project Data, and confirm the physical path.

Now go to base SAS and use code similar to the below. I mocked up my test data, but you can start on the section Score manually.

/*Create a test set. To mock it up, I dropped the target variable*/
data test;
set sampsio.hmeq;
drop bad;
run;

/*Score manually*/
%let path=D:\EM\EM_Projects\EM14.1\mim\demo\Workspaces\EMWS1;
%let nodeid=Boost3;

filename emflow "&path.\&nodeid.\EMPUBLISHSCORE.sas";

data test;
set test;
%inc emflow;
run;

Hopefully scoring manually gets you somewhere?

Also, if you have time, send me the log of the Score node that is failing. When you are on the results, go to View-SAS Results-> log. Do CTRL+F to try to find "error: ". You can use direct message if that is better, or through tech support.

HIH!

-Miguel

[albe]

Hi Miguel, I'll try to run the score manually soon. In the meantime, I checked the log of the score node.
The first time "error" appears is here:

1578353 filename _pref "C:\SasEnMin\dati\KHS2\Workspaces\EMWS1\Boost\EMPUBLISHSCORE.sas";
1578354 options validvarname=ANY;
1578355 proc FCMP NOEXECUTE IMPUTEDDEPS OUTDEPS=_temp;
1578356 %INCLUDE _pref;
ERROR: Out of memory.
NOTE: Line generated by the invoked macro "DMNORMIP".
1 call dmnorm(&in,%dmnorlen);
-
22
WARNING: Apparent symbolic reference IN not resolved.
ERROR: Out of memory.
NOTE: Line generated by the invoked macro "DMNORMIP".
1 call dmnorm(&in,%dmnorlen);
-
22
WARNING: Apparent symbolic reference IN not resolve

[JasonXin]

Albe, this DMNORMIP error suggests it fails to call up underlying Java function at scoring due to resource condition. Perhaps time to open a technical support ticket? They may advise you how to alter memsize... to make it through. thanks. Jason Xin

[albe]

Hi Jason, thanks for the hint!
I found in the sas.com knowledge base how to alter the memsize (thought it should be quite easy, even without opening a ticket): I found the option right away in sasv9.cfg file. It was 2GB, I edited it to 6GB, and now the scoring node is running fine (still running, but it used to stop just after a few minutes).
I took a look at the ram allocated: the process related to the scoring actually hit like 5,3GB alone!
Thank you again for your support.

[JasonXin]

If you set it to 0, it will give you dynamic allocation, so you don't have to check as you go. But proceed with caution. If others are sharing the resource with you, they may 'scream' at you. Or if they don't, your system admin may call you. Glad it is progressing forward. Thanks. Jason Xin