Hello,    I don't have Viya. I do have Jupyter notebook.  Just run Pd. read_parquet to read in your parquest file. Write out to CSV and then use SAS proc import. to read it back in. I run batch to  process many GB size. CSV is your best bridge between SAS and open source data files.  It has a distinctive speed edge over SAS Hadoop Access. I once tested Viya and felt it still has too many layers in the background like 'old SAS'.  It may be optimizing something, but for 195 MB? you don't need that elaboration. You just need straight shot, simple and fast.    Jia ... View more

Hello @iuri_leite ,   I initially thought you were talking about Random Survival Forests scikit  recently released. That was purely technical variation.    The claim paper you provided shows  typically how survival analysis has been done. As you see, there isn't  any special invention on the HPFOREST procedure part to extend the model from logistic approaches. As in all survival analysis, heavy duty is in data preparation. If you have analysis data ready for logistic regression, it is ready for HPfOREST.    Jia ... View more

The mere fact the event=200 requires treatment. So the original question may not be imbalance. No, event =200 is not going to give you any strong, sustainable model regardless.    Manually running weight is essentially the same as oversampling/oversampling. One goes by count, sampling goes by ratio. Manually weighting is not recommended because such action almost guarantees inability to sum the resampled total weight properly.  At using the Weight statement of proc logistic, pay some attention to the norm option on the statement. You will see the impact of using vs not using that option on your model   Last, but not the least, I doubt this subject has much to do with objective function.  ... View more

In EM, there is setting where you can find the default score cut above which your prediction is flagged as yes. Otherwise NO. I think that default is 0.5. Play with different cut-off values to see you can just manage away the problem. This is perfectly legal. Nobody says you have to accept 0.5 as cut-off. This is low-hanging fruit kind of approach.    A bit harder route is when you exhaust your play above and the symptom remains. To find out, take the validation data set from EM. Now the prediction 1 and 0 based on your 'optimized' score cut above is the new prediction target. Run another model, preferably intuitive decision tree which allows you see the drivers in the background. You cannot do this with your original target because that means you simply are adding a separate stand alone model alongside your ensemble.    Jia Xin ... View more

You may use, reuse the 200 event=1. Randomly sample from the non-events, say, 1000 (any count that constitutes normal percentage with the 200 events). Say you have 10 or 5 such samples. Build regular LR models. Then assemble /average the predicted scores. This is often called K-fold method. Motivation sometimes is not because the ratio of 1 and 0, but because the scanty count of 1. This can be a bit hacking, especially if you sample a lot. And it does not help with driver explanations    If possible, run a HPSPLIT or EM decision trees. Sometimes non event count overpowers due to some unbalanced exclusion, the root-cause method, a much more business-friendly method.   Oversampling: with logistic regression,  very OK to try it.    Jia Xin     ... View more

Hello,  try this,  https://blogs.sas.com/content/iml/2018/11/14/compare-roc-curves-sas.html   This idea is to pretend your RF score is the only predictor of the target. You can put on any ML score you like.    Jia Xin   ... View more

Thanks. Better than mine. ... View more

Let me further simply.  Go by _n_ order (forget about that Month field). When _n_ =1 to 3, first sum of dollar field. when _n_=4 to 6, second sum.  how to order _n_ using month or other field is not the subject here. There are 27 rows originally. So look for 9 sums.    I can use 'clumsy ways' to get it done. like to hear some elegant idea. Thank you.  ... View more

I look up documentation. Not seeing where there is list of procedures that are exception as such. Does anyone know where to find it? Thank you ... View more

Hello, If the HPFOREST node does not turn RED, that indicates there is no mechanic problem with the flow. Agree 10 lines for RF is just not right. First could be setting driven, for example, one of the stopping rules is too stopping too early. Not hearing if you have modified setting on the HPFOREST. If you are running on default setting, then your data is not sufficient to drive the model to begin with. ... View more

This is a bit beyond programming. Hear me out please.    After moving to SAS code, with all the inter-tab partitions disappeared, gliding over  a long piece of sas code (if I deck all the code up front on the driving piece, we are looking 4000 lines +) is like navigating a long snow slope in Salt Lake City valley. The distinction is obvious. With Excel, no matter how analytical we say we are, jamming at eyes and minds are data, cell, bits (with all due respect). But gliding at, and staring at SAS code, first, and foremost, the focus has to be analytical, logical first. In my case, obviously I have to place sections of datalines to source in Excel inputs, but in doing that I barely pay attention to data points (unless missing delimiters...) + now If I shrink back from SAS to Excel, I end up cutting out ~1/3 tabs. Many cross-references embedded in Excel are just so cliched redundancy because  mind is default-full with cells, cells, data points... not on cogent logic streams. So you like your Excel, by all means keep it. Just that if you have to move it to SAS, there could be intrinsic analytic benefits that you may actually like.  ... View more

Update:  With the code corrected, therefore controlled, managed to isolate why that bad code lasted for 2 days. Turned out the installation had some "flu-like symptoms. It was actually randomly "doing /not resolving" macros. We applied a patch Saturday. Now the bad code never took off without error me out. Without the code being correct, hard to realize the system side. So thank you all. ... View more

Thank you.    First, it is business decision to go ahead using SAS to replace Excel, one by one, tab by tab. Your opinion or my opinion does not count. I actually believe it is great direction to go at this moment. The numbers match with Excel outcome, with several hundred SAS macro variables. It has cost too much time/effort, some embarrassment on me albeit.     I concur that back in, say, 2002, perhaps using SAS to replace Excel was generally wrong/bad idea. This is 2022. SAS is much better, more open to work TO/FROM other packages. I used to use DDE, for example, to work with SAS, <Y2K. Now this sounds such a long bygone.    Second, I wish I have the liberty to disclose the log and HTML  details that show my now notorious original post worked, for a while of 2 days. Not to start debate, but your NEVER statement does have exceptions, practically. As wisdom goes, never say NEVER.    A bit more background, as I am allowed to: we run long SAS prep code (long-running jobs) that pulls, cleans, organizes the data fed into the spreadsheets. Years ago we  decided to continue the subsequent analysis using Excel not SAS, half a dozen reasons I recall. Today is time of WFH. Excel is not working as it used to in this new, remote setting. Clipboard, for example, is constantly under pressure to work properly,  thanks partially to corporate security.... Many linked formula sometimes automatically update, sometimes this-some automatically that-some don't,, vice versa. We spend a lot manpower to check, double-check, just to avoid misjudging people's quality of work, attitude....   So we return to the glory of SAS. If the users are all SAS savvy, I could live with my rough version one as is. To accommodate not-that-savvy users, I need to polish up quick a bit. One big  benefit as I see at this moment is SAS output is so much closer to Excel-readable, especially if using SAS Studio.      ... View more