alisio_meneses Tracker

Re: Using SAS PROC S3 to access a bucket using CEPH

alisio_meneses — Tue, 30 Apr 2024 12:44:35 GMT

I managed to resolve the issue by incorporating a custom region using a different approach. Since there was no response, I'll share my solution for anyone who might benefit from it.

Here's what I did to make it work:

Set a unique name for the custom region. Previously, I had assigned a custom region name identical to an existing AWS region, so I opted for a distinct name instead.
Utilized PROC S3 to introduce the custom region instead of relying on environment variables.

/*Add custom region*/

proc s3;
REGION ADD HOST="storage.middle.earth" NAME="shire";
run;
/* Listing all regions (including the custom region) */
proc s3;
region list;
run;

This resulted in the following log output (note the added custom region):

1 %studio_hide_wrapper;
83 proc s3;
84 REGION ADD HOST="storage.middle.earth" NAME="shire";
85 run;
NOTE: PROCEDURE S3 used (Total process time):
real time 0.01 seconds
cpu time 0.00 seconds

86
87 proc s3;
88 region list;
89 run;
Amazon Regions
us-east-1 s3.amazonaws.com Default 0
us-east-2 s3-us-east-2.amazonaws.com Default 0
us-west-2 s3-us-west-2.amazonaws.com Default 0
us-west-1 s3-us-west-1.amazonaws.com Default 0
eu-west-1 s3-eu-west-1.amazonaws.com Default 0
eu-central-1 s3-eu-central-1.amazonaws.com Default 0
ap-southeast-1 s3-ap-southeast-1.amazonaws.com Default 0
ap-southeast-2 s3-ap-southeast-2.amazonaws.com Default 0
ap-northeast-1 s3-ap-northeast-1.amazonaws.com Default 0
sa-east-1 s3-sa-east-1.amazonaws.com Default 0
us-gov-west-1 s3-us-gov-west-1.amazonaws.com Default 0
us-gov-west-1 s3-fips-us-gov-west-1.amazonaws.com Default 0
ca-central-1 s3-ca-central-1.amazonaws.com Default 0
ap-south-1 s3-ap-south-1.amazonaws.com Default 0
ap-northeast-2 s3-ap-northeast-2.amazonaws.com Default 0
cn-north-1 s3-cn-north-1.amazonaws.com Default 0
cn-northwest-1 s3-cn-northwest-1.amazonaws.com Default 0
eu-west-2 s3-eu-west-2.amazonaws.com Default 0
eu-west-3 s3-eu-west-3.amazonaws.com Default 0
ap-east-1 s3.ap-east-1.amazonaws.com Default 0
eu-south-1 s3.eu-south-1.amazonaws.com Default 0
eu-north-1 s3.eu-north-1.amazonaws.com Default 0
me-south-1 s3.me-south-1.amazonaws.com Default 0
af-south-1 s3.af-south-1.amazonaws.com Default 0
ap-northeast-3 s3.ap-northeast-3.amazonaws.com Default 0
Custom Regions
shire storage.middle.earth Default 0
NOTE: PROCEDURE S3 used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds

90
91 %studio_hide_wrapper;
102
103

Now, with the appropriate credentials, it's possible to test bi listing a bucket content:

proc s3 KEYID="isItSecret" SECRET="isItSafe" region="shire";
list "/bucket" ;
run;

That's it.

See ya!

Using SAS PROC S3 to access a bucket using CEPH

alisio_meneses — Thu, 25 Apr 2024 23:44:13 GMT

Hello There!

I'm having trouble connecting to a CEPH object storage service using PROC S3 in SAS Viya 3.5 on Linux. CEPH is an open-source distributed storage system capable of providing S3 compatible buckets. I've tried setting a custom region using the TKS3_CUSTOM_REGION environment variable and a config file with access credentials, but I keep getting an error saying "The AWS access key Id you provided does not exist in our records." Here's my code:
options set=TKS3_CUSTOM_REGION="us-east-1,storage.yada.yada,0,0,TRUE,TRUE";
%let myvar_value = %sysget(TKS3_CUSTOM_REGION);
%put &myvar_value;

PROC S3 
	config="/home/user/tks3.cfg";
	LIST "/";
RUN;
The S3 "/home/user/tks3.cfg" file contains the following:
region=us-east-1
keyID = thisIsAnAwesomeKeyID
secret = whoaThisSecretIsSafe
Output Log:
1    %studio_hide_wrapper;
83   options set=TKS3_CUSTOM_REGION="us-east-1,storage.yada.yada,0,0,TRUE,TRUE";
84   %let myvar_value = %sysget(TKS3_CUSTOM_REGION);
85   %put &myvar_value;
us-east-1,storage.yada.yada,0,0,TRUE,TRUE
86   
87   PROC S3
88   config="/home/user/tks3.cfg";
91   LIST "/";
92   RUN;
ERROR: Could not get user buckets.
ERROR: The AWS access key Id you provided does not exist in our records.
ERROR: The AWS access key Id you provided does not exist in our records.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE S3 used (Total process time):
      real time           0.58 seconds
      cpu time            0.02 seconds
      
93   
94   %studio_hide_wrapper;
105  
106  
Currently I am able to interact with data from the CEPH bucket using the command-line tool for managing Amazon S3 storage, s3cmd, the same way I use it to interact with AWS S3 buckets. I guess this rules out problems caused by the CEPH service.

Can someone point what might be wrong?

Thank you.

Environment info:

SAS Viya 3.5 running on Linux

Re: Saving an In-Memory CAS Table as a Single Parquet File

alisio_meneses — Mon, 08 Apr 2024 11:46:32 GMT

hello, thanks for the reply. Using save generates a single folder named <tablename.parquet> with multiple parquet files inside. I guess thats for optimization purposes. Not sure.

Saving an In-Memory CAS Table as a Single Parquet File

alisio_meneses — Fri, 05 Apr 2024 17:44:02 GMT

Hi there,

I see 'PROC CASUTIL SAVE' can write an in memory CAS table to multiple parquet files stored inside a CASLIB folder. I wonder if it is possible to do it into single file parquet file.

Is it? if so, how?

Env. Info: SAS Viya 3.5 running on linux with multiple cas workers.

Thank you!

Re: How to perform key Deduplication of CAS Tables using Hash Tables.

alisio_meneses — Fri, 05 Apr 2024 17:15:15 GMT

Hi there @Mazi, thank you for responding.

I tried using proc cas and and also proc sort (nodupkey) with similar results. they both worked fine. thank you!

Re: How to perform key Deduplication of CAS Tables using Hash Tables.

alisio_meneses — Wed, 03 Apr 2024 18:06:19 GMT

Hi! Thanks for replying. I think you may be right, and it might work. But if it should work, doesn't it limit the usage of CAS resources (like massive parallel processing), thus making it possibly slower than using PROC SORT. I'll have a go anyway.

How to perform key Deduplication of CAS Tables using Hash Tables.

alisio_meneses — Wed, 03 Apr 2024 17:32:45 GMT

Hello there.

I would like to know how to perform key deduplication of a CAS table more efficiently than using proc sort. According to this document it is possible to perform this task using hash tables, but the doc uses non-CAS tables.

Heres a sample code using non CAS tables:

data testdata(drop=i j);
   array vars var1-var5;
   do i=1 to 10e6;
      do j=1 to dim(vars);
         vars[j]=rand('integer', 1, 10);
      end;
      output;
   end;
run;

proc sort data=testdata out=test1 nodupkey;
   by _ALL_;
run;

data test2;
   if _N_ = 1 then do;
      declare hash h(hashexp:20);
      h.defineKey('var1', 'var2', 'var3', 'var4', 'var5');
      h.defineDone();
   end;
 
   set testdata;
 
   if h.check() ne 0 then do;
      output;
      h.add();
   end;
run;

When trying to use the same code with CAS, I noticed that the resulting table still contains duplicate records, and I assume this is related to parallel execution on different workers, but I'm not sure.

I would appreciate the community's help: is it possible to perform record deduplication of a CAS table using hash tables? If so, how?

Environment information: SAS Viya 3.5 with 4 CAS workers running on linux

Thank you!

Re: Error when using CALL EXECUTE with a CAS Table

alisio_meneses — Sun, 24 Dec 2023 21:28:59 GMT

Oh, I see now.

Guess I'll have to find an alternative approach. One of the uses CAS tables.

Thanks for pointing that out.

Error when using CALL EXECUTE with a CAS Table

alisio_meneses — Sun, 24 Dec 2023 16:10:11 GMT

Hello there,

I am currently facing an issue with a SAS script involving the use of CAS tables, and I would greatly appreciate your insights to help resolve the problem.

The script creates a CAS session, defines a CAS library, creates a CAS table named "table" with a variable "text," defines a macro "test" that prints its parameter, and then applies the macro to each observation in the "table" CAS table using the call execute routine.

Here is a simplified version of a script that executes a simple macreo:

cas mysess;
libname mycas cas;

data mycas.table;
    length text $10;
    input text $;
    datalines;
This 
is
a 
sample
table
;
run;

%macro test(param);
    %put param=&param;
%mend test;

data mycas.test/sessref=mysess;
    set mycas.table;
    call execute ('%test(' || text || ')');
run;

When attempting to use call execute on a CAS table, I encounter the following error:

ERROR: The subroutine EXECUTE is unknown, or cannot be accessed. Check your spelling. Either it was not found in the path(s) of 
       executable images, or there was incorrect or missing subroutine descriptor information.
ERROR: The action stopped due to errors.

Interestingly, when the sessref=mysess option is removed from the data statement, the code executes without errors:

data mycas.test;
    set mycas.table;
    call execute ('%test(' || text || ')');
run;

It seems that there might be an issue related to the sessref option when using call execute with CAS tables. as the %test macro is defined properly and is accessible when sessref option is not in use.This option is being used to ensure that all processing is done in CAS to prevent unwanted data transfer.

I would appreciate any guidance or suggestions on how to resolve this issue. thank you in advance!

Re: Counting Dictionary Matches in Text Data

alisio_meneses — Fri, 15 Dec 2023 11:47:08 GMT

I've managed to accomplish the task using PROC TEXTMINE. Apart from completing the specified task, it enables the use of other features such as stemming, entity detection, and so on.

The following code does the trick and, aside from the sample tables code, it runs in CAS.

Thank you all for helping!

Sample tables:

/* dictionary words*/
DATA mycas.dictionary;
   length word varchar(*);
   input word;
   cards;
   apple
   banana
   orange
   grape
   ;
run;

/* text and associated IDs */
data mycas.TEXT_DATASET;
   infile datalines delimiter='|' missover;
   length text varchar(*);
   input text$ id;
   datalines;
   This is an apple and a banana|1
   I love eating an oranges for lunch|2
   Grapes are delicious, especially red grapes|3
   I have an apple and an orange for lunch|4
   ;
run;

And the word count script.

/*************************************************************
* Count dictionary matches in text
* Author: Antonio Alisio de Meneses Cordeiro
* Date: 10/12/2023
* Description: This SAS program performs text mining on a dataset
*              containing textual information. It includes the
*              creation of a dictionary, parsing of text data,
*              and counting the frequency of dictionary words
*              in each document.
*************************************************************/

/* Connect to CAS session */
cas mysession;

/* Assign a library reference to the CAS session */
libname mycas cas sessref=mysession;

/* Perform text mining on the 'mycas.TEXT_DATASET' table */
proc textmine data=mycas.TEXT_DATASET;
   doc_id id;
   var text;
   parse
      notagging nonoungroups
      termwgt        = none
      cellwgt        = none
      reducef        = 1
      entities       = none
      outpos          = mycas.outpos
      ;
run;

/* Create a new table store parsed text information */
data mycas.outpos/sessref=mysession;
   set mycas.outpos;
   format Term;
   format Parent;
   format DOCUMENT;
run;

/* Filter out only the words in the dictionary from the parsed text */
data mycas.outpos_filtered;
   merge mycas.dictionary(IN=A)
         mycas.outpos(IN=B rename=(Parent=word));
   by word;
   if B AND A;
run;

/* Count word frequency by document in the filtered output */
data mycas.WORD_BY_DOC_COUNT/sessref=mysession;
   set mycas.outpos_filtered;
   by DOCUMENT;
   if first.DOCUMENT then dict_word_count = 0;
   dict_word_count + 1;
   if last.DOCUMENT then output;
   keep document dict_word_count;
   rename document = id;
run;

/* Merge word frequency information back to the original text dataset */
data mycas.TEXT_DATASET_WITH_COUNT/sessref=mysession;
   merge mycas.TEXT_DATASET(IN=A)
         mycas.WORD_BY_DOC_COUNT(IN=B);	
   by id;
   if A;
   if dict_word_count = . then dict_word_count = 0;
   keep text dict_word_count;
run;

Re: TF-IDF with SAS Viya

alisio_meneses — Fri, 08 Dec 2023 17:26:08 GMT

Hello there, @BigRider

Thank you for sharing your code.

As I reviewed the SAS code provided, I couldn't locate the specific step where TF and IDF values are multiplied to calculate the TF-IDF scores. If you could kindly point me in the right direction or provide some tips on this, I'd greatly appreciate it.

Thanks in advance.

Re: Counting Dictionary Matches in Text Data

alisio_meneses — Sun, 26 Nov 2023 12:23:16 GMT

Assuming 'pineapple' is in the table 'dictionary', I would expect a count of 1

Re: Error when creating a Visual Data Mining and ML Project using Model Studio on SAS Viya 3.5

alisio_meneses — Fri, 24 Nov 2023 17:23:28 GMT

Hello there, thanks for your reply.

I could not find the advisor logs as instructed. Thank you anyway. I've been told by the admin there were permission problems regarding my account.

Cheers!

Re: Counting Dictionary Matches in Text Data

alisio_meneses — Mon, 20 Nov 2023 20:38:33 GMT

Hi @s_lassen , thank you very much.

Your sample code performs as expected and functions well with a temporary ('work') library. However, I encounter an error (log attached) when attempting to change your code for use with CAS tables using "Public" Caslib, specifically the 'dictionary' table.

ERROR: The POINT= data set option is not valid for the data set PUBLIC.DICTIONARY, the data set must be randomly not sequentially
accessible for POINT= processing.

Here the code slightly altered using CAS Tables:

DATA public.dictionary;
input word $;
cards;
apple
banana
orange
grape
;run;

data public.TEXT_DATASET;
  infile cards truncover;
  input text $char200.;
cards;
This is an apple and a banana
I love eating oranges.
Grapes are delicious, especially red grapes.
I have an apple and an orange for lunch.
;run;

data public.want;
  set public.text_dataset;
  if _N_=1 then do _N_=1 to words;
    array prxids (1:1000) 8 _temporary_;
    set public.dictionary point=_N_ nobs=words;
    prxids(_N_)=prxparse(cats('/\b',word,'s?\b/i'));
    end;
  dict_word_count=0;
  do _N_=1 to words;
    dict_word_count+prxmatch(prxids(_N_),text)>0;
    end;
  drop word;
run;

Is there any additional step required for use in with CAS?

PS. : thanks for the tip regarding sample data as datastep code.

Counting Dictionary Matches in Text Data

alisio_meneses — Mon, 20 Nov 2023 15:24:22 GMT

Hello there!

I am struggling with a simple task: I have two tables. The first one is a word dictionary named ''DICTIONARY", containing a single field named "word." The second table is called TEXT_DATASET, which includes a single field named 'text,' where each row consists of text. Using SAS, how to add a field to the TEXT_DATASET

table, named 'dict_word_count,' which will store the count of words from the DICTIONARY table that exist in each 'text' field of the TEXT_DATASET table?

The following tables are samples:

Table DICTIONARY:

word

apple

banana

orange

grape

Table TEXT_DATASET:

text

This is an apple and a banana

I love eating oranges.

Grapes are delicious, especially red grapes.

I have an apple and an orange for lunch.

Resulting table TEXT_DATASET_WITH_COUNT:

text	dict_word_count
This is an apple and a banana	2
I love eating oranges.	1
Grapes are delicious, especially red grapes.	1
I have an apple and an orange for lunch.	2

Any ideas?

Thank you!

Error when creating a Visual Data Mining and ML Project using Model Studio on SAS Viya 3.5

alisio_meneses — Mon, 20 Nov 2023 13:19:32 GMT

Hello there!

I am getting the following error when creating a Visual Data Mining and Machine Learning Project Project using Model Studio on SAS Viya 3.5. The project's Data Table contains 22 fields of double type.

Error: The project could not be initialized.

Details: The project could not be opened. Check the advisor logs for details.

The error message does not describe much, but I was hoping someone might know what to do.

Screenshot:

Thank you!

SAS cannot find trained model parameters file for Fine-tunning a BERT Model

alisio_meneses — Mon, 18 Sep 2023 13:01:50 GMT

Hello there,

I'm attempting to utilize BERT and DLPy for sentiment analysis based on the tutorial outlined in the paper titled "SAS4429-2020 NLP with BERT: Sentiment Analysis Using SAS® Deep Learning and DLPy". I'm currently facing a challenge at step 4 (ATTACH MODEL PARAMETERS), which involves linking the trained model parameters from the HDF5 file to the BERT model. The Deep Learning actions must have access to the HDF5 file. However, since the client computer is typically separate from the server, the following Python code snippet establishes a "server_dir" to which the generated parameters must be copied for accessibility by the Viya server:

import os
from shutil import copyfile

server_dir = 'path/to/your/server-directory'

copyfile(os.path.join(cache_dir, 'bert-base-uncased.kerasmodel.h5'),
         os.path.join(server_dir, 'bert-base-uncased.kerasmodel.h5'))
.......

However, there's a catch: I’m using a SAS Viya 3.5 environment with a distributed server architecture, and the Python runtime lacks a direct connection to the file server, so the ‘copyfile’ function is a no go .

While I can manually copy the file from the client to the server and load it from there, it raises the question o of "where to place the HDF5 file within the distributed server architecture (cas controller, cas workers, SPRE)?".

thank you!

For more details, you can refer to the paper "SAS4429-2020 NLP with BERT: Sentiment Analysis Using SAS® Deep Learning and DLP" here.

Thank you.

Re: SAS Viya Trial - Downloading tables

alisio_meneses — Tue, 08 Aug 2023 13:02:43 GMT

Hi @SAS_Cares

I am using a licensed SAS Viya 3.5, but did not manage to find out how to export CAS tables in .sashdat (only csv)

Would you care to share how to do it?

Re: SAS Viya - Download .sashdat

alisio_meneses — Tue, 08 Aug 2023 12:58:45 GMT

Hi there, @mdcervantes. Did you manage to download .sashdat from SAS Viya? If so, how dit you do it?

Re: How to convert double to int32 and varchar to varbinary

alisio_meneses — Mon, 07 Aug 2023 17:24:15 GMT

Hello @Tom.

Regarding the VARBINARY field: the original table, the one exported to CSV, was generated using the 'save' Action from the 'Tables' Action Set. It contains attributes from another table and includes a field with this data type (along with another field of type INT32).

The issue I'm facing is that I'm attempting to transfer data between different SAS Viya environments while trying to maintain the same table attributes.

proc cas;
   ldaTopic.ldaScore /
      docDistOut={caslib='acaslib', name="documents",
                  replace=TRUE
                 }
      docId="documents_id"
      modelTable={caslib="Public", name="lda_topics"}
      table={caslib="acaslib", name="input_prepared"}
      text={{name="text_prepared"}};
   run;
quit;