BookmarkSubscribeRSS Feed
☑ This topic is solved. Need further help from the community? Please sign in and ask a new question.
HuseyinKaya
Fluorite | Level 6

I have a dataframe contains 3399 rows and 16 columns running in a Python program. I want to create a table from it on SAS library. 

I used 

SAS.df2sd(df,"LIBRARY.Table_Name")

After i create the table there no rows in the table. 

HuseyinKaya_0-1703510834876.png

Then i used the following code for understanding the problem 

SAS.df2sd(df.iloc[:1000],"LIBRARY.Table_Name")

HuseyinKaya_1-1703511001183.png

Now i can see all the 1000 rows and 16 columns but i want to create all the rows.

There is no space problem in the server, I cannot identify the problem, can somebody help me?

1 ACCEPTED SOLUTION

Accepted Solutions
Kurt_Bremser
Super User

Character variables in SAS datasets are technically limited to 32767 bytes. I guess this limit also kicks in when processing rows from the dataframe.

How is that dataframe defined?

Please post the complete log of the step(s) creating the dataframe, and the complete log of the failing step by copy/pasting the log text into a window opened with this button:

Bildschirmfoto 2020-04-07 um 08.32.59.jpg

View solution in original post

10 REPLIES 10
Kerim1
Calcite | Level 5
I have same problem, please help me.
IbrahimTurkoglu
Calcite | Level 5

I would like to ask for your help on this issue. It does not convert the existing dataframe into a SAS table, regardless of the number of columns and records, for no reason. Sometimes I experience this situation, sometimes I don't.

Patrick
Opal | Level 21

@HuseyinKaya , @IbrahimTurkoglu , @Kerim1 

What really would help us help you is if you could provide some as simple as possible sample code that allows to replicate the issue (i.e. load sashelp.class into the data frame). 

Also please state what version of saspy and Viya you are using.

HuseyinKaya
Fluorite | Level 6

Actually, the code is so basic, I prepare a python df with 3k+ columns and try to create a sas table and insert it with df2sd function and get no error, only it says "Socket connection failed during data transfer", column's name lenth less than 22, you may try with dummy data, thanks for reply.

HuseyinKaya
Fluorite | Level 6
By the way i don't use saspy and viya = 2023.3
Patrick
Opal | Level 21

@HuseyinKaya wrote:
By the way i don't use saspy and viya = 2023.3

Method df2sd comes with the saspy package so not sure what you're talking about. 

https://sassoftware.github.io/saspy/api.html 

 

Without seeing your code and ideally log there isn't much more I can say.

sastpw
SAS Employee

I have to believe the user is referring to using Proc Python in Viya. The proc also has similarly named methods for moving data between pandas and SAS, though the method signatures are different. 

Have you looked at the log after the failure to see what errors or other information might be available?

 

SAS.df2sd(df, "LIBRARY.Table_Name")

SAS.printLOG()

 

Do you know what's in your dataframe? Realize that dataframes can contain things SAS datasets cannot. Each cell of a dataframe can contain anything. They aren't bound by 'relation table' constraints like SAS data sets or database tables. Relational tables require that the data type in every row of a table be the same for a given column. data frames don't require that. There are all kinds of ways a dataframe can't be converted into a SAS data set. But that's speculation at this point as I have no idea what's in your dataframe of what errors might be in log stating what went wrong.

 

So, let's start by looking at the log for the method you ran that didn't seem to work right. BTW, you can change the setting so the log is simply displayed to begin with:

 

SAS.hideLOG(False)

SAS.df2sd(df, "LIBRARY.Table_Name")

 

It's also possible that there could be an enhancement or bug fix that's happened since the version of viya you're using. it's almost a year old. So, again, seeing what actually happened will help with that. Looking at the LOG is the first thing to do.

 

Thanks,

Tom

HuseyinKaya
Fluorite | Level 6

Hi, thanks for the help. 

I used "SAS.hideLOG(False)" code and now i can identify what is the problem. 

HuseyinKaya_0-1704711070328.png

My data takes up too much space.(One row exceeds  32768 Bytes) 

 

How can i resolve this issue, is there any options for increasing 32768 Bytes?

Kurt_Bremser
Super User

Character variables in SAS datasets are technically limited to 32767 bytes. I guess this limit also kicks in when processing rows from the dataframe.

How is that dataframe defined?

Please post the complete log of the step(s) creating the dataframe, and the complete log of the failing step by copy/pasting the log text into a window opened with this button:

Bildschirmfoto 2020-04-07 um 08.32.59.jpg

Patrick
Opal | Level 21

With .sas7bdat files character variables are limited to 32KB.

Depending on what you need the data in SAS for you could eventually investigate if loading the data directly into CAS would be an option using VARCHAR(*). 

I have never done it but the SWAT package looks promising. 

https://communities.sas.com/t5/SAS-Communities-Library/Better-Open-source-Integration-through-the-Py... 

https://sassoftware.github.io/python-swat/ 

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

How to choose a machine learning algorithm

Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 10 replies
  • 2152 views
  • 6 likes
  • 6 in conversation