For programmers! Use SAS procedures, Python, Lua, REST APIs, and more!

Numpy array from CASTable gets truncated at 10k entries

Reply
New Contributor
Posts: 2

Numpy array from CASTable gets truncated at 10k entries

Hi everybody,

 

I cannot retrieve the entire values from a CASTable, as they get truncated after 10000 entries when trying to save them as apndas dataframe.

 

i am working with SAS Viya in Python and was trying to access values in the CASTable in order to save them in a pandas dataframe.

When i print infos concerning the table i get (correctly) 21401 rows in the table

 

CASTable(u'my_table', caslib=u'CASUSER(myusername)')
Data columns (total 6 columns):
                 N   Miss    Type
id           21401  False  double
age          21297   True  double
sex_id       21297   True  double
sire_id      21297   True  double
dam_id       21297   True  double
prize_money  21297   True  double

When i try to access the values and print the shape of the values array:

my_table.casTable.values.shape

(10000L, 6L)

So the array gets truncated after 10000 entries .

How can i address that, and get the full numpy array?

 

 

Thanks

 

Luca

Valued Guide
Posts: 505

Re: Numpy array from CASTable gets truncated at 10k entries

I know nothing about Viya, but if the CASTable can be converted to a SAS dataset both python and R can read it
library(haven) in R
import SAS&BDAT with sas7bdat in Python?
New Contributor
Posts: 2

Re: Numpy array from CASTable gets truncated at 10k entries

Thanks, in the end i managed to solve this problem by exporting the CASTable to SASFrame and then converting the SASFrame to Pandas Dataframe

 

import pandas as pd

my_dataframe =  pd.DataFrame(my_table.to_frame())

Ask a Question
Discussion stats
  • 2 replies
  • 97 views
  • 1 like
  • 2 in conversation