topic Numpy array from CASTable gets truncated at 10k entries in Developers

Numpy array from CASTable gets truncated at 10k entries

lcarcano — Mon, 03 Apr 2017 13:43:24 GMT

Hi everybody,

I cannot retrieve the entire values from a CASTable, as they get truncated after 10000 entries when trying to save them as apndas dataframe.

i am working with SAS Viya in Python and was trying to access values in the CASTable in order to save them in a pandas dataframe.

When i print infos concerning the table i get (correctly) 21401 rows in the table

CASTable(u'my_table', caslib=u'CASUSER(myusername)')
Data columns (total 6 columns):
                 N   Miss    Type
id           21401  False  double
age          21297   True  double
sex_id       21297   True  double
sire_id      21297   True  double
dam_id       21297   True  double
prize_money  21297   True  double

When i try to access the values and print the shape of the values array:

my_table.casTable.values.shape

(10000L, 6L)

So the array gets truncated after 10000 entries .

How can i address that, and get the full numpy array?

Thanks

Luca

Re: Numpy array from CASTable gets truncated at 10k entries

rogerjdeangelis — Mon, 03 Apr 2017 14:50:14 GMT

I know nothing about Viya, but if the CASTable can be converted to a SAS dataset both python and R can read it
library(haven) in R
import SAS&BDAT with sas7bdat in Python?

Re: Numpy array from CASTable gets truncated at 10k entries

lcarcano — Tue, 04 Apr 2017 07:45:07 GMT

Thanks, in the end i managed to solve this problem by exporting the CASTable to SASFrame and then converting the SASFrame to Pandas Dataframe

import pandas as pd

my_dataframe = pd.DataFrame(my_table.to_frame())

Re: Numpy array from CASTable gets truncated at 10k entries

KrunalLathiya — Sat, 22 Aug 2020 18:03:11 GMT

To convert the Pandas object to DataFrame, use to_frame() method. The to_frame() method is used to convert a Series object into a DataFrame. Series is a one-dimensional array with axis labels, which is also defined under the Pandas library.

You can export the CASTable to SASFrame and then convert the SASFrame to Pandas Dataframe. Pandas Dataframe.to_numpy() is an inbuilt method that is used to convert a DataFrame to a numpy array.