Input string with an unknown variable length

[Mikeyjh]

Hi, Hopefully a simple question.

I need to input a string variable from a csv file that can be of any length from 0 to 32K characters. The maximum length is unkown at the time of import and I would like to set the variable size to the actual maximum length in the csv file. So I think I need to use $varyingw but looking at examples it seems like the length needs to be defined in the input file or is set to the record length. What is the best way of inputting this field?

Thanks for your help.

[Tom]

If you are reading it from a CSV file you do not need to do anything other than define the variable long enough.

data want ;

  infile 'myfile.csv' dsd lrecl=2000000 ;

  length id 8 string $32767 ;

  input id string;

run;

[Mikeyjh]

Thanks Tom. So is there any disc space issues in setting the length so big? Also, if the maximum length was 259 characters (for example, but I dont know this beforehand) is there is no way I can implement the input statement to default to 259 when executed?

[Tom]

If you know that the maximum possible length then define the variable using that length.  So in my example change the 32767 (which is maximum possible variable length) to 259.  You can also change the LRECL on the INFILE statement, but I doubt that would have much impact on performance.

SAS stores all character variables as fixed length.  But if you use the COMPRESS dataset option then the extra empty space does not take much space on the disk.

[Tal]

is 2000000 the max lrecl,Tom?

[Tom]

I am sure it depends on the operating system and SAS version. On SAS 9.2 running on 64 bit unix I can use LRECL up to

1,073,741,823 (which 2**31 -1) without getting a syntax error.

[Tal]

thanks Tom

[Dmitry · Accepted solution]

In such cases I sometimes read file with maximum length (32767, e.g.) and then decrease the length of the variable down to the maximum length of values.

proc sql noprint;

     select max(length(variable)) into :maxlen

     from mydata;

quit;

options varlenchk=nowarn;

data mydata;

     length variable $&maxlen;

     set mydata;

run;

options varlenchk=warn;

VARLENCHK-option is necessary if you want to avoid warning from SAS that you may truncate some values.