Part of the variable names changed when I tried to load a txt file

[ballardw]

Yes there is a limit on variable name length. However that is not the issue with your data. It is a combination limitationa of proc import and your actual data content.

[data_null__ · Accepted solution]

This is the program I used to read the data you posted here:

http://dropbox.unl.edu/uploads/20141112/2cb2cc0a5d5e994b/18Fratio.txt

I just ignored the field named "name" it's value is 18F.  I had to increase memory with SAS invovation option "-memsize '12g'  I don't know if that is too much but it worked, my system is UNIX.  I ran the program in batch.  You have to be careful that you don't make SAS "dump" the input record to the LOG that's what all those ?? marks are about.  If SAS dumps those giant records the log is unusable.  I don't know if this is the format you want but it is the only one that makes sense to me.

data names;
   infile FT33F001 dsd dlm='09'x obs=1;
   length name $32;
   input name @@;
   if first(name) eq 'R' then do;
      /*R00000001_D0002/D0001*/
      i = input(scan(name,1,'R_'),??8.);
      r = input(scan(name,-2,'D/'),??8.);
      c = input(scan(name,-1,'D'),??8.);
      end;
   run;

==============

NOTE: 1 record was read from the infile FT33F001.
      The minimum record length was 242565850.
      The maximum record length was 242565850.
NOTE: SAS went to a new line when INPUT statement reached past the end of a line.
NOTE: The data set WORK.NAMES has 11025722 observations and 4 variables.

data _18Fratio;
   infile FT33F001 dsd dlm='09'x firstobs=2;
   set names;
   input x :?? @@;
   format x 10.6;
   run;

===============

NOTE: 1 record was read from the infile FT33F001.
      The minimum record length was 99347717.
      The maximum record length was 99347717.
NOTE: There were 11025722 observations read from the data set WORK.NAMES.
NOTE: The data set WORK._18FRATIO has 11025722 observations and 5 variables.

[call_me_elaine]

I just tried a similar code using my example file. Do we need to transpose the final dataset since right now it spans in rows not columns. And I try to add more observations, but the code can only read the first observation. I don't know why.

[data_null__]

If you are referring to the first program I posted Yes it does need to transpose data NAMES to create a data set that has variables created from the values of name in the names data set.  This was a bit of a trick I used to create a data instead of generating a long list of name in macro variables.  I don't know if it will work for the "big" data.  I will try it and see and let you know.

[call_me_elaine]

I tried your newest code and it did work!!! Thank you so much. You mentioned you increased memsize opition in SAS. I am not sure how to do that so I just keep it by default. But it did work.

The reason I want to transpose this tall format to the wide one is that looks the same with my original txt file. And I was told SAS can span in the width instead of length. However, when I try to transpose it, SAS still reports insufficient memory.

Here is another question. Now in my txt file there is only one observation. It will have hunderds observations later. Then in the data "_18Fratio", how should I rewrite the input statement?

Thank you so much!

[data_null__]

Yes I was experimenting with PROC TRANSPOSE on the big file.  It will "never" work too many variables.

What do you mean now in my TXT file there is only one obs?  I thought this last file what the real data.  I don't know what to change because you don't tell the whole story.

[call_me_elaine]

Sorry for some misunderstandings. The last file is part of the real data which I am still working on it. It contains more than 11M variables. And right now it only has one observations (i.e., the second line). It will have about 300 observations later. So in my final txt file, it will have 11M columns and 300 lines.

BTW, we leave out the first variable ("name') which value is "18F". How should we deal with this character variable?

Thank you!

[data_null__]

I don't follow your description.  Please post the "real" data when you have it.

[call_me_elaine]

It may take me several days or even weeks to get my real data. Here is the link to download a newest one. I added one more observation into it. So the file has 11M variables and 2 observation. After I figure this out, I know how to deal with my real data. Thank you so much!

http://dropbox.unl.edu/uploads/20141112/bd4e91882b7a305a/18Fratio2.txt

[art297]

Elaine: While you already have your answer, depending upon how you will eventually want your file created, here is yet another approach that produces a file very similar to a SAS matrix input file:

data matricies;

  infile 'c:\temp\18fratio2.txt' dsd dlm='09'x firstobs=2 lrecl=320000000;

  length rname $32.;

  length varnum 8.;

  array var(3321);

  length name $32;

  input name @@;

  input score @@;

  rname=catx('_',name,score);

  do varnum=1 to 3321;

    do j=1 to 3321;

      if varnum eq j then call missing(var(j));

      else do;

        input number @@;

        var(j)=number;

      end;

    end;

    drop name score number j;

    output;

  end;

run;