01-12-2017 03:28 PM
'm using the following commnad to concatenate 1000 files. But all 1000 files have header it them. How do i just merge data not header from all data files?
x %sysfunc(quote(copy "F:\Test\*monthly*.csv" "F:\Test\2017_monthlytsv")) ;
01-12-2017 03:38 PM - edited 01-12-2017 03:56 PM
You're using OS commands there, so you need an OS command to process the files.
Are you planning to read these files into SAS or is this just a process of using SAS to execute OS commands? If you're reading them into SAS you don't need to combine it, SAS will use the wildcard to read all the files at once.
You'll probably get faster help on a forum dedicated to your OS scripting language, if that's Windows or Unix.
01-13-2017 02:52 AM - edited 01-13-2017 02:54 AM
On UNIX, one would simply do
tail -n +2 file.txt > file.stdout
to remove the first line. Have no idea how to do it on a toybox.
The whole shell script would look like
cd /test OUTFILE=2017_monthlytsv rm $OUTFILE for FILE in *monthly*.csv do tail -n +2 $FILE >> $OUTFILE done
If you install a UNIX environment like cygwin on your Windows, you will get a lot of the useful UNIX utilities, including bash for shell scripting and commands like tail.
01-13-2017 04:30 AM
You can append all the files very simply in one dataset - or actually read it all in in one go:
For both I had two csv files, just with:
This one just reads all the files and writes out one single file with all the data no headers:
data _null_; infile "s:\temp\rob\*.csv" dlm="¬"; file "s:\temp\rob\processed.csv"; input; if not(substr(_infile_,1,5)="abcd,") then put _infile_; run;
You could of course modify this to do all the processing:
data want; infile "s:\temp\rob\*.csv" dlm="¬"; file "s:\temp\rob\processed.csv"; input; if not(substr(_infile_,1,5)="abcd,") then do; a=scan(_infile_,1,","); b=scan(_infile_,2,","); output; end; run;