proc datasets append force

[alepage]

PROC DATASETS LIBRARY=  DATA
                NODETAILS
                NOLIST
                FORCE;
 APPEND BASE= DATA.REPORT&YYYY.&MTH.
 DATA= WORK.B2;
 RUN;

Hello, two new fields were recently added into the dataset B2, which are var1 and var2 (name are changed for safety  purpose), but they are not included in DATA.REPORT&YYYY.&MTH.

Here's the log error.

WARNING: Variable var1 was not found on BASE file. The variable will not be added to the BASE file.
WARNING: Variable var2  was not found on BASE file. The variable will not be added to the BASE file.

Also, during the process (main program) mainy macro function are called which use the proc datasets procedure.

So at the begining, the data.report&yyyy.&mth. may not exist but after the first reference to that procedure, this dataset will exist and the dataset b2 will be added.

How to solve that issue since the force option does not seems to work on Unix Server ?

[Reeza]

Those are not errors, they're warnings.
Is the correct data set being created?
A warning tells you of unexpected behaviour.

[Reeza]

Also try adding the NOWARN statement to see if it suppresses the warning if this is the intended output.

[ballardw]

Append has never allowed adding variables as far as I know. This behavior is because you are not recreating the set header information when using Proc Append or the Proc Datasets Append statement.

If you know that you need to add variables then that is a manual step or if you need to attempt such programmatically then some code that compares your base data set and the new data to append and if the sources are different enough use a data step to combine the sets.

You can use a data step to append data and create a new master data set with something like:

data new;
   set DATA.REPORT&YYYY.&MTH.   work.b2;
run;

If you are real confident of no other issues (I wouldn't be without checking variable properties) you could use the base data set name in place of the New in the example code.

Force is more intended to append sets when the new data is missing variables in the base data set OR a property like lengths of variables may differ but not catastrophically. Which may lose data when appending new values that are 50 characters long into an existing variable that is only 10 characters.

[Tom]

I cannot tell from your description what you want to happen.

If the new dataset has extra variables you cannot APPEND the new variables.  There is no place to store the values.

Do you want the new variables to be ignored?  If so it sounds like your program worked fine.

Do you want the new variables to be included? If so you cannot use APPEND to do that.  You will need to make a new dataset first that has the new variables.

You also mentioned that this it part of some looping code that perhaps is making the BASE dataset  automatically when it does not yet exist.  In that case just make sure to order the processing of the datasets so that the one with ALL of the variables is appended first.  Then the new BASE dataset will have enough variables.

[Kurt_Bremser · Accepted solution]

Since additional variables change the structure of a dataset, a complete rewrite of the dataset is needed, so you cannot do that with PROC APPEND.

If you want the new variables to appear in your dataset, you have to use a DATA step to concatenate the datasets and create a new one.

[alepage]

I have read all your message and I have found them quite interesting.

I did some tests with the dataset sashelp.class and the procedure proc datasets append force to see / understand it's behavior.

So, to conclude, the solution that I found is to create a new data structure as proposed by Kurt.

Here's my code 

data class;
set sashelp.class;
rownum = _n_;
run;

data class1;
set class;
if _n_ lt 10 then output;
run;

data class2;
set class;
if _n_ ge 10 and _n_ le 15 then output;
run;

data class3;
set class;
if _n_ gt 15 then output;
run;

data test;
if _n_ = 1 then delete;
run;

data test1;
set test class1;
run;

data test2;
set test1 class2;
run;

data test3;
set test2 class3;
run;

data test;
set test class1;
run;

data test;
set test class2;
run;

data test;
set test class3;
run;

/**********************/

proc contents data=class;
run;

/*  Create the libray data ***/
%let path=;
libname data base "&path./data";

data data.class_modif;
set class (drop=sex weight);
stop;
run;

proc contents data=data.class_modif;
run;

/****** Let's test how the proc datasets append procedure works ***/

%let yyyy=2023;
%let mm=05;
%let testsuf=_a;

  PROC DATASETS LIBRARY=  DATA
                NODETAILS
                NOLIST
                FORCE;
       APPEND  BASE= DATA.REPORT_&YYYY.&MM.&TESTSUF.
               DATA= WORK.class1;
  RUN;

 /************ start with class_modif ***/

%let yyyy=2023;
%let mm=05;
%let testsuf=_b;

  PROC DATASETS LIBRARY=  DATA
                NODETAILS
                NOLIST
                FORCE;
       APPEND  BASE= DATA.REPORT_&YYYY.&MM.&TESTSUF.
               DATA= WORK.class_modif;
  RUN;


  /*** adding class1  ***/

%let yyyy=2023;
%let mm=05;
%let testsuf=_b;

  PROC DATASETS LIBRARY=  DATA
                NODETAILS
                NOLIST
                FORCE;
       APPEND  BASE= DATA.REPORT_&YYYY.&MM.&TESTSUF.
               DATA= WORK.class1;
  RUN;

data work.REPORT_&YYYY.&MM.&TESTSUF.;
set DATA.REPORT_&YYYY.&MM.&TESTSUF.;
run;

/******** The solution   ******/
data model_1 ;
   attrib 
          Age       label="Age" 		length=8 	format=8. 	informat=8.
		  Height    label="Height" 		length=8 	format=8. 	informat=8.
		  Name    	label="Name" 		length=$8. 	format=$8. 	informat=$8.
		  Sex 	 	label="Sex" 		length=$1. 	format=$1. 	informat=$1.
          Weight 	label="Weight" 		length=8 	format=8. 	informat=8.
		  rownum 	label="Row Number" 	length=8 	format=8. 	informat=8.
   ;
STOP;
run;

%let yyyy=2023;
%let mm=05;
%let testsuf=_c;

data DATA.REPORT_&YYYY.&MM.&TESTSUF.;
set model_1;
run;

data work.REPORT_&YYYY.&MM.&TESTSUF.;
set DATA.REPORT_&YYYY.&MM.&TESTSUF.;
run;

PROC DATASETS LIBRARY=  DATA
                NODETAILS
                NOLIST
                FORCE;
       APPEND  BASE= DATA.REPORT_&YYYY.&MM.&TESTSUF.
               DATA= WORK.class1;
  RUN;

data work.REPORT_&YYYY.&MM.&TESTSUF.;
set DATA.REPORT_&YYYY.&MM.&TESTSUF.;
run;