04-06-2015 04:07 PM
I have created a basic dataset. In the future, the files will be modified to create new fields. How can I append the dataset to include the new files?
This is what I am using:
append base=Cumulative data=Current force;
Of course it works, but the new variables are not being added. I won't know what the new variables are.
I guess I should also include that there is a possibility of an nonexistent dataset. If I use the SET, it will not create because the dataset does not exist. Is there a possible work around for this?
04-06-2015 04:52 PM
Force option only forces the append procedure to continue if it finds variables 1) not in the base data, 2) do not have the same type or 3 are longer than in the base data.
No new variables are added though.
You might find a datastep and SET statement to be more amenable if the contents change frequently:
##WARNING: Even though the construct is legal very great care must be used when using the source and output data set as the same. I might recommend adding a variable for records from current to indicate date appended or some such.
ALSO, with this approach if any variable of the same name are of a different type you'll get errors and fail to append. If any character variables are longer in the Current set they may get truncated to the length in Cumulative.
If the only issue is new variables than you should be good;
To mark the "new" records you could use:
if in2 then DateAdded=today();
04-07-2015 08:44 AM
It would sound to me like you haven't put enough thought into the dataset intially. If you have a base dataset to which further data is added, then it should be robust enough to cover further data. For instance, it sounds like your dealing with transposed datasets, i.e. something like this:
ID JAN15 FEB15...
A 10 20
Each time new data is recieved then new columns would be present. In this situation I tend to advise against having columns with data. Parameterise your data and normalise it:
ID PARAMETER RESULT
A JAN15 10
A FEB15 20
This way your base dataset does not need to change each time, observations can be appended with no problems. If later on you really need a transposed dataset then proc tranpose the base dataset to your requirement.
Know your data and what you want to do with it, thats the key point. Trying to fight that, e.g. by using force, or allowing SAS to guess what you want to do will just give you problems further down the line, say for instance you allow more columns, any program down the line then breaks.
04-08-2015 03:57 AM
ID PARAM RESULT
A DueDate xyz
A AmountPaid xyz
A Datepaid xyz
ID PARAM RESULT
B DueDate xyz
B AmountPaid xyz
B AmountDue xyz
B Datepaid xyz
Set the two together. However if its only one additional variable, then just set the two datasets together. This should all be in your transfer specifications document which documents what is to be transfered, the structure, schedule of transfer etc. I.e you should know what is to be received and when, so just setup from there.