maroulator Tracker

How can I isolate influencial obsevations in logistic regression outliers?

maroulator — Thu, 06 Sep 2018 20:31:39 GMT

To whom it may concern,

I am running the following snippet of code and I am trying to capture outliers using the Pearson Residual, Hat Matrix Diagonal, and DfBeta values. This a very straightforward task if my dataset is less than 10,000 observations. My problem, however, is that once I start working on a dataset with observations in the millions, sifting through my output file becomes an extremely cumbersome process; this is because my output file lists out every single one of the observations in the dataset along with the associated leverage/influence metrics and graphs. Is anyone aware of a way to have my output only display the observations that have a DfBeta, or a Person Residual, or a Hat Matrix diagonal value above a certain level?

proc logistic data=Dataset descending;

model Y=var1 var2 var3 var4/ plcl plrl waldcl waldrl

lackfit rsq

influence iplots

itprint;

run;

Re: Problem importing a txt-file

maroulator — Tue, 28 Nov 2017 22:27:07 GMT

Hi Tom, thanks for the reply. Just to clarify, how did you notice the two different delimiters in the input file? Just by eye-balling it?

Re: Problem importing a txt-file

maroulator — Tue, 28 Nov 2017 21:44:57 GMT

I tried that earlier; the infile statement in the original version of my code looked like this:

infile '/Pathname/InterestRates.txt' dlm=' ' TRUNCOVER MISSOVER lrecl=32767;

The problem persists

Re: Problem importing a txt-file

maroulator — Tue, 28 Nov 2017 21:14:20 GMT

I have; my problem persists.

Problem importing a txt-file

maroulator — Tue, 28 Nov 2017 20:49:13 GMT

To whom it may concern,

I have been trying to import the attached .txt file into SAS using the code below. For some reason, I encounter a problem after var26 whereby some of the remaining variables in my input file either get set to blank or do not get picked up at all. Can someone please provide some insight?

Many thanks in advance.

data rates1;
infile "/Pathname/InterestRates.txt" delimiter='09'x;
input var1-var360;
format var1-var360 best32.;
run;

proc print data=rates1;
run;

Separating observations from one column into two new ones

maroulator — Mon, 06 Nov 2017 03:19:54 GMT

To whom it may concern,

I would appreciate any input you may have on the following:

My data looks as follows:

Column1

rateA

1.2234

rateB

1.5432

rateC

2.1323

I would like to make it look like this:

Column1 Column2

rateA 1.2234

rateB 1.5432

rateC 2.1323

Truncated variables

maroulator — Mon, 31 Jul 2017 23:10:26 GMT

data hpi;
 infile "/file_location/InterestRates.txt";
 input var1-var1000;
run;

proc print data=hpi;
run;

I have the code posted in addition to the attached data. I am trying to import my data into SAS while avoiding the severe truncation that seems to be going on. Could someone please help?

Thank you.

Re: Formating .txt files to specification

maroulator — Mon, 20 Mar 2017 14:04:19 GMT

Hi, sorry I was out of pocket for most of the weekend, so I got to this only just now.

I generated the output file that looks like one row (a.k.a "Output File") from an internal process; SAS has nothing to do with it, all I can see from this internal process is the Output File. A new file that I attach below is the log file generated by SAS after I attempted to import the Output File into PC SAS by using the import wizard. I am running PC SAS on Windows 7.

Re: Formating .txt files to specification

maroulator — Fri, 17 Mar 2017 20:18:48 GMT

Cynthia,

Thanks for reaching out. See the response that I have to your colleague above; I will add that this dataset was generated by an internal system and I am trying to get it into the format I described in my original post to make some downstream processes run more smoothly. Hope this clarifies matters.

Re: Formating .txt files to specification

maroulator — Fri, 17 Mar 2017 20:16:55 GMT

This might help clarify matters; what I posted earlier was a representation of a larger dataset. I attach this larger dataset below; I am trying to convert the attached .txt file into a SAS dataset that has the format I describe, in order to for it better feed into some other downstream processes that I am dealing with.

Formating .txt files to specification

maroulator — Fri, 17 Mar 2017 19:23:06 GMT

I have data that looks like this (the bold font is for effect and ease of reading):

AK 1.0021709654943634 1.004141858960591 1.0059412003301735 1.0075022638405597 1.0089031671832394 1.010313115448636 1.011885064154461 1.0136625876807632 1.0156754189935162 1.0177783819813138 1.0197049657091506 1.0214077118629967 1.023113899728484 1.024961812812696 1.0270842976676144AL 1.0020384552635933 1.004089941400261 1.0060428591630741 1.0078924898011783 1.0097674543713253 1.0116008228367985 1.0134279114567353 1.0152999141462276 1.0173619485116154 1.0194078638786026 1.0213379906824893 1.023262316198122 1.0251279177514947 1.0270029541971306 1.0289116457759255 1.030797300045038 1.0326897516732034AR 1.0018708370386706 1.003678770371349 1.0053816820500132 1.0069865858906413 1.0086739002665075 1.0103784241907938 1.0121787224179806 1.0140531846329055 1.015902586876032 1.0177222600477227 1.019508177957654 1.0212688572482178 1.0229772574375544 1.0247276546883293 1.026411549887851 1.0281958089836294 1.0301388665899276 




And I am trying to make it look like this, instead.
AK 1.0021709654943634 1.004141858960591 1.0059412003301735 1.0075022638405597 1.0089031671832394 1.010313115448636 1.011885064154461 1.0136625876807632 1.0156754189935162 1.0177783819813138 1.0197049657091506 1.0214077118629967 1.023113899728484 1.024961812812696 1.0270842976676144
AL 1.0020384552635933 1.004089941400261 1.0060428591630741 1.0078924898011783 1.0097674543713253 1.0116008228367985 1.0134279114567353 1.0152999141462276 1.0173619485116154 1.0194078638786026 1.0213379906824893 1.023262316198122 1.0251279177514947 1.0270029541971306 1.0289116457759255 1.030797300045038 1.0326897516732034
AR 1.0018708370386706 1.003678770371349 1.0053816820500132 1.0069865858906413 1.0086739002665075 1.0103784241907938 1.0121787224179806 1.0140531846329055 1.015902586876032 1.0177222600477227 1.019508177957654 1.0212688572482178 1.0229772574375544 1.0247276546883293 1.026411549887851 1.0281958089836294 1.0301388665899276

Can someone please propose code to accomplish what I am looking for above? I am at a loss.

Thanks in advance.

Re: Struggling to create .xml file from SAS dataset

maroulator — Tue, 14 Mar 2017 18:14:26 GMT

Thanks. Any ideas on how to deal with this .xml formatting error?

Re: Struggling to create .xml file from SAS dataset

maroulator — Tue, 14 Mar 2017 16:14:15 GMT

Understood; your point is very well taken. I have deleted the loop and I am now using PGStats' code below; the problem now is that I am only getting the AL record; I am missing the AR and AK records. Thoughts?

Re: Struggling to create .xml file from SAS dataset

maroulator — Tue, 14 Mar 2017 16:12:17 GMT

Thanks for getting back to me; I used your code. The problem now, however, is that I am getting only the AL record; I am missing the AK and AR records in my output .xml file.Thoughts?

Struggling to create .xml file from SAS dataset

maroulator — Mon, 13 Mar 2017 21:25:00 GMT

I have the code below which uses the attached .csv file as an input; my problem comes at the _null_ datastep.

I am trying to create a .xml file which holds the dataseries from all three states, but I can only get the last one. Ideally, my output would look smth like this. What am I doing wrong? Note value1 is different across state series.

Any thoughts would be extremely appreciated; thanks in advance.

<path> AL value1 value2................................................................................valueN </path>

<path> AR value1 value2...............................................................................valueN </path>

<path> AK value1 value2...............................................................................valueN </path>

data hpi;
 infile "FilePath\DataRaw.csv" dsd firstobs=2;
 length Date $6; 
 input Date $ AL AR AK;
run;

data newdata;
  set hpi;
   x=indexc(Date,'-');
   if x=3 then do;
     year=compress(substr(Date,1,2));
     month=compress(substr(Date,4,6));
   end;
   if x=4 then do;
     year=compress(substr(Date,5,6));
     month=compress(substr(Date,1,3));
   end; 
   year='20'||year;
   newDate=compress(cat(month,year));
   newDate2=input(newDate,monyy7.);
   format newDate2 monyy7.;
   keep AK AL AR newDate2;
   rename newDate2=Date;
run;

proc transpose data=newdata
                out=newdata1;
		   var AL AR AK;
		   id Date;
run;

data newdata2;
 set newdata1;
 rename _name_=State;
run;

/*Convert dataset into .xml file*/ 

data _null_;
  file "FilePath\HPA.xml";
  set newdata2 end=last;
   /*PUT @001 '<?xml version="1.0"?>' */
   do _n_=1 to last;
     PUT @001 '<path>' /
      state DEC2016 JAN2017 FEB2017 MAR2017 APR2017 MAY2017 JUN2017 JUL2017 AUG2017 SEP2017 OCT2017 NOV2017 DEC2017 
          JAN2018 FEB2018 MAR2018 APR2018 MAY2018 JUN2018 JUL2018 AUG2018 SEP2018 OCT2018 NOV2018 DEC2018
          JAN2019 FEB2019 MAR2019 APR2019 MAY2019 JUN2019 JUL2019 AUG2019 SEP2019 OCT2019 NOV2019 DEC2019 
	  JAN2020 FEB2020 MAR2020 APR2020 MAY2020 JUN2020 JUL2020 AUG2020 SEP2020 OCT2020 NOV2020 DEC2020
	  JAN2021 FEB2021 MAR2021 APR2021 MAY2021 JUN2021 JUL2021 AUG2021 SEP2021 OCT2021 NOV2021 DEC2021 
	  JAN2022 FEB2022 MAR2022 APR2022 MAY2022 JUN2022 JUL2022 AUG2022 SEP2022 OCT2022 NOV2022 DEC2022 
	  JAN2023 FEB2023 MAR2023 APR2023 MAY2023 JUN2023 JUL2023 AUG2023 SEP2023 OCT2023 NOV2023 DEC2023 
	  JAN2024 FEB2024 MAR2024 APR2024 MAY2024 JUN2024 JUL2024 AUG2024 SEP2024 OCT2024 NOV2024 DEC2024 
	  JAN2025 FEB2025 MAR2025 APR2025 MAY2025 JUN2025 JUL2025 AUG2025 SEP2025 OCT2025 NOV2025 DEC2025 
	  JAN2026 FEB2026 MAR2026 APR2026 MAY2026 JUN2026 JUL2026 AUG2026 SEP2026 OCT2026 NOV2026 DEC2026 
	  JAN2027 FEB2027 MAR2027 APR2027 MAY2027 JUN2027 JUL2027 AUG2027 SEP2027 OCT2027 NOV2027 DEC2027 
	  JAN2028 FEB2028 MAR2028 APR2028 MAY2028 JUN2028 JUL2028 AUG2028 SEP2028 OCT2028 NOV2028 DEC2028 
	  JAN2029 FEB2029 MAR2029 APR2029 MAY2029 JUN2029 JUL2029 AUG2029 SEP2029 OCT2029 NOV2029 DEC2029 
	  JAN2030 FEB2030 MAR2030 APR2030 MAY2030 JUN2030 JUL2030 AUG2030 SEP2030 OCT2030 NOV2030 DEC2030 
	  JAN2031 FEB2031 MAR2031 APR2031 MAY2031 JUN2031 JUL2031 AUG2031 SEP2031 OCT2031 NOV2031 DEC2031 
	  JAN2032 FEB2032 MAR2032 APR2032 MAY2032 JUN2032 JUL2032 AUG2032 SEP2032 OCT2032 NOV2032 DEC2032 
	  JAN2033 FEB2033 MAR2033 APR2033 MAY2033 JUN2033 JUL2033 AUG2033 SEP2033 OCT2033 NOV2033 DEC2033 
	  JAN2034 FEB2034 MAR2034 APR2034 MAY2034 JUN2034 JUL2034 AUG2034 SEP2034 OCT2034 NOV2034 DEC2034 
	  JAN2035 FEB2035 MAR2035 APR2035 MAY2035 JUN2035 JUL2035 AUG2035 SEP2035 OCT2035 NOV2035 DEC2035 
	  JAN2036 FEB2036 MAR2036 APR2036 MAY2036 JUN2036 JUL2036 AUG2036 SEP2036 OCT2036 NOV2036 DEC2036 
          JAN2037 FEB2037 MAR2037 APR2037 MAY2037 JUN2037 JUL2037 AUG2037 SEP2037 OCT2037 NOV2037 DEC2037 
	  JAN2038 FEB2038 MAR2038 APR2038 MAY2038 JUN2038 JUL2038 AUG2038 SEP2038 OCT2038 NOV2038 DEC2038 
	  JAN2039 FEB2039 MAR2039 APR2039 MAY2039 JUN2039 JUL2039 AUG2039 SEP2039 OCT2039 NOV2039 DEC2039 
	  JAN2040 FEB2040 MAR2040 APR2040 MAY2040 JUN2040 JUL2040 AUG2040 SEP2040 OCT2040 NOV2040 DEC2040 
	  JAN2041 FEB2041 MAR2041 APR2041 MAY2041 JUN2041 JUL2041 AUG2041 SEP2041 OCT2041 NOV2041 DEC2041 
	  JAN2042 FEB2042 MAR2042 APR2042 MAY2042 JUN2042 JUL2042 AUG2042 SEP2042 OCT2042 NOV2042 DEC2042 
          JAN2043 FEB2043 MAR2043 APR2043 MAY2043 JUN2043 JUL2043 AUG2043 SEP2043 OCT2043 NOV2043 DEC2043 
	  JAN2044 FEB2044 MAR2044 APR2044 MAY2044 JUN2044 JUL2044 AUG2044 SEP2044 OCT2044 NOV2044 DEC2044 
	  JAN2045 FEB2045 MAR2045 APR2045 MAY2045 JUN2045 JUL2045 AUG2045 SEP2045 OCT2045 NOV2045 DEC2045 
	  JAN2046 FEB2046 MAR2046 APR2046 MAY2046 JUN2046 JUL2046 AUG2046 SEP2046 OCT2046 NOV2046 DEC2046 
     ' </path> ';
    end;
    output;
run;

Converting dates from external file

maroulator — Fri, 10 Mar 2017 19:32:59 GMT

I have the data in the attached .csv file; the Date variable containes future dates for the appropriate Index. The Date variable gets imported into SAS as a character variable after I use the appropriate proc.

I would like to either a) import the variable into SAS as a numeric variable or b) convert the variable into numeric after I have imported it into SAS. Also, I would appreciate any thoughts on how I can deal with the issue that some of my values in the Date variable show up as mmm-yy, rather than yy-mmm.

I would appreciate any insight that you may have to offer.

Thanks in advance.

NOTE: Line generated by macro function

maroulator — Sat, 22 Oct 2016 04:38:43 GMT

I have the split macro below that I have been trying to run unsuccesfuly because I get the following error; NOTE: Line generated by the macro function "I." I have tried some solutions I found online, but I really think that I'm missing smth very obvious; could someone please take a look and let me know what they think? The in.cur_mod2 dataset has 59,325 observations and I am trying to split it into smaller datasets.

Thanks in advance.

options mlogic mprint symbolgen;

%macro split(num);

data _null_;

if 0 then set in.cur_mod2 nobs=count;

call symput('numobs',put(count,8.));

run;

%let n=%sysevalf(&numobs/&num,ceil);

data %do J=1 %to &num; orig_&J %end;;

set in.cur_mod2;

%do I=1 %to &num;

if %eval(&n*(&i-1))<_n_<=%eval(&n*&I)

then output in.cur_mod2_&I;

%end;

run;

%mend;

%split(10);

Filling in blanks

maroulator — Thu, 15 Oct 2015 20:53:41 GMT

My data looks as follows:

LoanID Modification_Year

3322 .

3322 2010

3322 .

3322 2013

3322 .

1341 .

1341 2011

1341 .

Instead, I would like to have it look like the data below. Does anyone have any code to recommend? Thanks in advance.

LoanID Modification_Year

3322 2010

3322 2013

1341 2011

Filling in missing values

maroulator — Wed, 16 Sep 2015 14:45:50 GMT

My data currently looks like this:

LoanId Product_Code

32456 Fixed-30

32456

75639 Fixed-15

75639

84732 Adjustable-Rate

84732

I need to fill in the blanks in the product_code variable so that my records have no blanks; please not that product_code is a character variable. Utlimately, I need my data to look like this.

LoanId Product_Code

32456 Fixed-30

75639 Fixed-15

84732 Adjustable-Rate

Any help would be extremely appreciated; thanks in advance.

Selecting records

maroulator — Wed, 16 Sep 2015 05:57:04 GMT

My data looks like this:

LoanId Mod_Flag

12345

12345 Y

34521

68749 Y

68749

78423

Is there any way that I could select a) all records corresponding to those LoanIds that have a blank (ie. non-Y) mod_flag throughout the LoanId group, in addition to b) the records from those LoanId groups whose mod_flag may either be blank or mod_flag='Y', corresponding to those LoanIds that ONLY have a mod_flag = 'Y'? To be clearer by final data would look as follows:

LoanId Mod_Flag

12345 Y

34521

68749 Y

78423