topic Re: Creating a file of bypassed records from reading an input file in SAS Programming

Creating a file of bypassed records from reading an input file

BeerSultan — Fri, 14 Jun 2024 19:05:31 GMT

Brand new to SAS.

I'm doing something that I expect is relatively easy, but I'm struggling to get it right. The requirement is basic - I'm reading in a long file with many rows, if a record matches a criteria I want it to be shipped to a different "bypass" file that I'll work on in a later process, but it should be removed from the original read-in file.

Environment is zOS - input file is defined and passed in via JCL (all working).
I get the following errors if I remove my actual logic - if I leave the FILENAME statement in without referencing it there's no errors.
ERROR: Invalid logical name.
ERROR: Error in the FILENAME statement.
ERROR: Invalid file, BYPASS_FILE.

FILENAME BYPASS_FILE BYPASS1;

/***********************************************************/
/* READ FILE */
/***********************************************************/
DATA DATAIN;
INFILE DATAIN;
INPUT
@ 00001 RECORD_NUMBER $CHAR8.
@ 01769 COMPONENT_STATUS $CHAR1.
;

/*******************************************************************/
/* CREATE BYPASS FILE - USED TO STORE RECORDS WHICH HAVE AN */
/* INACTIVE COMPONENT: COMPONENT_STATUS = I */
/*******************************************************************/
IF COMPONENT_STATUS = 'I' THEN DO;
 FILE BYPASS_FILE;
 PUT;
 DELETE;
END;

Re: Creating a file of bypassed records from reading an input file

s_lassen — Fri, 14 Jun 2024 19:07:59 GMT

I am not quite sure, but the message "Invalid logical name" sounds like it is the name "BYPASS_FILE" that is too long. It may be possible to have filename references in SAS (depending on the version) that are more than 8 characters, but if it gets passed to z/OS as a file reference/DD name, it probably won't work. Try using a shorter name, e.g. "BP_FILE".

Re: Creating a file of bypassed records from reading an input file

BeerSultan — Fri, 14 Jun 2024 19:57:12 GMT

Yeah - no luck. Still giving invalid error.

Re: Creating a file of bypassed records from reading an input file

Kurt_Bremser — Fri, 14 Jun 2024 20:36:35 GMT

The parameter of a FILE statement must either be a file reference (without quotes) created with a FILENAME statement or coming from the JCL, or a physical path name enclosed in quotes.

Since you did not use quotes, the file reference must be 8 or less characters long, and must have been defined upstream (in the JCL or SAS code).

If you want to supply a physical name, it must meet the requirements of z/OS.

Re: Creating a file of bypassed records from reading an input file

BeerSultan — Fri, 14 Jun 2024 21:28:24 GMT

I was able to work through a couple iterations of it and got past those file-related errors.

The code thus far is as follows:

  IF component_status = 'I' THEN DO;
    FILE bypass;
    PUT
      @ 00001 record       $CHAR8.
    ;
    END;
    ELSE DO;
    FILE inputf
    PUT 
        @ 00001  lots of fields
        ;
  END;

For whatever reason the record meeting the IF criteria is still getting into the other file. I want it just in the bypass file, and to be excluded from the data flowing in via the input file...I thought the IF condition would exclude it from going into the PUT for the other file.

Re: Creating a file of bypassed records from reading an input file

BeerSultan — Fri, 14 Jun 2024 21:31:00 GMT

I do have it coming via JCL and it's named accordingly. I ended up abandoning using the FILENAME approach altogether...I don't know how to make it work that way.

BYPASS DD DSN=XY.#xyz1234.TEST.DATA.BYPASS,
DISP=(NEW,CATLG,DELETE),UNIT=TST1,
DATACLAS=FXB080CP,LRECL=40,
SPACE=(40,(1,1),RLSE),AVGREC=K

Re: Creating a file of bypassed records from reading an input file

Astounding — Sat, 15 Jun 2024 06:56:50 GMT

Please tell us this is just a typo and you really do have a semicolon after

FILE inputf

Re: Creating a file of bypassed records from reading an input file

Kurt_Bremser — Sat, 15 Jun 2024 06:57:51 GMT

As soon as a FILE statement executes, it opens the file, and the file is kept open. Multiple FILE statements mean multiple concurrent destinations.

The FILE Statement documentation shows how the output file can be changed with the FILEVAR= option, but you have to use physical names in this variable, not file references.

Re: Creating a file of bypassed records from reading an input file

Tom — Sat, 15 Jun 2024 16:30:52 GMT

The indentation is a little messed up but it looks like you are just missing s semicolon.

data _null_;
  set have;
  IF component_status = 'I' THEN DO;
    FILE bypass;
    PUT
      @ 00001 record       $CHAR8.
    ;
  END;
  ELSE DO;
    FILE inputf;
    PUT 
        @ 00001  lots of fields
    ;
  END;
run;

So that code is WRITING from a DATASET to a FILE. Which is not the impression I got from your original post. It sounded more to me like you were trying to READ from a file and spit out the other LINES from the file to a separate file.

So a data step like this will create WANT dataset from the input file IN. It will check something on the line and decide whether it should write that line to the OUT text file or continue with the rest of the data step that reads the line and does any other things like calculating any derived variables.

data want;
  infile IN ;
  file OUT ;
  input @1 check $ @1 @ ;
  if check='X' then do;
     put _infile_;
     delete;
  end;
  input ....;
  ....
run;

And if what you meant was just about SAS datasets then no need for FILE or INFILE or PUT or INPUT statements.

data good bad;
  set have;
  if check='X' then output bad;
  else output good;
run;

Re: Creating a file of bypassed records from reading an input file

Tom — Sat, 15 Jun 2024 23:14:07 GMT

@BeerSultan wrote:
I do have it coming via JCL and it's named accordingly. I ended up abandoning using the FILENAME approach altogether...I don't know how to make it work that way.

BYPASS DD DSN=XY.#xyz1234.TEST.DATA.BYPASS,
DISP=(NEW,CATLG,DELETE),UNIT=TST1,
DATACLAS=FXB080CP,LRECL=40,
SPACE=(40,(1,1),RLSE),AVGREC=K

When you define the access with a DD statement then in the FILE statement you just use the DDNAME you used in the DD statement. Which in your example DD statement is BYPASS.

file bypass ;

If you wanted to mimic the DD statement in a SAS filename statement instead then it would look very similar to the DD statement, but it would look like SAS code instead of JCL code.

Check the syntax of the filename statement on z/OS.

filename bypass disk 'XY.#xyz1234.TEST.DATA.BYPASS' DISP=(NEW,CATLG,DELETE) ..... ;

Re: Creating a file of bypassed records from reading an input file

BeerSultan — Sun, 16 Jun 2024 19:35:40 GMT

I've got it mostly working, but now the record is in both files... I want the record to go to only the bypass file when it's found, otherwise only go to the other file (which is just the incoming file)

  IF COMPONENT_STATUS = 'I' THEN DO;
    FILE LTABYPSS;
    PUT
      @ 00001 RECORD_NUMBER       $CHAR8.
    ;
    END;
    ELSE DO;
    FILE DATAIN
    PUT
        @ 00001  lots of values
...
        ;
  END;

Re: Creating a file of bypassed records from reading an input file

Kurt_Bremser — Sun, 16 Jun 2024 21:45:51 GMT

As already said, you need a single FILE statement with a FILEVAR option, and change the contents of that variable. Or you run two separate DATA steps, one for each output file.

Re: Creating a file of bypassed records from reading an input file

BeerSultan — Sun, 16 Jun 2024 22:23:14 GMT

It's part of an existing process that I'm expanding to exclude this bypass file - which is why I was attempting to use the existing reading infile processing and just shunt off the "found" records that I want to exclude. The bypass file will be used in a next step to build a report with counts/rationale.

I don't want to mess with the existing processing too much because it'll cause a larger change than what I'm comfortable doing.

Re: Creating a file of bypassed records from reading an input file

Kurt_Bremser — Sun, 16 Jun 2024 22:32:11 GMT

I have given you the possible solutions, it's your choice which to pursue.

Re: Creating a file of bypassed records from reading an input file

Tom — Mon, 17 Jun 2024 01:51:21 GMT

@BeerSultan wrote:

It's part of an existing process that I'm expanding to exclude this bypass file - which is why I was attempting to use the existing reading infile processing and just shunt off the "found" records that I want to exclude. The bypass file will be used in a next step to build a report with counts/rationale.

I don't want to mess with the existing processing too much because it'll cause a larger change than what I'm comfortable doing.

You have not provided enough information to give you complete solution, only suggestions.

So if you have an existing program that reads lines from a text file (either to make a dataset or make another file, I cannot tell which from your descriptions and the snippets of programs you have shared) that you now have to change it to also have it copy some of the lines from the source text files to a new text file.

So you need to change it in the following ways.

1) Come up with logic that detects which lines it is that you want to write to the file.

2) Conditionally execute the FILE and PUT statement to write the line to the new file.

I showed this before, so let's try again. Let's assume you have some normal program that read a text file into dataset.

data want;
  infile myfile ;
  input ..... ;
run;

So first you have to add the logic to detect the lines you want to skip. This could be a simple as jut reading some part of the line that indicates what type of line it is. So perhaps you want reject the lines that have X in column 32.

Then you can add code that will write those lines to the new file and not include them in the normal output of the data step.

data want;
  infile myfile ;
  input @32 check $1. @1 @ ;
  if check='X' then do;
    file newfile;
    put _infile_;
    delete;
  end;
  input ..... ;
  drop check;
run;

Re: Creating a file of bypassed records from reading an input file

Tom — Mon, 17 Jun 2024 02:05:10 GMT

You are still missing the semicolon on the FILE statement. (and your indentation is still very confused).

Assuming you fixed the FILE statement that code would NOT write to both of those text files. At least not the little snippet. But since you did not DELETE the observation that you wrote to the BYPASS file than that observation would make it into the DATASET that the step is creating.

Why are you writing to a file you are pointing to with a FILEREF (or DDNAME) of DATAIN? Normally that would be the file you want to READ from. Is there some later process that will read from that file? Is that part of this SAS code? Or some other JCL step in your JOB?

Re: Creating a file of bypassed records from reading an input file

BeerSultan — Mon, 17 Jun 2024 03:30:48 GMT

Long-story short...

I'm modifying an existing report that takes hundreds of thousands of insurance policies and calculates financial reserves. A change to a very small subset of policies now has some benefits becoming inactive - so we want to exclude them. However, this indicator data field is only available up to this current point in the job I'm modifying, as it's output file does not contain this status code. The next job step is the summary part, where it would be easy to count up this bypassed-due-to-inactive-status policies. Logically the simplest solution would be to expand the output to include this field so we can just count in the next step - but it is a widely used data structure, and so would involve making changes to a couple dozen other jobs if expanded. We don't have the bandwidth to tackle that kind of work at the moment, but we want to get these policies off the books as we're setting aside not-insignificant amounts of money that could instead be reinvested.

The goal of my process is to find the policies that don't need to have their reserves calculated, pull them out of the mainline processing (the incoming read file) and ship it to a second file. This second file will be sent to the next job step and I'll just count the rows and slap it into the summation section for # of policies skipped due to inactive benefit component. Fulfilling the requirement of the business now, versus in months, and not significantly impacting anything outside of this specific need.

The code I provided has the record on both files coming out of this job.

Re: Creating a file of bypassed records from reading an input file

Kurt_Bremser — Mon, 17 Jun 2024 06:58:16 GMT

While writing to multiple files in one DATA step is possible, because of the need for physical filenames it can be tricky on z/OS.

Instead I would write the data to two WORK datasets (the OUTPUT statement can be given a destination name to write to, which is not possible with PUT), and then run a DATA step for each to create the files.

Re: Creating a file of bypassed records from reading an input file

Tom — Mon, 17 Jun 2024 13:24:33 GMT

I still don't see why this is such a problem. But you still have not explained the process with enough detail for us to help you.

Are you reading in TEXT files (I know that old time IBM mainframes like to call plain old file "data sets" but they have nothing to do with an actual SAS dataset. They are just file.) in this step that does have the information you need?

What is the current OUTPUT of that step? Is it another TEXT file? Or is it a SAS dataset?

If it is a TEXT file then just skip the code that writes the observation to the text file (however many lines it writes) and they will "disappear" from the file consumed by the downstream systems.

If it is SAS dataset then just skip writing to the that dataset. Does the current step have any OUTPUT statements? If so then skip if for those cases. If not then you need to either add an OUTPUT statement for the observations you to make it to the next step or add a DELETE statement for the ones you don't want to make it to the next step.

Re: Creating a file of bypassed records from reading an input file

BeerSultan — Mon, 17 Jun 2024 14:19:48 GMT

I also don't know why it's such a problem, which is why I'm asking for help.

I feel I explained enough that it should be evident what the underlying ask is - like you said this shouldn't be a problem. But, here's a recap; 2 (text)files - the second has no bearing on the ultimate process - come in from the mainframe, it changes a few fields based on criteria, and it outputs a single merged (text)file. This merged output file does not contain the field necessary in the next step to count/summarize in the next step. I cannot expand the output file to accommodate at this time because it has extensive impacts across dozens of other jobs which would need to be changed and tested and validated.

I want the input file with the status code to count the to write a bypass record in the new bypass (text)file when the status code is inactive. I want the record to not be on the existing output file because the values in that file are used to calculate reserve cash, and an inactive policy does not need to reserve because the benefit is gone. I'll count the records in the bypass file in the next step and add it to the summary page. That's it.