Programming 1/2 content: removed INFILE and added macro vars?

[Quentin]

Hi,

Saw this great post about changes to the SAS base certification process https://blogs.sas.com/content/sgf/2019/02/22/thinking-about-getting-sas-certified-check-out-the-new-... , but it raised some questions for me (as an experienced SAS user) about the design of current content for Programming 1/2.  

The big shock for me was that SAS Programming 1/2 does not teach how to read data from raw text files with the INFILE / INPUT statements, it only teaches PROC IMPORT for reading raw files.  Really?  When I took Programming 2 many years ago (I know it was a different time, different course structure), it was 3 days, and I feel like they spent 1/2 the first day on reading text files.  I know that there is a lot more data in SAS datasets or RDBMS than there was 20 years ago, but reading data from text files is still a core strength of SAS.  As an employer, if I sent a new hire to programming 1/2 and they came back to work and saw code with an INFILE statement and said "what's that?", I'd be surprised, and not in a good way.  I wouldn't expect them to be able to recite the difference between flowover, truncover, and missover.  But I would expect that they had been shown at least a few data steps that read from text files in different ways, and been exposed to trailing @.  I never got great at reading crazy text files myself, but it still comes up a few times a year, and I still regret that I managed to lose my programming 2 course book with my notes about 10 years after I took it.

The second disappointment was that the post says macro variables are now taught in programming 2.  I'm a big fan of the macro language, but I don't think it should be taught that soon.  I was lucky to work with Ian Whitlock early in my career.  He pointed out to me that many SAS programmers make the mistake of trying to learn the macro language too soon, i.e. before they learn the SAS language.  You need to really understand the SAS language before you should try learning a new language in order to generate SAS code.

The other thing Ian taught me was that it's probably a mistake to introduce the macro language by introducing macro variables first.  He thought it was better to introduce a simple macro.  If in programming 2 you introduce the concept of a macro variable (via %let maybe?) and nothing else, students may leave with the idea that it's a good idea to use open code %LET to make a global macro variable in order to make their SAS code re-usable.  Then if they take the macro course, they'll learn that global macro variables cause a lot of problems that can be avoided by using parameterized macros with local macro variable scopes.

I hope this doesn't sound too critical.  Trust me, I have an insane amount of respect for SAS training, trainers, and the courses you offer.   If I hadn't had such a good experience with SAS training 20+ years ago, I wouldn't have my career.  And over those 20 years I've been lucky to have many more excellent opportunities to learn from SAS training, and was even able to bring SAS trainers in to my company for on-site training, and they happily allowed us to customize the training course.

So I'm a huge fan of SAS, and of SAS training.  I was just surprised enough about the decision to drop INFILE from SAS programming 1/2, and add macro variables, that I would ask if anyone could share more of the rationale, and maybe get some feed back from other experienced SAS programmers as well.

Thanks!

Quentin

[Astounding]

Agreed on all counts.

This might explain the nature of some of the questions that we see ... people asking macro language questions who know hardly any macro language, and very little SAS.  And tons of questions on importing data.  That's what they're being taught to do, so that's the basis for their questions.

There have been numerous occasions where I bite my tongue, instead of saying, "The real problem is that you are attempting to use macro language when you don't know macro language and you don't know SAS."  It's unfair to the students, forcing them to learn two things at the same time.  So make room on top of that soapbox  You've got company.

[Kurt_Bremser]

proc import is an add-on that can be used as a quick helper, but nothing more. Teaching it can only be of use if the automatically created data step is used as a learning example, but it's better covered as a side-note.

Dropping infile and input in favor of proc import only serves to retard the development of a new programmer.

[SASKiwi]

@Quentin -Great post. Always good to hear about current training trends. I too get the impression that new SAS users are jumping to using macro before they have mastered basic SAS coding. It's no wonder the cart ends up in front of the horse.

IMO there is definitely a trend to more structured data file input like delimited files, XML, JSON etc. However PROC IMPORT only goes so far and you still need to resort to DATA step INFILEs from  time to time.

In my line of work DATA step, base PROCs and SQL, both SAS and PASSTHRU are what I use most, followed by macro. I would definitely put SQL ahead of macro when learning SAS.

[ChrisNZ]

Thanks for the question. That's interesting to know.

I am not as critical as you about removing INFILE from the first course. I always wondered why SAS training started there.

proc print, tabulate, etc are easy and meaningful introductions to the SAS language. Then I'd say basic data steps and/or SQL and/or formats.

People only taking Programming 1 are better off knowing these tools than spending a large part of the training on the various INFILE options. Importing raw data is a more advanced topic than using data. Level-1 SAS users normally have their data provided to them in a ready-to-use form.

So without knowing the details, the idea of INFILE in a later course seems sensible to me.

As for the macro language, the questions we see here really do plead for a more cautious introduction. 

Seemingly clueless people try to use it without understanding it, and it can only end in confusion.. 

[Quentin]

Thanks @ChrisNZ .  I could maybe understand leaving INFILE out of programming 1, if the argument is the important thing to learn there is how the data step works, and how procs work, so hold off on teaching INFILE until programming 2.  But INFILE is also not taught in programming 2.  So you can complete Programming 1 and Programming 2, and never learn INFILE.  This also means that with the new certification exam being updated to match the Programming 1 & 2 courses, you can be a certified base programmer and not know how to read raw data.  That seems like a knowledge gap.

I just took a quick look at the programming training pathway (http://support.sas.com/training/us/paths/prg.html), thinking maybe they would teach INFILE as part of the advanced course, but I don't see it there either.  Which I think would mean that you could become a certified advanced programmer without knowing how to read raw data.

[ChrisNZ]

Not having INFILE taught in the first course seems totally sensible and is a good move imho.

Whether it should be in Programming #2 or #3 has no clear answer. Priorities and so forth....

Taking INFILE completely out of the curriculum leaves a gaping hole, we agree there.

[yabwon]

Hi Quentin,

100% agree with you. Dropping the infile/input statement and arrays concept from base doesn't sound good to me. And with all respect to SAS training center - I'll keep both in course notes for my students.

all the best

Bart

[Astounding]

Having had time to mull this over a bit ...

It wouldn't shock me if SAS Institute is working on a project to eliminate the INPUT statement entirely.  Just create a TEXT engine for the libname statement.  I haven't attempted to list the many obstacles and how they might be addressed.  And I have no idea what the timetable might look like for such an effort (except to note that there must have been some progress made, or we would still see the INPUT statement in Programming 1).  And I have no idea whether any of this is true ... pure speculation on my part as I try to view the curriculum as being the result of a logical decision.

[Quentin]

Well, that's an interesting idea:

---

@Astounding wrote:

Having had time to mull this over a bit ...

It wouldn't shock me if SAS Institute is working on a project to eliminate the INPUT statement entirely.  Just create a TEXT engine for the libname statement.  I haven't attempted to list the many obstacles and how they might be addressed.  And I have no idea what the timetable might look like for such an effort (except to note that there must have been some progress made, or we would still see the INPUT statement in Programming 1).  And I have no idea whether any of this is true ... pure speculation on my part as I try to view the curriculum as being the result of a logical decision.

---

As you say, I can't imagine how they could work all the flexibility of the INPUT statement into an engine that would not need 100s of options.  But it's an intriguing idea.  I know SAS bought DBMS copy many years ago.  I can't remember if DBMS copy had much magic in it for munching oddly formatted text files.  I used it for Excel<->SPSS and similar, in my pre-SAS days. 🙂

Here's another speculation:  Maybe it's SAS taking a stand against badly formatted text files.  That is, by saying "all your data should be in a database, or SAS dataset, or if a text file it should be simple CSV which can be handled by PROC IMPORT, or XML/JSON", maybe they're trying to raise the bar on what data formats SAS programmers should expect to receive.  So you replace the idea "SAS can parse anything you throw at it" (trailing @ and regular expressions etc etc ) with "SAS can read reasonably-structured data."

To me, that feels more appropriate for the "analyst" track where analysts may be able to expect to receive structured data.  I fear programmers will be dealing with odd text files for the foreseeable future.

[StaceySyphus · Accepted solution]

I love this discussion!  I know you are all passionate about SAS and training up the next generation of SAS programmers, so thank you for taking the time to share your thoughts and ask for more information.  I was one of the authors for both courses and would like to provide a little insight regarding the development process to create the new Programming 1 (PG1) and Programming 2 (PG2) courses.  My response is a bit of a novel, but I know those of you reading this thread care deeply about this topic and I hope you will take the time to read it.  Here were our main goals:

• Teach what our customers say are the most important and frequently used features in Base SAS programming.
• Showcase those features in a simple and straight-forward way.
• Give new programmers confidence that they can get started quickly using SAS in their work and help experienced programmers (Python, R, etc.) see the unique advantages SAS offers.
• Create both eLearning and classroom learning experiences that are modern and engaging. 
• Be efficient and respectful of each learner’s time.

I’m sure we all can agree that sending someone to PG1 to “learn SAS” is like saying after a child takes an online driver’s training course, they are a good driver.  Each new learner obviously has to start with the fundamentals, but then it takes time and experience for SAS programmers to master their particular area of expertise.  We want the PG1 and PG2 courses to expose students to the breadth of what SAS can do - namely access data, explore data, prepare data, analyze and report on data, and export/share results.  And if you’ve been using SAS enough, you know that the depth of what SAS can do related to each of these topics is endless!  We can’t teach everything in class, but we know the modern learner wants to first know what is possible, then use their research skills or other learning opportunities to dive deeper into the topics that are most relevant. 

A great deal of external customer research went into determining the topics and format of the new PG1 and PG2 courses. We created a survey asking advanced certified programmers to rank 72 different Base SAS skills in terms of importance. We received over 1,000 survey responses from around the world. We also conducted several personal interviews with managers of SAS programmers in various industries.  So we took all this customer feedback, and I’m sure you can imagine – we analyzed it and debated it for more hours than I care to admit.  Here are some of the details about decisions we made: 

1. Course delivery method – In an effort to make our courses more engaging and appropriate for the modern learner, we wrote the new versions with more of a hands-on focus.  We have fewer slides and more activities and practices where students are discovering and reinforcing key concepts while they are writing or modifying code. 
2. Course timing – The managers of SAS programmers we interviewed and management within SAS encouraged us to reduce both courses to 2 days. This was the biggest challenge.  Going from 3-day to 2-day courses obviously means we can’t cover as much material.  While our teaching methods are more efficient, we still had to make tough choices to determine the highest priorities to cover in our intro courses.  We used our customer research and LOTS of internal discussion to make those decisions.  Now that we have taught the new versions of the courses for about 6 months, we have looked carefully at our course evals and listened to feedback from our instructors.  The feedback indicates that those learners who come to class with some programming experience in another language seem ok with the pace and length of the course. However, those learners that are new to programming have expressed they want more time to learn and a third day would be beneficial. So later this year, we will go back to 3 days for PG1, which will allow learners more time to learn and practice the material.  There will be time for students to complete a comprehensive case study.  We will also develop several supplemental lessons that instructors can teach depending on time and interest.  For example, one of the lessons will be about using INFILE/INPUT to read raw data.  Another lesson will be about data visualization using SGPLOT and SGMAP.  We’re constantly trying to improve the learning experience and value of our training, and I think this will definitely do that!   The PG2 course seems to fit well so far in the 2-day format, so we aren’t making any changes there.     
3. Reading text files with INFILE/INPUT – Over the 18 years I have taught programming courses, I have observed interest in this topic steadily decline.  My fellow instructors all agree.  In our customer survey, we asked customers about 29 different skills related to data access and manipulation. The 2 topics related to reading data with INFILE/INPUT ranked 26^th and 28^th.  When I interviewed managers of SAS programmers, they indicated that this is not used nearly as much as it used to be.  One person we interviewed commented, “Reading data with INFILE/INPUT is an incredibly important skill...to a really small group of programmers.”  So although I think we would all agree it is a critical skill for those who need to read non-standard or complex text files, given the higher priority of other topics and the obvious constraints we have with the amount of material we can cover in class, this topic was removed.  Using PROC IMPORT/EXPORT and other LIBNAME engines ranked much higher in our customer survey, so we cover those concepts.  But we do let students know that if PROC IMPORT doesn’t do what they need, the DATA step gives them even more control.  Currently, if a learner takes the classroom or live web version of PG1, we give them access to an online Extended Learning Page (ELP).  The ELP includes supplemental material we think would be helpful.  The INFILE/INPUT sections from the old PG1 and PG2 eLearning courses are available on the ELPs for both PG1 and PG2.  For those learners that NEED that information for their job, we have great material that they can still use.  And as I mentioned in point #2, when PG1 expands to 3 days, instructors can teach the lesson on INFILE/INPUT if learners are interested. 
4. Creating macro variables with %LET statements in PG1 - This topic ranked #3 overall in our customer survey. People learning SAS with experience in other programming languages expect SAS to have this functionality, and should see the basics in our introductory course. It is a powerful part of the SAS language and we need to showcase it in a simple way. We don’t spend a ton of time on it, but just show learners how text can be stored with %LET, then substituted into a TITLE and WHERE statement. When we tested this course with students who were new to SAS programming and not experienced in other languages, they collectively commented that the section on macro variables was straightforward and easy to understand.  They were also successful in completing the activities and practices. Granted, to say we ‘scratch the macro surface’ is an understatement.  Can someone get in trouble using macro variables without an Art-Carpenter-level of macro knowledge?  Sure.  I would say the same thing about the DATA step, but we have to start somewhere!  As a HUGE part of the power of SAS, we feel it is critical our new learners know the macro language exists very early in their training. And we clearly explain that there is much, much more to learn!
5. SQL – The final lesson of PG1 is an overview of SQL. It is designed to be about 45-60 minutes.  In the survey, creating simple SQL queries was #1 overall, and SQL topics were 3 of the top 5.  Hard to argue with those results!  We want people who are familiar with SQL to learn in PG1 that they can use their SQL knowledge in combination with the other SAS syntax they have seen.  And we want people who are new to SQL to be introduced to its capabilities and motivate them to learn more.
6. Merge vs. Join – Our survey indicated that PROC SQL is more commonly used to join tables than the DATA step (SQL joins ranked #4 and MERGE ranked #18 overall).  Both are clearly important, but there just isn’t enough time in PG1 to cover both topics.  We also believe that to truly understand and appreciate the DATA step merge, you need to understand the PDV and process of execution. For this reason, we chose to include SQL INNNER JOIN in PG1, and save DATA step MERGE for PG2 when we can cover it thoroughly.  In the previous versions of PG1 and PG2, the MERGE topic was split between the two courses. Instead we felt it was important to have a single, comprehensive MERGE lesson. 
7. Arrays – Believe me, we would have loved to have kept arrays in PG2, but it just didn’t fit because of the addition of other topics that ranked higher in the survey.  Instead we are including it in our new PG3 course.  It actually fits very nicely.  Before, arrays (one and multi-dimensional) and hash tables were split between PG2 and PG3.  Now all those topics flow logically together in the new PG3 course.  I’m really excited about it!

I hope this gives you a little insight into the reason  behind the changes to our programming curriculum.  We are now writing our advanced programming courses (Programming 3, SQL and Macro) to follow the same delivery style.  We hope these 5 courses and the certification exams that correspond will produce competent and enthusiastic SAS programmers for many years! 

~Stacey

[Astounding]

Thanks, Stacey.  That helps a lot.  

I appreciate the value in asking customers what they want.  Honestly, I took the opposite approach.  Based on my experience, I included a topic if I thought customers would need it sooner rather than later.  When I got to the more complex section of my Intro to SAS curriculum (2 days), I always prefaced it with something like this:

The remaining topics are somewhat more difficult.  Depending on your experience so far, you might be saying to yourself, "At last, we're on to the good stuff!"  Or you might be saying to yourself, "Boy, I'm really in trouble now!"  If you're in that latter group, think of this as an overview of what you will be doing down the road.  At least when the time comes, you will have been exposed to it.

With that type of explanation (not always possible for an online course), all the students generally remained engaged.  That's not to say one approach is right and the other wrong, it's just how I approached course design.

Even in an introductory course, I always showed a few functions, a few software-supplied formats, creating BY variables, creating an output data set from PROC MEANS.  Of course I didn't cover all the details, just enough so students could see how it was done and what the results were. 

Looking at the questions asked here on SAS Communities boards, I see how curriculum design affects the nature of the questions.  There are far too many students trying to use macros to solve a base SAS problem.  There are many questions that would be easily solved if students knew how to create a summary data set and merge it back to the original data.    There are many questions about PROC IMPORT, including a fair number that stem from users seeing the generated DATA step code but having no idea how to change it to improve the results.  My rants aren't a criticism of the curriculum, more of different viewpoint:  sometimes you have to give people what they need, rather than what they say they want.

[Quentin]

Thanks so much @StaceySyphus !  I really appreciate your taking the time to provide all that info.  Of course, I can't say I agree with every decision, but it would be impossible for everyone to agree with everything.  Your post nicely illustrates the trade-offs we all have to make when designing a course (or a presentation, or an application), where there are always real-world constraints.  

I would have never thought to introduce SQL join before merge, but reading your post, I think that's an excellent decision for PG1.  Both to provide a doorway for established SQL-heads to start interfacing with SAS, and because you're of course correct that you need to *really* understand how the DATA step works, and the PDV, to understand MERGE.

I'm glad to hear that PG1 is going back to 3 days, which will allow a slot for INFILE/INPUT as optional material.  

Thanks again!

[StaceySyphus]

"...but it would be impossible for everyone to agree with everything."  

This revision project certainly confirmed that statement!  However, the discussions we had internally and with customers were really helpful. I learned so much myself through the process of planning and writing the courses!  Even when we did disagree on certain aspects of the course, we tried to lean more towards the data from our research and the majority opinion. 

@Quentin I really appreciated your comment about SQL join vs. DATA step merge. That definitely was a hot-button topic and perfect example of a decision we made based on the data from our customers.  If you think of PG1 and PG2 as a package, I think the flow of inner join first, then DATA step merge second works really well. 

@Astounding I also appreciate your approach giving people what they need, rather than what they say they want.  When we did our survey, we specifically targeted people who were advanced certified, since we figured they were likely more established SAS programmers and could give us a better idea of what skills they are actually using. If we would have asked programmers, "What do you want to learn?" we likely would have received very different results. Usually our brand new PG1 students are in a position where they don't know what they don't know. We hope all those advanced certified programmers together with our experienced SAS Education team have selected the basic skills our newbies need to get started and feel excited to learn more.  

~Stacey 

[Kurt_Bremser]

The way I see it, the structuring of SAS education has not kept up with the growth of the SAS System itself.

One cannot expand the scope of the software by at least a factor of 2 (or even more over the time I've been involved with SAS) and expect that teaching it can be done in the same time or even less. (Before I was "sent out into the wild", I completed 4(!) weeks of courses in Heidelberg, and that was just the kick-off to my own private learning journey)

If you want to create more than a "button-pushing monkey", you have to make it clear that learning SAS to a point where one can really make use of the features that are used in selling SAS as a superior tool takes its time, and can't be accomplished in just 4 or 5 days. Make that weeks, rather.

In the same vein, to reach a wider audience, more of the programming courses need to be made available for free e-learning. We can see the effects of "hiding" the more-than-just-the-simplest-basics in cost-intensive courses here on the communities, where we (the "old hands") provide the teaching of things to the newbies that, honestly, are stuff that should have been included in a (useful) basic education in the field.

My rant for today.