EG 7.1 / 6.1 / 5.1 issue to run one program piece by piece in sequence.

[yzhou2004]

In EG 4.2, in a case of running a long program while we like to see the intermediate results step by step, we usually submit code piece after piece in sequence.

Say, for the SAS program checkdata.sas, we submit first few steps and then submit next few steps in a separate program checkdata(2), then next few steps in checkdata(3), etc (by selecting no while we were  asked ‘do you want to replace the results from previous run which is original checkdata.sas).  In such a case, when the original checkdata was
running, checkdata(2) and checkdata(3) would be queued in the task status.

Running in sequence as above works prefect fine

1. Until EG 5.1 introduced ‘parallel’ mode which caused above sequence code in the same project space lose connections to the same work space on the server side (or rather connecting to different work space).  Then EG  5.1 kept complained data set or libname created or defined in early steps do not exsit.  That is truly annoying.

2. Then EG 6.1 seems fixed the loss connection issue above, BUT the EG 6.1 window starts getting frequently locked up (not responding to mouse clicks or whatsoever) when we submit the subsequent code.  The status bar just stucks at ‘connecting to server……’.  Usually the EG will come back alive after quite a while.  But this makes our sequent runs impossible.  We have to wait until the window coming back alive (we don’t know when) to switch between programs and submit the subsequent code, or even do anything else within EG.

I wish EG 7.1 would solve EG 6.1 window lock up issue, but apparently it has not yet.

Did anyone have the similiar issue and confirm there is a fix or not?  Thanks much. 

[jakarman]

Your working method is correct but technically incorrect.
Correct: developing your code in small independent pieces. (Structure programming, object oriented, Nassi–Shneiderman diagram - Wikipedia, the free encyclopedia)

Incorrect: you are obviously using the SASWork for intermediate files.

        Do not do that use a dedicated semi-permanent storage area (staging?) for that.
Incorrect: you are obviously not using the FLOWS in Eguide for dependicies. They are designed for that use those for that goal.

       When going for grid/scheduling the same logic with those source code be used although there is a conversion of tools needed.

      With neither version of EGuide this should be problematic, also parallel code can be used with all advantages.

Incorrect: replace the precious result is only about the sas-log/results in your EGuide project not about the processing of data.

     You stored logging.results can have grown too big.

Problem Configuration/Installation the connection to server long wait is a signal there is an issue on that level. Either limit on usage have been set to prevent you to work or more logical the OS settings are somehow having got generic settings the work is not capable to give the needed service. What OS level? who is responsible for the installation/configuration?      

[yzhou2004]

Hi Jaap, thanks much for your reply.  But I feel your answer is way beyond my level.  I have software engineer background in many other modern programming laungages (C/C++/C#/JAVA) for many years before I start to work with SAS.  I have to say SAS is least intunitve (not mentioned anti-OOP) language I have ever used.

The case I described above was intentional to use SASWork to run some data analyses.  I do NOT want to keep all those data sets in any (semi-) permenant storage at all once the process is done.  I don't think I need to use FLOWs for the dependencies because I am running code step after step.  FLOWs is just conceputally over kill here.  Also, I know 'replace previous results' won't replace the data sets but logs/results instead.  But my sole intention to run code in steps is to keep logs/results spit out in steps on the go.  I don't want to see a huge lum-sum logs/results in a single page.

In your last paragraph, I assume you are saying some installation issue causing the long wait to connect to the server.  I don't understand 'what OS level' question.  But I am running EG on Windows 7 while the server was installed on LINUX.  To your last question, the person who configured and installed the SAS server platform is someone pretty senior who our company hired from SAS a few years ago.

Thanks much for your input again

[SASKiwi]

I suggest you check to see if you have the project log option switched on in EG. If it is switched on, turn it off. I've found having a big project log or even any project log can cause EG to "lock up" for significant amounts of time, and when I turned it off it has reduced this problem.

[jakarman]

yzhou2004 software engineer - system programmer background that is a small step but a big difference.  The difference is mostly on thinking in the different layers that are there.

Flow overkill? Surely not.  See your code block in EGuide as some objects in a flow.

The flow-processing is the interaction between those objects. Every code object:

- can have input (prompts)

- running on a server( there could be more of those)

- having advance dependicies  as of other events.

- by using task you get other types aside code block into this collection.

At that level it is OOP no not at the program coding level but that is the engineering into architecting background switch.

The SASWORK is session bound the same way as you can see the temporary tables being used with a RDBMS. They are  lost after the session is closed has ended.

That is why you see the staging area as a standard concept with ETL processing and dwh concepts. They have to build it in simple verifiable steps. Do the same with SAS and you don not have an issue with that. I have to  agree these engineering/design skills are mostly lacking with SAS users and SAS consultants. Sometimes I am getting the feeling they have learned some spreadsheet (excel) and want to do the same in SAS. Sales promoting proudness of not IT involvement needed.

With the OS is am meaning the Operating System. Eguide is running at your desktop and Windows 7 is the OS at the desktop. Your SAS processing is done at a Linux Operating System.

What happens if you want do some work? Login with Eguide is connecting to the SAS metadataserver. The metadataserver is using an in memory database having your user being defined. That one must be found. When found it will validate you user at the OS level using low-level OS  (root calls). When succesfull it will start an Unix session for your is switch the  user context and pass that for a port-port process with Eguide.  When you are getting "connecting to server" messages something in that process is far too slow or got blocked.
You can get external IT-staff for that to analyze but they are needing to right skillset to understand all issues from start of bare iron virtualization Os tuning and so on. There are very good papers at SAS for that. The best info to start search for margaret crevar http://blogs.sas.com/content/sgf/2015/02/25/configuring-storage-for-sas-new-guidelines-available/
Your environment is setup by an external guy hired from SAS? What is with the daily support monitoring performance&tuning and more of those on going tasks? This could be one of the root-causes not being aligned with in house IT-support. It could have been installed for a sort time goal up and running to show but not acceptable for mainstream usage.  (setup klik klik   SAS tells us it should be this way, shut up with difficult questions of compliancy).  Check your situation and issues with you local IT support.  
  

System programming YStem

[yzhou2004]

Jaap, I prefectly understand what you meant above about system programers vs software programmers.  I had experience in the both area for years.  The projects I worked in my earlier career were mostly client/server distribution systems on both windows and LINUX.

But my SAS code in trouble is a set of simple procudures (proc contents, proc freq, proc means and proc compare) on two huge data sets in SPDS libraries.  The reason I want to submit the code in sequential pieces is each simple procedure could take some time because of the data set size is huge (hundred of million records with 100+ columns).  I would like to see the results/logs at each step in case something wrong so that I don't have to waste time to run the whole thing.  There is nothing fancier than this.  That's why I feel the flows is an over kill in this case.

Thanks for your suggestion on contacting local IT support.  Yes, the guy worked for SAS for years before our bank hired him as in house SAS administrator 5 years ago.  The issue could easily due to configuration.  But I don't understand is why EG 5.1 has 'disconnect to SAS work space' issue but not 'lock up' issue, while EG 6.1/7.1 has 'lock up' issue but not 'disconnection' issue. I suspect there is something different/inconsistent on EG clients too.  I installed EG 5.1/6.1/7.1 on my computer and did not do any specific configurations after pretty much straightforward installation, but they performed in different ways.  Again, EG 4.2 never gave me such a trouble.  So I suspect this could be a case of fixing one bug but introducing another at new EG releases.

[jakarman]

yzhou2004, nice you have experience on both sides. What I am missing is experience with that including schedulers at the system programmers area. With Mainframes these are almost a pre-req to do (opc ca7 etc). The whole goal of using schedulers (job application flows) is seeing results/logs at each step so you can correct and restart easily parts without wasting too much of time. Going into testing (milo/euro) not the scheduling by triggers-time were important but running the same programs over and over again with different data to eliminate errors. That is automated testing, again flows.
For fun I build an automated tool version  and releasemanagement included. That was far before ISTQB did exist. Test Automation Engineering - ISTQB® International Software Testing Qualifications Board .  That was a long time ago (25+trs). At the moment I am building something like but now fully focussed on SAS and build in SAS it is needed by the user groups. (They have Eguide by the way).    

>>> Flows are not an overkill in contrary they are one of the basic building blocks. Change your mind and use them.

With Eguide flows, you can still run then individually, but also as a logical series (flow). It will stop the flow when some code is incorrect. You still have all results/log of each step.  

Huge dataset SPDS 100M 100+ columns look to me normal sized datasets. Time is changing and the sizing matters  is just what you are seeing form day to day work.
Working at a bank and installing software on your own desktop. That is a bad signal as desktop in those environments are normally closed. Only when following some release/test processes you get those updated. It looks you are in some isolated area. The guy responsible for your SAS installation should have tested and validated this all including performance and UAT testing. Never mind there are a lot of possible pitfalls at the system programming level on the Windows and Linux that are far beyond the SDW setup approach.  Eg locking on files can behave different that is no real bug but a misfit of the in house desktop policies in house Linux policies and the SAS SDW ideas. That misfit of thinking by the way is a major design bug of the SDW.