Desktop productivity for business analysts and programmers

Poll: EG for programmers or not?

Reply
PROC Star
Posts: 1,673

Poll: EG for programmers or not?

Hi,

I am unsure this is the right place for this, but I am just back from the SAS Forum in Sydney and am all wound up again. So I am curious to see what you guys think.

=>What on earth pushes sas to want programmers to use EG?

=>How can they believe that populations as different as end users and coders have similar needs and should use the same tool?

EG's code window is cramped, and it doesn't even have line numbers for goodness' sake! How am I supposed to efficiently fit my 1000- or 2000- lines programs in that tiny crippled space?


While it is very nice to be able to see a process flow, and EG does offer some pluses, it has many shortcomings over the DMS, which is not that great to start with.

To name some of the most obvious that came to mind, in no particular order (this might have marginally improved in the latest version of EG I haven't kept track):

-Data step debugger disabled (it should be overhauled instead)

-Either windows are way too small (especially table-browsing, code panes) due to more interface clutter, or impractical toggling of window size with ctrl-M

-Dataset-browsing pane: cannot run WHERE clauses or run SHOW command or hide/unhide variables or display one record at a time.

-Cannot toggle easily from 'Display Labels' to 'Display Column Names' (need to go to General Options) or change variable format

-Dataset Properties window is not resizeable and is tiny and does not indicate if dataset is sorted or what the index is or number of lines and columns or compression or date

-Cannot copy column names from properties/columns dialog (Drag'n'Drop would be even better)

-Library property does not indicate path

-No line numbering in code window

-Log window full of EG-related garbage: one-line statement submitted results in a 1-page log.

-Catalogs are hidden

-Cannot configure keys

-Server list pane not split explorer-like (libraries | tables)

-No access to OS (call system, x, %sysexec, pipes, named pipes) (so no MPconnect piping optimisation)

-and on and on


=>Why cripple an already limited coding interface? How can these shortcomings be seen as acceptable? Do sas Institute developpers use EG?


In the 2009 sasware ballot (I haven't looked at the 2010 one, nor have I bothered to check whether any of the requested features have been implemented), 3 of the top 4 *overall* items relate to application development and debugging:

1- assign line numbers to code seen in a log generated by a macro in order to correlate with any LINE:COL messages given at the bottom of a DATA Step

3- provide an option that enables you to turn off or on certain notes that are written to the SAS log; for example, notes about data set compression, invalid data, or invalid argument to a function

4- provide the functionality to stop a SAS process in interactive mode exactly as the ERRORABEND option stops a process in batch mode - immediately stop the process upon error, produce a log, and not terminate the session



How that kind of wishes fits with EG I can't imagine.

On the other hand, the DMS is outdated too, lacks many features users have requested for years (like the ballot's macro-line log-numbering/debugging), and also lacks tools other IDEs have had for many years, like good color coding, predictive typing, drag and drop debugging, source code management (both to retrieve older version of the code and to share code), etc.

So I agree the DMS needs to go, and why not a java applet connected to a server if that is the way sas wants things to go. But not an applet with even fewer features than the poor old DMS!

My opinion is that the "SAS Enterprise IDE" java interface is long overdue, that coders need it badly, and that sas is on the wrong track trying to push them to use EG. I might not be alone thinking that seeing that sas is still pushing many years later.


=>What do you think?

So 4 sets of questions here (including for sas). I am curious to see comments/reactions. Thanks.
Super User
Posts: 3,235

Re: Poll: EG for programmers or not?

I don't use EG much so I won't comment on editor differences between EG and DMS.

What I would like to say is that where I work, we continue to use full SAS on our desktops even though we primarily use a Windows SAS server. Why is this? One of the important reasons is that there is NO DR (Disaster Recovery) plan for our SAS server. If for any reason it is not available we retain the capability of running our business-critical processes from our desktops (after our data is restored on a file server) so at least we have some minimal DR/backup capability. All our programs are written with this flexibility in mind. I wonder how many SAS sites are in the same boat? If your SAS server has no DR or backup and you only use EG - tough, you have no alternative!

BTW I use DMS line numbers all the time as a very useful productivity aid.
Trusted Advisor
Posts: 2,114

Re: Poll: EG for programmers or not?

Chris,

Your "poll" reads more like a "rant."

I've never used the DMS editor and use the EG editor mostly for viewing. Having grown up in TSO and then UNIX, I became comfortable with external editors and never bothered to switch. I now use EMACS on both the Unix and the PC for primary editing. I run mostly in batch, even in the EG interface, due to large data volume.

EG 4.3 does seem to have lot of changes to help programmer productivity. With any luck we'll be able to get our hands on it in a week or so.

One of the features that I find really helpful in EGuide is the process flow. The ability to link things together and to run the process flows is a great benefit. We have about 20 SAS programmers in our shop (and another 40 statisticians), so the documentation that it provides can be very helpful. Both you and SASKiwi likely have good ways to document your work, but the a hand-off between the two of you might involve a fair learning curve on the process.

SASKiwi's challenge is really a local decision. Disaster recovery processes are in the realm of risk-based management decisions that, of necessity, also include a cost-benefit analysis.

Doc Muhlbaier
Duke
PROC Star
Posts: 1,673

Re: Poll: EG for programmers or not?

Yeah, Duke, maybe it is a rant, but I feel strongly about it, and I made clear I was expressing my opinion and was asking for feedback.

The messenger (me) is not what matters here in any case, let's discuss the message.

The points you make about the message are valid, but limited:
- yes seeing the process flow very good, and the hand-over will be easier with EG
- yes the DMS is better than emacs or vi. But how good an environment is that? JimBaillie's Eclipse or visual studio environment examples are worlds ahead. How much productivity would you gain by having an efficient IDE?

My other points stand: EG lacks these useful, no, fundamental programming features, not to mention those available in modern IDEs

I am starting to have a feeling that *maybe* the difference between coders satisfied with EG and those not possibly satisfied might be down to the complexity of the code (please don't take this personally, it is just an hypothesis).

My n000-line code typically :
- merges several years of monthly detail tables in a view (deriving things like how many of x happens in the next y months after the current month),
- summarises the view,
- adds various bit of infomation,
- then *for each* line of the summary table:
- more data can optionally be retrieved,
- annotation data is created for each graph
- tooltip data is added for each graph
- a bunch of graphics are created in an html page with funny names for the pages and the individual image files, which also have to be derived before-hand.

All this is encapsulated and controlled by macros defining date spans, table versions and the such. The code is modular in the sense that i try to put each processing step (say graphA with its specific data transformations, axes and legend parameter definition, annotate and tooltip creation, then the final plots) in a macro, in turn called from higher-level macros.

But the whole reporting program is one whole, and cannot meaningfully be broken into independent bits. And as you can imagine, it takes time before it runs successfully.

If you know in advance what data you are going to process, what parameters you will delimit it with or break it down by, don't need macro code to process special cases, don't create annotations and the such, don't access the OS, don't need to debug complex data steps at times, don't care about combining steps to speed up the overall process, then EG might be workable or even better than the DMS.

In my case, EG would drive me mad (and halve and more my productivity) with all the hoops it would put in my way just to do extremely basic stuff like display only a couple of a dataset's variables with a where clause applied, or find where the hell my graphA macro is in the program, plus all the other things I mentionned.
Community Manager
Posts: 2,889

Re: Poll: EG for programmers or not?

Chris,

It was good to meet you in person at the SAS Forum in Australia. I was glad for the opportunity to speak with you about your concerns, even though I realize that you have yet to decide to add SAS Enterprise Guide to your own personal SAS toolbox. The format of SAS Forum in Sydney did not allow me to show a demo. Did you take advantage of the hands-on EG 4.3 session provided by SAS Education during the conference?

I've had the chance to speak to hundreds of SAS users here in the region (and I'll visit many more in the coming week). Most are very excited about the new features that are coming in SAS Enterprise Guide, especially for programmers.

It is a different environment than what you are accustomed to, and I know from our previous discussions that you would prefer an Eclipse-based solution. However, that is not the direction that we're taking SAS Enterprise Guide, whose primary audience includes business analysts, as well as programmers and statisticians.

EG does support line numbers, of course, as well as many other standard program editor type features. You can explore these in the Tools->Editor Options and also the Editor macros/commands features. You can easily maximize the EG program window with Ctrl+M to increase your workspace.

You do present some valuable ideas for additional features that SAS programmers can use to be more productive, and I appreciate that. Please keep the constructive suggestions coming.

Still Down Under,
Chris
PROC Star
Posts: 1,673

Re: Poll: EG for programmers or not?

Hi Chris,

Nice talking to you indeed. I spent less than 20 hours in SYD and had to leave early to be back in AKL on Friday, and I only managed to use Add-on for office and dataflux, so I still have to catch up on some of the upcoming EG features.

Do you think my hunch that EG is good if the processes involve fixed input data, few changing parameters, and a fixed sequence of procedures and sql?
And if the power of macros is required (either for variability/parameterisation or automation), if data steps are used (and all the efficiency that comes with them: multiple output datasets, hash tables, arrays, etc), or some other features like OS access or MP connect, then EG is very limiting.

Note that this does not describe the comfort of doing development-related things like checking the data in a table or looking up a catalog's contents.
Community Manager
Posts: 2,889

Re: Poll: EG for programmers or not?

Chris @ NZ,

None of the capabilities that you mention are inherently limited in SAS Enterprise Guide. We have many customers who use EG with Grid Computing (MP Connect), use parameters/prompts with programs, and access all of the power of the DATA step via their own programs.

I've published blog posts that describe how to enable system commands and view SAS catalogs using SAS Enterprise Guide; it requires just a few tweaks to get to these features, which aren't needed by everyone but that might be missed by long-time display manager users.

Chris @ SAS
Occasional Contributor
Posts: 8

Re: Poll: EG for programmers or not?

> I've published blog posts that describe how to enable
> system commands and view SAS catalogs using SAS
> Enterprise Guide; it requires just a few tweaks to
> get to these features, which aren't needed by
> everyone but that might be missed by long-time
> display manager users.
>
> Chris @ SAS


Do you have a direct link to that, please?
Respected Advisor
Posts: 4,135

Re: Poll: EG for programmers or not?

Chris

When I first read your thread I thought: O.K! I can understand you in a way but isn't EG also giving us a lot and it's only a question of getting used to the tool?

Since then I had for the first time to use EG 4.2 seriously as a development tool and I'm getting more and more annoyed with it.

Today for example I wasted at least 2 hours investigating an issue where I just couldn't understand what’s going wrong (the log only said that there is an error and listed records) - and then I finally found the answer: http://support.sas.com/kb/18/103.html
Just great! EG adds it’s bits and pieces to my code and stuffs it up and there is no option to just turn off all this unwanted.....

I’m also always really “happy” when I run a piece of code and then make the mistake to click on the OUTPUT tab.
It might be great to look at an output data set and check the data – it’s less fun if the first of these data sets is actually a rather CPU intensive view which then gets executed.

My list of frustrations gets longer and longer on a daily basis and I can’t wait to see EG4.3 hoping that at least some of the issues are resolved.

So Chris@NZ: If you are the next time in Sydney let me know! Another beer is awaiting you.


To add also something constructive:
What I would like to see as a developer is a tool which helps me to develop and test code.
The EG editor window recognises data and proc step boundaries (this plus and minus signs on the side). It would be great if this would translate into a program flow similar to DIS4.2.
If I’m working on a piece of code then I’m not interested to see all the code I’ve developed and saved in a project. I’m only interested in this one piece of code. But it would be great to have a program flow window which breaks up this single code node. So after executing the code I wouldn’t just see another code node in the project window with 20 output tables – I would see a dynamically generated program flow in a program window where each data and proc step represents a node (with tables and views in between). So in the end something very similar to DIS4.2 only that the flow is dynamically generated instead of static.

And then following what DIS4.2 does: I would like to see which node threw a warning/error, and I would like an option to directly jump into the point in the code and log where the issue occured.
I would like to be able to change the code in this node and then to have the option to re-run only this node – or only up to this node – or only from this node downstream. And I also would like to have an option to step through the data/proc step in the way (or even enhanced) PC SAS gives me (with break points and everything).

And thinking further: When there is an issue with a node and I select it I can press F1 and I’m getting already a selection of SAS notes and doc entries related to the issue/commands used.

Things like this would help me a lot to start liking EG. So far EG still feels a lot like something created for analysts and not for developers.

Cheers
Patrick
Occasional Contributor
Posts: 8

Re: Poll: EG for programmers or not?

Patrick, that's a great vision for a dynamic program editor. I have had a similar idea about a mashup of googlemaps+streetview and sas coding. Imagine the ease of use in googlemaps, where I can drag the map around and zoom in at will with easy scroll wheel. If I end up zooming in really close, it switches over to streetview and lets me see the house.

If I need to read and understand some big sas program (either my own, or someone else's), I print it out on A3 paper with color coded syntax, and then I start drawing blocks around datasteps and procs, and connecting them with lines. Then I put them all over the table, and stand up to get an overview.

This manual process would be solved by your idea.
PROC Star
Posts: 1,673

Re: Poll: EG for programmers or not?

Wow, good idea. This would be really useful: zoom in and out of the code as needed. Why settle for VS features when we can have much better?



I also propose that we can set an option for the number of levels shown for the collapse/expand line in the program editor. So 3 three levels would show the do block in a data step, the data step boundaries, and the outer macro start and end lines.



Now, with all these fantastic contributions and ideas, the challenge is on sas' side to deliver the best IDE. Smiley Happy



> So far EG still feels a lot like something created for analysts and not for developers.



That's because that is the case.

Plainly.



> So Chris@NZ: If you are the next time in Sydney let me know! Another beer is awaiting you.



Mmm, I think I can see through you guys' plan now. You want to silence me for some reason. You want to take me out. I am not paranoid, but you must be afraid or envious or something. Of the competition? Of the fantastic discussion generated here? And you want me to suffer agonising pain as my liver slowly shuts down. Well, too bad, I uncovered your plan, so you'll have to think of something else now. Fyi, I am very partial to dark chocolate and durian...
=;o)
N/A
Posts: 0

Re: Poll: EG for programmers or not?

Chris -
The biggest disadvantage that EG has over DM is that it is NOT actually being run as part of the SAS system. So all of the information about the environment that you have setup cannot easily be passed back and forth between the programmer and the executing SAS code. Similarly the need to use .NET or other totally different programming languages to manipulate the EG environment makes it a very steep learning curve. In DM you can make command line macros using the same macro programming tools you use to make macros for executing SAS statements. In DM you can make SAS/AF programs interact with users and your data using the same SAS functions that you use in your data steps and macros.
This disconnect between the tools/skills needed to use the programming environment and those needed to actually create productive SAS jobs makes it much more difficult to become productive in EG. It is hard enough finding skilled SAS programmers that understand both SAS and our particular application area (banking, medical research, statistics) but to also have to have .NET and other programming skills just narrows the applicant pool even more.
- Tom
Frequent Contributor
Posts: 91

Re: Poll: EG for programmers or not?

Tom,

I could not disagree with you more. In SAS/AF, you are forced to learn a new programming language (SCL) which cannot be used for any other application. In fact, SCL uses programming concepts very similar to JAVA, and other object orientated languages that .NET uses. If you are going to learn SCL, why not learn vb.net or cs.net? All you are doing with .NET is wrapping your normal SAS code up into user friendly windows, whilst also allowing you access to other objects SAS EG uses to provide a rich and more powerful environment that SAS/AF. And perhaps you dont realise that .NET in fact does interact with users and data using the same SAS function that you use in your data steps and macros - the .NET programming bit is for displaying the GUI and accessing other objects, which is exactly what SAS/AF does.
So the fact that EG is not actually being run as part of the SAS system gives you the best of both words. You leverage the statistical analytics of SAS, and use .NET, which is a powerful, transferable and ever evolving platform to access these SAS methods. Why would companies continue to invest in SAS/AF when they could hire a .NET developer (easy to find), to work with their SAS programmers to develop custom made applications that blow out of the water anything that SAS/AF can and will ever possibly be able to do?


Nick
Frequent Contributor
Posts: 96

Re: Poll: EG for programmers or not?

.NET is not required to use EG. it is only needed if EG is extended and it isn't that hard to do IMO. It does require some skillset in .NET but there are many, many EG coders who will never have a need to extend the environment.

When EG 4.3 comes out, I will switch over. There are simply too many new features to stick with DMS and I have disliked DMS for years.

When i moved from ISPF to the PC, I felt lost at first. However, after a period of time, I couldn't see myself back in ISPF unless it was required.

Change is hard but there is no reason whatsoever for SAS to continue to invest in AF based technologies. What might be useful is SCL to C#/Java convertors if it is possible.
Ask a Question
Discussion stats