Just re-reading your post, I only talked about DIS in my reply. Data Management Studio and Server are a separate discussion, albeit that Metadata and DIS are involved as well.   Which software are you hoping to use, and what for?   Actually, a better question - what are you hoping to achieve? ... View more

So, if you’ve got EG and MDS working, then you’re most of the way there.   Now, it should noted that DI Studio is a discrete product, licensed separately from EG, and MDS. These days, it's usually found as part of the Data Management bundles, Standard or Advanced, though I think it's also present in other packages as well.   Assuming you’re working against an appropriate licence, all you then need is to install the DI Studio application on your PC, and connect to the Metadata Server by creating a profile (connection details). Assuming that works, which it should from the sounds of things, and you don’t get an error saying “metadata not configured for DI Studio” (which usually means you’re not working with a correct licence, or somehow the SAS server was installed in an odd way), you’ve got the basics sorted.   Now, one key thing to know about DIS as compared to EG; you are only ever really working on the server, your PC is not really involved at all. So, unlike EG, you cannot access or work with files on your PC. Given you’re using a Linux Server, that may also mean that you can’t work with files on any Windows file servers, unless they’re connected appropriately to the Linux server in a way that's been communicated to you by whoever looks after that sort of stuff for you. (Linux to Windows connectivity is a whole topic in its own right)    In terms of documentation etc, there’s quite a lot on the SAS website to the best of my knowledge, and of course SAS run DI Studio training courses. These are great, in terms of the basics of using the tool, features and functions, navigating the menus, creating jobs etc, but using it in anger, for real world tasks, good and bad patterns, actual practical usage is a bit harder to come by.   My own “DI Studio WORST practice” presentation is posted nearby, and does give some insights into this sort of thing, but I was to some degree “preaching to the converted” (quickly) and I only had ~ 25 minutes. ... View more

The way to supply a filename to a file reader is in the external file definition. If it varies, use a parameter on the file object. There is hardly ever a good reason to override code in a node in DIS. ... View more

Don't put ANY expressions in a file reader, defer that to an extract node later in the job.   File Readers read, Table Loader load, neither should IMHO contain any "logic". ... View more

IHMO the BEST way to do this is to create a generic "Move Files" User Transform, as a reusable node in your "toolbox", rather than even consider using user written code nodes.   Couple this with other UTs for taking directory listing etc and you can create some VERY powerful raw file handling frameworks, that will scale to hundreds of differentiated sources, and 100s of thousands of raw files, and will also keep your GDPR/Compliance/Audit people happy.   Actually, the tracking of files from "unprocessed", to "being processed" and then "been processed" is a pretty common pattern.   You can tackle it in a number of ways, from actually moving or renaming the files after they have been processed, or creating a "tracking table" that has a record for each file and a status for each, which hosts the processing status value. "Intrinsic tracking" or "external tracking". The advantage of intrinsic tracking is, well that it's intrinsic. The advantage of external tracking is that you can that it's more flexible and extensible, and that you can run multiple discrete instances off the same physical file system, say to support Dev/Test/Live, or Lev1/Lev1, without duplicating the files.   I've build a number of these frameworks in the past, and I'm a massive fan of decoupling - so rather than have the archiving in-line with the processing, I prefer to defer it to a housekeeping job. In this pattern, ingest flows transition files through the processing stages (above), and a later point the housekeeping job(s) transition the "been processed" files to "archived" including physically moving them. Later on after that, archived files are aged off, and deleted or zipped or moved to even cheaper storage, but their details are keep in the tracking tables for audit purposes. Kind of like HSM for raw data files.   I gave a talk this week at the SAS UK Forum on advanced scheduling techniques that leverage this pattern, as frankly the triggering in LSF is far too crude and unsophisticated. Look for a post soon with the slide deck from that talk.   Using the file tracking approach, powered by write once, use again and again file handling user transforms, and the scheduling techniques, and you can build raw file handling and ingest frame works that will cope with hundreds of differentiated data sources, and keep your GDPR/compliance/audit people happy as well. ... View more

So, in a nutshell, a compute server needs to be running Object Spawner, and be properly configured for running batch processes assuming you’re doing to be using those (which is pretty much the raisin d’etre of DIS) as well as being configured via Metadata Server, as well LSF four scheduling and orchestration.   This isn’t so much an issue of deciding what to run, more one of planning, installing and configuring an environment, usually at install time.     Perhaps by way of clarification, I have a “self-contained” Data Integration Studio environment on my laptop, which means I’m running Metadata Server, Object Spawner and LSF on it along slide other SAS services for things like Data Management Server etc. With this I can quite happily use DIS on my laptop to work against the metadata server, as well as against remote ones, by switching between different profiles.   This also means that Enterpise Guide on my PC can talk to ”local” SAS directly, and to the local “compute server” that’s effectively confoigured on my PC as well, via the local metadata server. (That’s not really a normal configuration at all, as Metadata Server isn’t normally available for use on PCs rather than servers. It’s not really a configuration that makes sense in a “real” environments, it’s more or less restricted to training, R&D and demonstration use cases. ... View more

So, DI Studio absolutely opens multiple SAS instances on the server(s) it's running on, and no just for each job window. Additional ones get spawned for things like view tables directly from the folders etc.   Enterprise Guide uses just one, with a single work library (and physical directory).   The being able to determine the physical directory via %sysfunc(pathname(work)) can be very useful, but watch out if you're on a grid, as you could end up on multiple different servers.   There are a range of macro variables, and sysfunc calls you can utilise to find out the server you're on, the PID of the SAS process you're using etc, which open up a realm of possibilities, depending on your environment. ... View more