My latest DI transformation takes a JSON file as input, but there are potentially multiple files to read, from which I'm extracting just one dataset. The size of the file doesn't matter, because the JSON engine is very quick at reading even a multi-million observation file.
For ease of management, I'm calling the transformation multiple times in a job, with the output work dataset from the transformation being the JSON entry. It works, and works very well.
I previously had the JSON file as File or directory (input) option in the transformation, but with multiple transformation calls in a single job (I have reasons) I have put that file reference in as a job parameter. And that works fine. What doesn't is that I don't seem to. be able to reference that parameter from the job - I have to go into the parameter and change it instead of having it as a prompt that I can easily edit.
Interestingly when I set up the parameter, I can use Test Prompts… in the same way as I would a transformation option. I just don't seem to be able to reference it any other way. Is there something obvious I have missed?
I don't fully understand what your problem with the parameter is but... have you considered using a loop transformation for calling your "json read" transformation parameterized so you need to define the transformation only once? A loop transformation would also allow you to execute the reads in parallel.
A loop wouldn't be appropriate - each file has different metadata which the transformation acts upon. Remember the issue is the name of the json file, not the entries within it.
When I open and run the job from within DI, I want to select the JSON file, as I would if I were running it from just a transformation.
From a transformation I open it, click on Code options and select the file I want. From the job, I have to open the parameter, edit it and change the default to the file I want. There's no Code options at job level.
DI Studio is a development tool for ETL jobs. Running jobs directly out of DIS is for unit testing but your end-goal must be to deploy the jobs and run them in batch.
Whatever you develop via DIS must be fit for batch processing where there is no user interaction.
If you want something with user interaction more on an adhoc bases then use EG/SAS Studio or eventually implement a Stored Process.
I've been SASsplained! What an ignorant and patronising reply. And wrong. I've been using SAS for 45 years, and DI for twenty. I know what it can do. I know it can be used, at least as well as anybody else.
Now if you aren't going to explain to me how I can use a facility built into a SAS/DI job, which doesn't appear to me to work, I suggest you delete what you just wrote and go and SASsplain to someone else.
Secure your spot at the must-attend AI and analytics event of 2024: SAS Innovate 2024! Get ready for a jam-packed agenda featuring workshops, super demos, breakout sessions, roundtables, inspiring keynotes and incredible networking events.
Register by March 1 to snag the Early Bird rate of just $695! Don't miss out on this exclusive offer.
Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.
Find more tutorials on the SAS Users YouTube channel.