BookmarkSubscribeRSS Feed
LaurieF
Barite | Level 11

My latest DI transformation takes a JSON file as input, but there are potentially multiple files to read, from which I'm extracting just one dataset. The size of the file doesn't matter, because the JSON engine is very quick at reading even a multi-million observation file.

 

For ease of management, I'm calling the transformation multiple times in a job, with the output work dataset from the transformation being the JSON entry. It works, and works very well.

 

I previously had the JSON file as File or directory (input) option in the transformation, but with multiple transformation calls in a single job (I have reasons) I have put that file reference in as a job parameter. And that works fine. What doesn't is that I don't seem to. be able to reference that parameter from the job - I have to go into the parameter and change it instead of having it as a prompt that I can easily edit.

 

Interestingly when I set up the parameter, I can use Test Prompts… in the same way as I would a transformation option. I just don't seem to be able to reference it any other way. Is there something obvious I have missed?

5 REPLIES 5
Patrick
Opal | Level 21

I don't fully understand what your problem with the parameter is but... have you considered using a loop transformation for calling your "json read" transformation parameterized so you need to define the transformation only once? A loop transformation would also allow you to execute the reads in parallel.

LaurieF
Barite | Level 11

A loop wouldn't be appropriate - each file has different metadata which the transformation acts upon. Remember the issue is the name of the json file, not the entries within it.

 

When I open and run the job from within DI, I want to select the JSON file, as I would if I were running it from just a transformation. 

 

From a transformation I open it, click on Code options and select the file I want. From the job, I have to open the parameter, edit it and change the default to the file I want. There's no Code options at job level.

Patrick
Opal | Level 21

DI Studio is a development tool for ETL jobs. Running jobs directly out of DIS is for unit testing but your end-goal must be to deploy the jobs and run them in batch.

Whatever you develop via DIS must be fit for batch processing where there is no user interaction.

 

If you want something with user interaction more on an adhoc bases then use EG/SAS Studio or eventually implement a Stored Process.

LaurieF
Barite | Level 11

I've been SASsplained! What an ignorant and patronising reply. And wrong. I've been using SAS for 45 years, and DI for twenty. I know  what it can do. I know it can be used, at least as well as anybody else.

 

Now if you aren't going to explain to me how I can use a facility built into a SAS/DI job, which doesn't appear to me to work, I suggest you delete what you just wrote and go and SASsplain to someone else.

Patrick
Opal | Level 21

It's sometimes really hard to tell on which expertise level someone is just based on a few sentences in a forum. Sorry if I got yours wrong. 

 

Patrick_0-1687766157413.png

May-be I misunderstand but wouldn't that just be a one-time config? So why is this an issue?

 

sas-innovate-2024.png

 

Secure your spot at the must-attend AI and analytics event of 2024: SAS Innovate 2024! Get ready for a jam-packed agenda featuring workshops, super demos, breakout sessions, roundtables, inspiring keynotes and incredible networking events.

 

Register by March 1 to snag the Early Bird rate of just $695! Don't miss out on this exclusive offer. 

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 5 replies
  • 593 views
  • 2 likes
  • 2 in conversation