Dataflux 2.8 - create a log file

[beamer108]

I have a long running job that can process up to a million rows or more.  I have created a log file for the script that runs the job but it just shows which data process it is in which makes it difficult to determine if the job is running or hanging.  It can sit on the one workflow for several hours and I am not sure if its still processing or not . I would like to create a log file that gets produced and shows the number of records its received so I can see the progress it is making while the job is running.  Im assuming this is done in an expression node and uses maybe the print function or logmessage function?  I've tried a couple things but realizing I really have no idea how to accomplish this. Please help.  Thank you in advance.

[SASKiwi]

If you are running this in DataFlux Management Studio you can easily open Windows Task Manager and see the CPU usage of DataFlux.

I'm not aware of any way for your log to include a row progress counter.

[VincentRejany]

Hi

You can configure job level logging, but mind that it will affect performance negatively.

Have a look to the "unofficial" document attached, it gives few tips for configuring and using log files within the SAS DataFlux Data Management platform.

[audrey]

Hello,

I would first make sure the job log is enabled. For Data Management Studio, that would be in dfwfproc.log.xml (look for the Additional Process Logging section). Once this is enabled, it will be default create a log for each of your job run. This log might already be enough for your needs.

If it not, you can indeed add an Expression Node in your job and use a code like this:

//Pre-processing Expression
integer rowcount
rowcount = 0

// Expression
rowcount = rowcount + 1
logmessage("This is processing Row Number " & rowcount)

This will add lines in your dfwfproc log, similar to:

2021-08-05T09:46:12,808 [00000189] DEBUG DF.StepEngine.Page - execute (Using logmessage.ddf)
2021-08-05T09:46:12,808 [00000189] INFO DF.StepEngine.Plugin.Expression - This is processing Row Number1
2021-08-05T09:46:12,808 [00000189] INFO DF.StepEngine.Plugin.Expression - This is processing Row Number2

Hope this helps.

Audrey

[beamer108]

Audrey - sorry just getting back to this issue. Can the expression be created within the datajob or does it have to be in the process job?  I created a counter expression node within the datajob where I'd like to see the progress of record ingestion but the log file on the server just shows me the name of each node it processed and 100% completed. That is when I thought maybe its part of the process job and not within the data job where I need it. 

Thanks
Carrie

[audrey]

Hi Carrie,

I hope you are doing well.

It can be created in a Data Job, it doesn't have to be in a Process Job. See this very simple example I created:

Did you make sure the Process Logging is enabled in the configuration file?

You might want to send your configuration file if you are not sure.

Thanks,

Audrey

[beamer108]

Hi Audrey 

I've attached the dfwproc file, it doesn't look like we have it turned on, but Im not exactly sure when it says Remove this line - how much of that line I am supposed to remove.  And will this slow down processing time for all our  jobs or only the jobs that contain an expression node that writes the counts to the log?

[audrey]

I made the modifications to your file (the entire line needs to be deleted).

It will have a small impact on the performance, for all the jobs - we haven't enabled a DEBUG or TRACE level of logging, so the impact should be minimal.

Let me know if this helps.

Audrey