Slow performance of Alert Generation Process (AGP)

[Reem97]

The fcf_replicate_transactions execution time within the alert generation process has surpassed the usual duration, extending beyond 10 hours and beyond.

Steps Taken to Address the Issue:

Troubleshooting Measures:
1. Examined AGP logs to identify anomalies or errors during processing.
2. Compared logs with regular executions, where the replication step typically completes within 10 minutes. Investigated cases where the duration extends to nearly 12 hours.
3. Conducted a comprehensive analysis of the FCFKC and FCFCORE tables using the fcf_analyze macro.
4. Checked the database server resources, increased redo log files, and added indexes for tables specifically utilized by AGP in the replication process.

Outcome:
Despite the undertaken measures, AGP remains stuck and continues to consume an excessive amount of time.

[SASKiwi]

Firstly, in what SAS product is this issue happening? You've posted this in the wrong Community category and you'll get a better response if it is moved to the right one which depends on the product.

Secondly, I'm guessing this is happening in a SAS off-the-shelf solution and opening a Tech Support track is the best option here if you haven't done so already.

[Patrick]

@Reem97 If you post questions in the "correct" community then you've got a bigger chance that the people with the specialty solution knowledge get aware of it. I believe your question should go to https://communities.sas.com/t5/Fraud-AML-and-Security/bd-p/fraud 

@Kurt_Bremser: Can you move this post?

As for your actual question:

Comparing the logs between a run with normal runtimes and one with overlong runtimes should at least tell you which steps differ significantly.

Analyzing a whole set of logs should also allow to tell how data volumes and runtimes relate to each other - like: Is there some data volume threshold where runtimes increase exponentially or is it not directly related to data volumes? Did runtimes for similar data volumes increase over time significantly? etc.

Analyzing the logs should also tell you if the overlong running steps are some that interact with the database (Oracle?).

If it's SAS waiting for DB where things "hang" then you need likely also to talk to a DBA - or IT infrastructure if it's the network.

Investigating performance issues, especially when they are "random", normally requires a deep-dive into the details - details which you likely can't share with us publicly.

If you can't get to the bottom of things then I suggest that you contact SAS Tech Support directly - SAS Technical Support <support@sas.com>

[Kurt_Bremser]

For now, I have moved this to Admin & Deployment, but if you tell us in which particular SAS application this happens, I could move it to that respective community.

But you need to open a ticket with SAS technical support in the first place. We could only make educated guesses once we see logs of normal and overlong executions to compare.

[Patrick]

@Kurt_Bremser  AGP stands for Alert Generation Process which is part of the SAS AML solution.

[FMinago]

Hi

You did not mention what database your AML (i believe 7.1) is based on, but I'm curious if you are utilizing the bulkload option to facilitate faster loadings?  

fcf_replicate_transactions macro primarily loads transactions of alerted entities into the KC database. Considering this, I presume that a significant portion of the processing time is allocated to this database.

Having said that i would suggest conducting a thorough analysis of the log to pinpoint the specific steps that consume the most time.

[Tyron_Benet]

Hi there.

-What version of AML are you running? 
-Is this just one process that is slow or all?

The best rout is a TS case logged.

Seams to be a either slow response from the DB you are using OR the services within the AML system (This is depending on the version of SAS you are using and version of AML)