Hi,   If you haven't tried setting your sort and clustering nodes to use a lot less memory, you should. If I remember correctly, the processing engine will allocate memory for those at the start of your job and it could mean there's not enough memory to either load the Quality Knowledge Base or use the data quality functions. If there are just 50 records in each source, the sort and cluster memory values can be a lot smaller than 512MB. Try starting with 16MB each. Ron ... View more

Hi,   You don't say if this is happening during job building or in the Customize component. If it was working and now you see an error without any other changes, you may have uncovered something like a memory leak. Have you tried closing down DM Studio, reopening the job and/or Customize, and seeing if the problem goes away for some period of time?   It sounds like you may need to open a support track. Ron ... View more

Hi,   You can put your macro variables and mappings in a macro configuration file. These can be edited through DM Studio > Administration > Macro Files. Once defined there, they can be accessed in nodes in jobs using the same convention of %%MACRO%%. The values you set here can be overwritten by values set in the job at run time. Ron ... View more

Hi, the best thing to do is contact a SAS sales representative who can share the options you have to gain access to trial versions of one of the SAS Data Management offerings.   Ron     ... View more

You have two cluster rules, one of which completely supersedes the other. When used together, your second rule will always define the clusters members because it is less specific than the first. The way clustering works in the application, it doesn't stop if one of the conditions is met, it will always process both.   It's curious that you don't have an indivdual's name (or business name) in your matching rules. Your first rule potentially matches all residents sharing an address while the second would match all residents of a building. Name, gender, phone number, or some other uniquely identifying information used in match rules would help your results.   You could use the Cluster Analysis and Sub-Clustering nodes after the main clustering step (using cluster rule 2) to further score the results in each cluster, using door_number as an element in one of the evaluation rules.   Ron ... View more

Assuming you are either building business rules or writing expression code to evaluate null/blank conditions, have you tried using the isnull() or the isblank() functions? ... View more

To answer a few of your questions: If your output has the same number of rows as the input tables combined, it's likely you do not have your matching rules set up correctly (or you have no duplicates at all - possible, but based on this post, it sounds like you are expecting matches). Loosen up your clustering rules and/or decrease sensitivity and examine the results. Does it look reasonable? If you are overmatching, tighten up the match criteria based on what you see in the results. There is a "Cluster Diff" node that can be used to compare one set of match results to another. Use this to test matching rules sets against each other to determine if your clustering rules/sensitivities are getting you closer to desired results. In the surviving record identification node there are options to "keep duplicate records" and "generate distinct surviving records." The former will let you collect records in a clutser of matches, rather than outputting only one survivor that represents the entire match cluster. The latter will create a completely new record to represent the best record based on the SRI rules you design. This can be useful when your requirements dictate that all orginal data must be preservered in the system. Using one or both of these options will have an effect on the number of rows you end up with. I can't help you determine percentage increase in matching after you run your match process. I don't know what the initial value is, how you created it, or what it's measuring. In the abstract, you would want to count the total number of input rows and compare that to only the number of surviving/best record rows that come out of the process. Assuming that the matching results are reasonable, you could then determine the number of duplicates that the process helped identify and resolve. Ron   ... View more

Take a look at the Match Codes (Parsed) node. It lets you map already parsed values by token into an Address match definition. Don't think you have to use every last bit of the address in your match rules. Maybe you have to if the address is the only aspect of the data that can differentiate duplicates but it's not uncommon to use names, phone numbers, gender codes, etc. with partial name and address data to identify matching records.   Ron ... View more

With clusters that large it's very likely your clustering rules should be adjusted. Clusters that size imply that you have very large numbers of duplicates in your source data. If that's the case then no worries but that's normally not the case. Use can use additional fields to tighten the matching or if using match codes, increase the sensitivity of the match codes being generated.   I can't provide you more details on how to select the best record aside from suggesting that you read the documentation. The Surviving Record Identfication node by itself typically is all you need to build that single best record for each match cluster.   Ron ... View more

That's where the Surviving Record Identification node is used. With it you build business rules that describe how you want that best record to be composed for each cluster of matching records. The documentation explains how the node works.   Ron ... View more

Read the documentation on cross-field clustering rules. That's the approach you will want to use to attempt to match addresses across columns. Look into using a Parsing node before the Union step to break apart data into smaller tokens so you can align things better. If a parse definition does not exist for your scenario, you may have to use an Expression node and some regular expression code to help you out. Or, if you really want to get into advanced topics, you can use the Customize component and create custom QKB parse definitions to break apart the address data to meet your needs. For the Surviving Record Identification node, if you add no rules, then one record from the cluster will be used to represent the entire cluster somewhat randomly. What you should really be doing is determining a) what record do you want  to select as the "base" best record (record level rules) and b) determining what field values you want to end up in the best record (field level rules). The documentation for DM Studio describes this in more detail. ... View more

Hi,   I'm going to assume the "result with the correct data" you are looking for is the combined "best view" of the matched data from both sources. Here goes:   Set up connections to your source and target data through the "Data" riser on the left in DM Studio. In DM Studio, create a new data job Use a data source node (for database tables) or other data input nodes to connect to the data in each source Use a Data Union node to bring the data together in one path. Try to align the fields correctly where for example, name data from both sources ends up in the same column. Be sure to add the extra columns from one side or the other where columns don't exist on the other side. If you are looking for an approximate match ("fuzzy"), use the Match Codes node on each columns that should be part of your matching rules. These match codes can be used as input to the clustering node described below. Use a Clustering node to create match rules using a combination of match codes where you want fuzziness in the match and non-match code columns where you want exact matches. Choose the sort output option. Once you have match clusters (these are groupings of records that appear, based on your match rules, to be similar enough to be considered the same data), use a Surviving Record Identification node to build rules to compose a "best record," which is a combination of the values you want from the matched rows, into a single row to represent all the data that has been matched in each cluster. The end result of this node should be single "best records" that represent unique data in your original two sources. Select a Data Target (Insert) node to choose your SQL Server database as the target and write the results there. If you haven't tracked down good examples on the SAS support site, one good alternative source is the online proceedings from past SAS Global Forums. Ron ... View more