cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
HrZiller
‎05-21-2024
  • 13 Posts
  • 3 Likes Given
  • 0 Solutions
  • 1 Likes Received

Re: SAS DI Studio Vs SAS Data Studio

by Fluorite | Level 6 in SAS Data Management
‎05-21-2024 03:51 PM
‎05-21-2024 03:51 PM
Hi Vincent, does this chart exist in a 2024 version? Thanks Jan ... View more

Re: SAS DM platform in a heavely devops inspired environment

by Fluorite | Level 6 in SAS Data Management
‎02-03-2021 09:09 AM
‎02-03-2021 09:09 AM
hi Chris, thanks for fast response! I do realize that this area probably will be very customer specific. With my many years in SAS and working with SAS and our customers, I have seen yes lots of approaches to promotion and handling "artifacts". The recents years high focus on Continues Development/Integration seems to be pushing the use of point&click coding tools like DI in a direction with "the fit seems is a bit too narrow". I have discussed versioning / metadata with Nancy/Stephen a number of times over the years - and can see the challenges ahead: keeping the right metadata available with the right versions. I would imagine similar challenges with Informatica (I believe their datajobs also are somewhat seudo code - XML/markup language ). would be great to hear from a customer that has manage to find a way to work with branching/merging - AND use DI for creating the code. Again thanks for your reply regards Jan ... View more

SAS DM platform in a heavely devops inspired environment

by Fluorite | Level 6 in SAS Data Management
‎02-03-2021 08:10 AM
‎02-03-2021 08:10 AM
hi there, back in 2017 I posted a similar question: Change management with SAS platform - DI, metadata - any experience with "Continues Delivery"??  - but unfortunately didn't get any responses 😞 I am trying again now - hoping for change of times has created miracles... 😀   At our customer site we are increasingly moving towards a very devops style of working. Most of the other teams developing applikations with more code based technologies are using Github, Jenkins and everything related. Our SAS DI team is the only not using these tools. Our environment:  5 environments: Dev, Test, 2x Preprod and Prod > 900 DI jobs, >1500 tables, many user written transformations, macros etc. - fairly large environment we have potentiel of 15 developers using DI Studio - with check-out/in enabled   I have seen the latest material available regarding Github: - Bogdant blog post: https://communities.sas.com/t5/SAS-Communities-Library/DevOps-Applied-to-SAS-9-SAS-DI-jobs-the-Git-Plug-in-GitLab-and/ta-p/677993  - https://communities.sas.com/t5/SAS-Communities-Library/SAS-DI-Developers-Unite-The-new-GIT-plug-in-in-Data-Integration/ta-p/549202 and a number of the more SAS code supported entries   But.... neither of these seems to head in the direction of the terms branching and merging. Bogdant's post is very weel describing the proces of automating the moving/testing the DI job/code over what is called "automation valley" in the post.   We are seeing many customers with these requirements in Data Warehousing projects - and I am hoping that some of your projects might have looked into solutions to this 😀 - please let me know how you have looked at solutions for this challenge.   @ChrisHemedinger - would you by any chance know of any customer stories regarding this area? Thanks!   Thanks in advance Jan Zilstorff Knowit ... View more

Re: Use of performance option DBKey with SAS EG when using point and c...

by Fluorite | Level 6 in SAS Enterprise Guide
‎09-17-2018 08:32 AM
1 Like
‎09-17-2018 08:32 AM
1 Like
Hi Chris,   can't believe I didn't see this in the window.... must get new glasses! Thanks a lot - huge performance difference with this on. Hope things are well in NC after the hurricane!   Cheers Jan   PS: my ninja-tech skills must be failing - even after +10 years in SAS support 🙂 ... View more

Use of performance option DBKey with SAS EG when using point and click

by Fluorite | Level 6 in SAS Enterprise Guide
‎09-11-2018 04:56 AM
‎09-11-2018 04:56 AM
hi there,   we are using SAS EG for adhoc users here at the customer I am working for. DW is stored in SQL Server. Simple joins create SAS tables, but when joining back to SQL server we sometimes get slow performance, because we are joining a SAS table with a DBMS table. This performance can be much better when using the DBkey option. I am aware that we can modify the work location to SQL Server as well - but this not the solution here in this situation.   EG Release:            7.15 HF2 (7.100.5.6112) (64-bit)   Code generated by EG: PROC SQL;    CREATE TABLE WORK.bygning AS    SELECT t2.*    FROM WORK.SFE t1    INNER JOIN _03_PLUS.Bygning t2    ON (t1.SFE_ID_ICE = t2.SFE_ID_ICE)    ORDER BY t2.Bygningsnummer,             t2.Bygning_ID_ICE,             t2.Virkning_fra     ; QUIT;   SAS log result: NOTE: PROCEDURE SQL used (Total process time):       real time           2:56.05       cpu time            2:55.75     Modified SAS EG code - added: (dbkey=sfe_id_ice) PROC SQL;    CREATE TABLE WORK.bygning AS    SELECT t2.*    FROM WORK.SFE t1    INNER JOIN _03_PLUS.Bygning(dbkey=SFE_ID_ICE) t2    ON (t1.SFE_ID_ICE = t2.SFE_ID_ICE)    ORDER BY t2.Bygningsnummer,             t2.Bygning_ID_ICE,             t2.Virkning_fra     ; QUIT;   SAS log result: NOTE: PROCEDURE SQL used (Total process time):       real time           0.27 seconds       cpu time            0.01 seconds   Difference with/without DBkey: 3min vs. less than 1 sec.   So my specific question is: How can I get this DBkey option added when using EG standard join functionality?   Thanks Jan     Full SAS log below: 1 The SAS System 08:28 Tuesday, September 11, 2018 1 ;*';*";*/;quit;run; 2 OPTIONS PAGENO=MIN; 3 %LET _CLIENTTASKLABEL='Program'; 4 %LET _CLIENTPROCESSFLOWNAME='Process Flow'; 5 %LET _CLIENTPROJECTPATH='\\ccta.dk\dfssystem\HomeFolders\w5\W92285\SAS\Find data til kommune-ejendomsnr - v1.egp'; 6 %LET _CLIENTPROJECTPATHHOST='C780320'; 7 %LET _CLIENTPROJECTNAME='Find data til kommune-ejendomsnr - v1.egp'; 8 %LET _SASPROGRAMFILE=''; 9 %LET _SASPROGRAMFILEHOST=''; 10 11 ODS _ALL_ CLOSE; 12 OPTIONS DEV=PNG; 13 GOPTIONS XPIXELS=0 YPIXELS=0; 14 FILENAME EGSR TEMP; 15 ODS tagsets.sasreport13(ID=EGSR) FILE=EGSR 16 STYLE=HTMLBlue 17 STYLESHEET=(URL="file:///C:/Program%20Files/SASHome/SASEnterpriseGuide/7.1/Styles/HTMLBlue.css") 18 NOGTITLE 19 NOGFOOTNOTE 20 GPATH=&sasworklocation 21 ENCODING=UTF8 22 options(rolap="on") 23 ; NOTE: Writing TAGSETS.SASREPORT13(EGSR) Body file: EGSR 24 25 GOPTIONS ACCESSIBLE; 26 options sastrace=',,,d' sastraceloc=saslog; 27 PROC SQL; 28 CREATE TABLE WORK.bygning AS 29 SELECT t2.* 30 FROM WORK.SFE t1 31 INNER JOIN _03_PLUS.Bygning(dbkey=SFE_ID_ICE) t2 32 ON (t1.SFE_ID_ICE = t2.SFE_ID_ICE) 33 ORDER BY t2.Bygningsnummer, 34 t2.Bygning_ID_ICE, 35 t2.Virkning_fra 36 ; OLEDB: AUTOCOMMIT turned ON for connection id 3 OLEDB: *-*-*-*-*-*-* COMMIT *-*-*-*-*-*-* on connection 3 OLEDB: AUTOCOMMIT turned OFF for connection id 3 OLEDB: AUTOCOMMIT turned ON for connection id 3 OLEDB: *-*-*-*-*-*-* COMMIT *-*-*-*-*-*-* on connection 3 OLEDB_10: Prepared: on connection 3 SELECT * FROM "_03_PLUS"."Bygning" OLEDB: AUTOCOMMIT turned ON for connection id 4 OLEDB: *-*-*-*-*-*-* COMMIT *-*-*-*-*-*-* on connection 4 OLEDB: AUTOCOMMIT turned OFF for connection id 4 OLEDB_11: Prepared: on connection 3 SELECT "SFE_ID_ICE", "Bygning_ID_ICE", "ICE_Registreret", "Grund_ID_ICE", "Adresse_ID_ICE", "Jordstykke_ID_ICE", "Kommune_ID_ICE", "Bygningspunkt_ID_ICE", "Ejerlejlighed_ID_ICE", "BFG_ID_ICE", "Bygning_ID", "Kommunenummer", "Ejendomsnummer", "Bygningsnummer", "BygningensAnvendelse", "AntalLejlighederMedKoekken", "AntalLejlighederUdenKoekken", "Opfoerelsesaar", "EjendomsnummerForEjerlejlighed", "OmTilbygningsaar", "EjerlejlighedsNummer", "MidlertidigOprettelseFuldforelse", "KodeForMereEndEnLejlighed", "DatoForMidlertidigOpfoertBygning", "Vandforsyning", "Afloebsforhold", "YdervaeggensMateriale", "Tagdaekningsmateriale", "SuplYdervaeggensMateriale", "SuplTagdaekningsMateriale", "AsbestholdigtMateriale", "KildeTilBygningensMaterialer", "SamletBygningsareal", "BygningensSamledeBoligAreal", "BygningensSamledeErhvervsAreal", 2 The SAS System 08:28 Tuesday, September 11, 2018 "BebyggetAreal", "ArealIndbyggetGarage", "ArealIndbyggetCarport", "ArealIndbyggetUdhus", "ArealIndbyggetUdestueEllerLign", "SamlArealAfLukOverdaekPaaBygning", "ArealAfAffaldsrumITerraenniveau", "AndetAreal", "ArealAfOverdaekketAreal", "ArealAabneOverdaekPaaBygningSaml", "AdgangsAreal", "BeregningsprincipCarportAreal", "BygningsarealerKilde", "AntalEtager", "AfvigendeEtager", "Varmeinstallation", "Opvarmningsmiddel", "SupplerendeVarme", "Sikringsrumpladser", "Fredning", "BevaringsvaerdighedReference", "StormraadetsOversvoemmSelvrisiko", "DatoForRegistFraStormraadet", "Byggeskadeforsikringsselskab", "DatoForByggeskadeforsikring", "Udledningstilladelse", "OmfattetAfByggeskadeforsikring", "MedlemskabAfSpildevandsforsyning", "EjendommensEjerforholdskode", "PaabudVedrSpildevandsafledning", "FristVedrSpildevandsafledning", "TilladelseTilUdtraeden", "DatoForTilladelseTilUdtraeden", "TilladAlternBortskafEllerAfled", "DtTilladAlternBortskafEllerAfled", "Sikklas", "TinglystAreal", "Registreret_fra", "Registreret_til", "Virkning_fra", "Virkning_til", "DataKilde", "Md5", "NyRaekke" FROM "_03_PLUS"."Bygning" WHERE ((("SFE_ID_ICE"= ? ) OR (("SFE_ID_ICE" IS NULL ) AND ( ? IS NULL )))) NOTE: Compressing data set WORK.BYGNING increased size by 100.00 percent. Compressed is 2 pages; un-compressed would require 1 pages. NOTE: Table WORK.BYGNING created, with 10 rows and 79 columns. 37 QUIT; NOTE: PROCEDURE SQL used (Total process time): real time 0.34 seconds cpu time 0.06 seconds 38 PROC SQL; 39 CREATE TABLE WORK.bygning AS 40 SELECT t2.* 41 FROM WORK.SFE t1 42 INNER JOIN _03_PLUS.Bygning t2 43 ON (t1.SFE_ID_ICE = t2.SFE_ID_ICE) 44 ORDER BY t2.Bygningsnummer, 45 t2.Bygning_ID_ICE, 46 t2.Virkning_fra 47 ; OLEDB: AUTOCOMMIT turned ON for connection id 3 OLEDB: *-*-*-*-*-*-* COMMIT *-*-*-*-*-*-* on connection 3 OLEDB: AUTOCOMMIT turned OFF for connection id 3 OLEDB: AUTOCOMMIT turned ON for connection id 3 OLEDB: *-*-*-*-*-*-* COMMIT *-*-*-*-*-*-* on connection 3 OLEDB_12: Prepared: on connection 3 SELECT * FROM "_03_PLUS"."Bygning" OLEDB: AUTOCOMMIT turned ON for connection id 4 OLEDB: *-*-*-*-*-*-* COMMIT *-*-*-*-*-*-* on connection 4 OLEDB: AUTOCOMMIT turned OFF for connection id 4 OLEDB_13: Prepared: on connection 3 SELECT "SFE_ID_ICE", "Bygning_ID_ICE", "ICE_Registreret", "Grund_ID_ICE", "Adresse_ID_ICE", "Jordstykke_ID_ICE", "Kommune_ID_ICE", "Bygningspunkt_ID_ICE", "Ejerlejlighed_ID_ICE", "BFG_ID_ICE", "Bygning_ID", "Kommunenummer", "Ejendomsnummer", "Bygningsnummer", "BygningensAnvendelse", "AntalLejlighederMedKoekken", "AntalLejlighederUdenKoekken", "Opfoerelsesaar", "EjendomsnummerForEjerlejlighed", "OmTilbygningsaar", "EjerlejlighedsNummer", "MidlertidigOprettelseFuldforelse", "KodeForMereEndEnLejlighed", "DatoForMidlertidigOpfoertBygning", "Vandforsyning", "Afloebsforhold", "YdervaeggensMateriale", "Tagdaekningsmateriale", "SuplYdervaeggensMateriale", "SuplTagdaekningsMateriale", "AsbestholdigtMateriale", "KildeTilBygningensMaterialer", "SamletBygningsareal", "BygningensSamledeBoligAreal", "BygningensSamledeErhvervsAreal", "BebyggetAreal", "ArealIndbyggetGarage", "ArealIndbyggetCarport", "ArealIndbyggetUdhus", "ArealIndbyggetUdestueEllerLign", "SamlArealAfLukOverdaekPaaBygning", "ArealAfAffaldsrumITerraenniveau", "AndetAreal", "ArealAfOverdaekketAreal", "ArealAabneOverdaekPaaBygningSaml", "AdgangsAreal", "BeregningsprincipCarportAreal", "BygningsarealerKilde", "AntalEtager", "AfvigendeEtager", "Varmeinstallation", "Opvarmningsmiddel", "SupplerendeVarme", "Sikringsrumpladser", "Fredning", "BevaringsvaerdighedReference", "StormraadetsOversvoemmSelvrisiko", "DatoForRegistFraStormraadet", "Byggeskadeforsikringsselskab", "DatoForByggeskadeforsikring", "Udledningstilladelse", "OmfattetAfByggeskadeforsikring", "MedlemskabAfSpildevandsforsyning", 3 The SAS System 08:28 Tuesday, September 11, 2018 "EjendommensEjerforholdskode", "PaabudVedrSpildevandsafledning", "FristVedrSpildevandsafledning", "TilladelseTilUdtraeden", "DatoForTilladelseTilUdtraeden", "TilladAlternBortskafEllerAfled", "DtTilladAlternBortskafEllerAfled", "Sikklas", "TinglystAreal", "Registreret_fra", "Registreret_til", "Virkning_fra", "Virkning_til", "DataKilde", "Md5", "NyRaekke" FROM "_03_PLUS"."Bygning" OLEDB_14: Executed: on connection 3 SELECT "SFE_ID_ICE", "Bygning_ID_ICE", "ICE_Registreret", "Grund_ID_ICE", "Adresse_ID_ICE", "Jordstykke_ID_ICE", "Kommune_ID_ICE", "Bygningspunkt_ID_ICE", "Ejerlejlighed_ID_ICE", "BFG_ID_ICE", "Bygning_ID", "Kommunenummer", "Ejendomsnummer", "Bygningsnummer", "BygningensAnvendelse", "AntalLejlighederMedKoekken", "AntalLejlighederUdenKoekken", "Opfoerelsesaar", "EjendomsnummerForEjerlejlighed", "OmTilbygningsaar", "EjerlejlighedsNummer", "MidlertidigOprettelseFuldforelse", "KodeForMereEndEnLejlighed", "DatoForMidlertidigOpfoertBygning", "Vandforsyning", "Afloebsforhold", "YdervaeggensMateriale", "Tagdaekningsmateriale", "SuplYdervaeggensMateriale", "SuplTagdaekningsMateriale", "AsbestholdigtMateriale", "KildeTilBygningensMaterialer", "SamletBygningsareal", "BygningensSamledeBoligAreal", "BygningensSamledeErhvervsAreal", "BebyggetAreal", "ArealIndbyggetGarage", "ArealIndbyggetCarport", "ArealIndbyggetUdhus", "ArealIndbyggetUdestueEllerLign", "SamlArealAfLukOverdaekPaaBygning", "ArealAfAffaldsrumITerraenniveau", "AndetAreal", "ArealAfOverdaekketAreal", "ArealAabneOverdaekPaaBygningSaml", "AdgangsAreal", "BeregningsprincipCarportAreal", "BygningsarealerKilde", "AntalEtager", "AfvigendeEtager", "Varmeinstallation", "Opvarmningsmiddel", "SupplerendeVarme", "Sikringsrumpladser", "Fredning", "BevaringsvaerdighedReference", "StormraadetsOversvoemmSelvrisiko", "DatoForRegistFraStormraadet", "Byggeskadeforsikringsselskab", "DatoForByggeskadeforsikring", "Udledningstilladelse", "OmfattetAfByggeskadeforsikring", "MedlemskabAfSpildevandsforsyning", "EjendommensEjerforholdskode", "PaabudVedrSpildevandsafledning", "FristVedrSpildevandsafledning", "TilladelseTilUdtraeden", "DatoForTilladelseTilUdtraeden", "TilladAlternBortskafEllerAfled", "DtTilladAlternBortskafEllerAfled", "Sikklas", "TinglystAreal", "Registreret_fra", "Registreret_til", "Virkning_fra", "Virkning_til", "DataKilde", "Md5", "NyRaekke" FROM "_03_PLUS"."Bygning" NOTE: Compressing data set WORK.BYGNING increased size by 100.00 percent. Compressed is 2 pages; un-compressed would require 1 pages. NOTE: Table WORK.BYGNING created, with 10 rows and 79 columns. 48 QUIT; NOTE: PROCEDURE SQL used (Total process time): real time 2:55.65 cpu time 2:55.35 49 50 GOPTIONS NOACCESSIBLE; 51 %LET _CLIENTTASKLABEL=; 52 %LET _CLIENTPROCESSFLOWNAME=; 53 %LET _CLIENTPROJECTPATH=; 54 %LET _CLIENTPROJECTPATHHOST=; 55 %LET _CLIENTPROJECTNAME=; 56 %LET _SASPROGRAMFILE=; 57 %LET _SASPROGRAMFILEHOST=; 58 59 ;*';*";*/;quit;run; 60 ODS _ALL_ CLOSE; 61 62 63 QUIT; RUN; 64   ... View more

Sample Environment Manager setup

by Fluorite | Level 6 in Administration and Deployment
‎04-19-2018 08:02 AM
‎04-19-2018 08:02 AM
hi all,   we are looking at enabling ENV.Manager at the site here. We have a 4 server setup with a traditional SAS DI/DM platform. On a daily basis we are running +700 DI jobs which are loading data from public systems to a central Data Vault data model and creating/publishing a data mart. I am interested in seeing sample dashboards from Environment Manager - and get inspiration to how you are using Env.Manager. At present I see the following metrics - but more could be added: Server health, DISK-IO / CPU / network - usage/peaks, user information, DI job information (execution/load times/cpu usage/IO/usage, number of records - incl. historic information), flow information, security/rights etc...  there should be plenty of possibilities.   All samples would be greatly appreciated!   Thanks in advance Jan   ... View more

Re: Change management with SAS platform - DI, metadata - any experienc...

by Fluorite | Level 6 in SAS Data Management
‎09-08-2017 03:30 AM
‎09-08-2017 03:30 AM
hi Patrick,   thanks for your thoughts. The more I think this through the more it seems a requirement for promoting pure SAS code rather than the full metadata incl. redeploy since the SPK packages are bit more complex to deal with when respect to track changes.   Could be interesting to hear from SAS customers that are seeing this in their development environments.   regards Jan   PS: not SAS employee any more - can not change the "SAS employee" label for my account. ... View more

Re: Change management with SAS platform - DI, metadata - any experienc...

by Fluorite | Level 6 in SAS Data Management
‎09-07-2017 05:29 AM
‎09-07-2017 05:29 AM
We are actually using subversion, and check-out/check-in when develuping in DI - so it is not this what I am looking for.   It is the handling of the concept of "versions" (which does not really exist in metadata)  and coordinating the traditional approach: dev test prod   example: We are promotion a version: dev - v1 promotion to test v1 promotion to prod v1   Now v1 is in production   Meanwhile we get new business requirements - which means that we start on v2: Dev v2 promoted to test V2 but our downstream teams are still working on the V1 version and not ready for v2 yet.   This is the complicated by the fact that their promotion processes are based on the approach: continues delivery/deployment - so they might be working in many version in many environments: dev, test, demo, training, preprod, prod etc.... and the add versions...    So the complexity is increased many times. At present our thoughts are that the downstream teams must handle this internally by housekeeping our data delivery in the amount of versions they require. This is not optimal since our development phases very fast can get put of sync.   So I am looking for thoughs and experiences in a similar setup.   regards Jan  ... View more

Change management with SAS platform - DI, metadata - any experience wi...

by Fluorite | Level 6 in SAS Data Management
‎09-07-2017 04:20 AM
‎09-07-2017 04:20 AM
hi all,   I am working on a large project with +40 developers within many different technoligy areas. The SAS Data Integration platform using DI Studio is used for the sourcing data, remodelling and the creating a datamodel being used by teams with a more "devops" approach to development processes.  In our team we are leaning towards the more traditional dev-test-prod approach, using partiel promotion - and (unfortunately) a bit to many manual processes involved in the promotion. We are looking at how this can be optimized.   My question is: Our downstream dependent teams are leaning towards a continues delivery approach, with a release management proces that is based on Github and at present seems very automated/script based. They can change "versions" very fast - due to the fact they are using code-based tools. So here it is: has anybody had experiences they can share on how a SAS DI+metadata environment can work efficiently with continues delivery.   At present we have approx. 6-700 DI jobs in the environment - this number is going slowly up.   All the information I have found regarding th use of release management tools and SAS is based on a *.SAS file approach - not using DI studio and metadata. We have considered not using metadata going info production - but using the LSF scheduling tool is giving us some dependencies of having metadata.   Thanks Jan ... View more

Re: Five papers on Recommended SAS 9.4 Security Model Design (part 2 o...

by Fluorite | Level 6 in SAS Communities Library
‎05-26-2017 03:24 AM
1 Like
‎05-26-2017 03:24 AM
1 Like
Awesome work David! Great to get it published here! regards Jan ... View more

Re: Five papers on Recommended SAS 9.4 Security Model Design (part 1 o...

by Fluorite | Level 6 in SAS Communities Library
‎05-26-2017 02:57 AM
3 Likes
‎05-26-2017 02:57 AM
3 Likes
hi David,   great to see you got the papers out in the community!! Wondering if Michelle is sleeping with the blog under the pillow - fast update from downunder 🙂 Going out next week to visit a customer to discuss security model!   Cheers Jan ... View more

Re: General activities on "data archiving" in a SAS Data Warehouse env...

by Fluorite | Level 6 in SAS Data Management
‎05-02-2017 08:36 AM
‎05-02-2017 08:36 AM
Hi Patrick,   Thanks for your input to my question! Great to hear what is being considered!   A valid question you are asking: am I talking about "keep older versions of data" or talking about backup/recovery? ->> I am specificially talking about "keep older versions of data" and NOT backup/recovery   Both data archiving and backup/recovery aree two very valid topics in any DW implementation - but for this post here - archiving data is what I am interested in.   I can add that for backup and recovery - Margaret Crevar from SAS has created this paper for SGF2017: http://support.sas.com/resources/papers/proceedings17/SAS0565-2017.pdf - good reference for the dialogue with IT regarding proper handling of backup/recovery procedures for the SAS platform - not a simple task to get right.   Back to the topic: Data Archiving   Basically it is the "data life cycle" that I am addressing here - and how to handle the archiving part: From "birth" in source systems -> entry into DW/Analytical architecture -> to retirement (available but perhaps rarely used) -> deletion and not available any more.   There can be many reasons why archiving is relevant: Ever increasing data volume Trying to reduce price pr. GB by moving to different lower cost disc-location Trying to ensure optimal performance in production DW/Analytical environment by reducing number of records Compliance reasons: regulations e.g.. upcoming EU GDPR regulation (ensure you are only processing data you are legally allowed to process) Ensure general business requirements are meet for what ever reason that may be valid Project/solution lifetime is long enough and therefore data archiving will be relevant. For more short-term implementations data archiving may not be relevant.   Then on top of the above you need to add your own policies/governance about what/when to archive.    As you mention prices per. GB is getting lower every day - and that is positive for current and future implementations - BUT I have no doubt that a new lower cost will only mean that cost baseline moves and therefore still will lead to thoughts on how money can be saved.  New technologies like Hadoop also are showing a price/GB that business are seeing as lower than traditional SAN disc implementations.   Whether the it is SAS platform or other ETL tools that are being used - data archiving seems relevant. I have seen plenty of very large DW implementations with the use of SAS Data Management platform, DI Studio, DataFlux tools, SPD Server - with specific SAS solutions on top.   The type of data archive that is implemented should adhere to more specific business requirements - like: Should I be able to access data with XYZ tool? Should data archive be considered as a part of the normal production environment and therefore available 24/7/365? Should there be backup of the archive? Etc…   Then the actual implementation can be done in ways that you are describing - as in integral part of the normal ETL routines in the DW setup - with use of database features like partitioning.   Personally I have not decided whether which one of the two below approaches I like more: Build archive functionality along with normal ETL development during DW project implementation Build afterburner ETL/programs that can processes selected relevant data for archive purposes Manual approach where data are moved based on individual choices   I am leaning towards the second approach, since the impact on the initial implementation project could be fairly low: design and ideas could naturally be surfaced early - it would seem obvious that thoughts on data model implementation can affect how data archiving can be done afterword's. You can then develop the required ETL jobs when required and not impact the normal project implementation. On the other hand - it can be difficult to get the funding for something that perhaps should have been done earlier???  With plenty of current DW implementations - the afterburner approach would be more realistic for many customers now.   All for now.   Please drop more ideas and thoughts for the topic!   Regards Jan ... View more

General activities on "data archiving" in a SAS Data Warehouse environ...

by Fluorite | Level 6 in SAS Data Management
‎05-01-2017 07:01 AM
‎05-01-2017 07:01 AM
Hi there,   I am working on a case where we are looking at establishing an archive function on the "back" of the SAS DW implementation. I have been trying to find ressources on data archiving and SAS - but it has been very limited what I have found at this stage. During many DW projects, scoping, estimating etc. - data archiving has rarealy been discussed - but in this time and age, it seems more and more obvious that I probably needs to be a more proactive required choice in a DW implementation. And now to the question: What are you doing out there at the SAS world. Are you adding extra disc-technology, eg. adding Hadoop, creating postprocesses, buying special DW appliance technology when trying to cater for large and expanding data volume in your environments.   Please drop me your thoughts for an open dialogue here.   Thanks in advance Jan zilstorff ... View more
Contact Me
Online Status
Offline
Date Last Visited
‎05-21-2024 08:18 PM
Group Hubs for HrZiller
View All
Likes given to
View All
Likes from
View All
Quick links