
01-16-2024
nar_sas
SAS Employee
Member since
06-23-2011
- 49 Posts
- 12 Likes Given
- 4 Solutions
- 20 Likes Received
-
Latest posts by nar_sas
Subject Views Posted 28331 08-04-2021 01:11 PM 4014 03-22-2021 05:53 PM 4228 03-08-2021 01:43 AM 772 10-24-2020 11:35 PM 2043 10-09-2018 10:51 AM 2306 07-19-2018 03:57 PM 1412 03-09-2018 05:06 PM 1890 12-13-2017 11:08 AM 67087 08-30-2017 10:53 AM 1323 08-30-2017 10:10 AM -
Activity Feed for nar_sas
- Liked SAS Viya: Control Dependencies in a Custom Step for MKQueen. 01-16-2024 05:15 PM
- Got a Like for Re: Dealing with 32 character restrictions. 12-07-2023 06:46 PM
- Liked Why Are Data Catalogue and Data Governance So Important? Q&A, Slides and On-Demand Recording for KumarT_SAS. 09-05-2023 04:36 AM
- Liked 3 Steps to Empower Your Information Catalog Users for Bogdan_Teleuca. 09-05-2023 04:32 AM
- Liked SAS Viya: Add a Customized Quality Knowledge Base (QKB) to CAS and SAS Compute for MKQueen. 09-01-2023 09:27 AM
- Liked SAS Viya surpasses SAS 9 in data management for jamesob. 06-08-2023 02:35 PM
- Liked Slow Pokes - Sustainable Aviation Fuel for the Aviation Industry for KCleburn. 03-13-2023 03:26 PM
- Liked SAS Viya 4: Loading External Path-Based Data for StephenFoerster. 02-23-2023 09:35 AM
- Liked Power-play in 5 ways - SAS Viya helps your data quality efforts for jarno. 09-05-2022 01:17 PM
- Liked Team Profile: BC_Æ_Team for BC_Æ_Team. 04-15-2022 05:09 PM
- Got a Like for Re: Dataflux 2.7 and Encrypted SAS Datasets. 03-07-2022 10:12 PM
- Liked Management Solutions - Enel for andreanavarrete. 03-01-2022 11:31 AM
- Got a Like for Help create physical table from metadata in sas code, i.e. no "clicking" in DI-Studio etc.. 08-12-2021 05:19 PM
- Posted Re: Reading Sas datasets in PowerBI on Microsoft Integration with SAS. 08-04-2021 01:11 PM
- Liked Clean electricity system based on renewable generation for YWF. 04-05-2021 11:12 AM
- Posted Re: Clean electricity system based on renewable generation on SAS Hackathon Team Profiles (Past). 03-22-2021 05:53 PM
- Posted Re: Clean electricity system based on renewable generation on SAS Hackathon Team Profiles (Past). 03-08-2021 01:43 AM
- Got a Like for Re: Getting Top and Bottom 5 distinct values. 10-25-2020 01:01 AM
- Posted Re: Getting Top and Bottom 5 distinct values on SAS Procedures. 10-24-2020 11:35 PM
- Posted Re: How to move Business Data Network Data from One Server to Other on SAS Data Management. 10-09-2018 10:51 AM
-
Posts I Liked
Subject Likes Author Latest Post 1 1 2 5 17 -
My Liked Posts
Subject Likes Posted 1 11-22-2011 02:46 PM 1 10-24-2020 11:35 PM 1 07-19-2018 03:57 PM 2 12-13-2017 11:08 AM 3 08-30-2017 10:53 AM -
My Library Contributions
Subject Likes Author Latest Post 0
08-04-2021
01:11 PM
You should be able to connect to SAS from PowerBI using OLE DB. Microsoft provides documentation on how to do this on their website here: https://docs.microsoft.com/en-us/power-bi/connect-data/desktop-connect-using-generic-interfaces#data-sources-accessible-through-ole-db
which uses the SAS Providers for OLE DB from the SAS Download site here: https://support.sas.com/downloads/package.htm?pid=648
I just tried this in my PowerBI installation on my desktop and it seemed to be working for me. I would expect that if you have something tricky like custom formats it might not apply them correctly, but this might be worth a try.
Editor's Note: I am including totoleheros link to the latest versions of SAS Providers for OLEDB: https://support.sas.com/downloads/browse.htm?fil=&cat=64
... View more
03-22-2021
05:53 PM
Looks so good! Nice job team!
... View more
03-08-2021
01:43 AM
Hello Everyone. My name is Nancy, and I am a computer and data scientist here at SAS. I did my University thesis on solar energy forecasting and am looking forward to assisting on this project. Here are some findings from that research that you might find helpful, and I am happy to send along the paper. I have also provided some additional references that you might find helpful.
- One important surprise predictor is air quality. I found up to a 10% reduction in energy output at certain times of the year based on poor air quality. Our area here in the east coast of the united states is bathed in particles from coal fired power plants. In summer months the atmosphere mixes more readily and so the impact of these particles are not as pronounced. However, during winter months, the atmosphere is more stratified, and the particles do not mix or dissipate as readily, particularly in early winter. During this time, solar irradiation is decreased due to these particles. Finally, this is very much a locality based predictor. Information related to air quality can generally be collected from government sources; in our country, daily numbers for most localities are freely available from the US Environmental Protection agency.
- You have already mentioned other useful predictors above including irradiance, date/time, temperature, solar azimuth, and windspeed. Wind was particularly interesting as a predictor; in our area, days of high wind produced less output, due to the turbulence and cooling effect associated with wind flowing over the panels. These numbers are available from government weather data sites. Note that we found it to be important to use meteorological data collected as close as possible to the installation site where the panels are located.
Another finding from our research into this area is the modeling techniques that can produce better accuracy with this type of data. We found that Random Forest is a good method for this type of forecasting because it support highly non-linear data that exhibits multi-collinearity. Deep learning methods that incorporate time series, such as RNN and Gradient Boost, are also quite efficient as predictive models, at the expense of more complexity and greater processing time. Since solar energy output produces an abundance of data, it lends itself very well to deep learning techniques.
Here are a few references you might find helpful:
https://www.researchgate.net/profile/Zhenxing-Lei/publication/336184094_A_review_of_deep_learning_for_renewable_energy_forecasting/links/60014485299bf14088975ffd/A-review-of-deep-learning-for-renewable-energy-forecasting.pdf
Abdel-Nasser, M., & Mahmoud, K. (2017). Accurate photovoltaic power forecasting models using deep LSTM-RNN. Neural Computing and Applications , 1-21.
Chandler, D. (2018, August 18). Air pollution can put a dent in solar power. Retrieved from MIT news: http://news.mit.edu/2018/air-pollution-can-put-dent-solar-power-0829
Hickey, H. (2018, July 24). WHY WINTER AIR IN THE EASTERN U.S. IS STILL SO DIRTY. Retrieved from Futurity.org: https://www.futurity.org/winter-air-pollution-emissions-1819872/
NCSU. (2019, May 8). Weather and Climate Database. Retrieved from North Carolina Climate Office: https://climate.ncsu.edu/cronos/?station=REED
NOAA. (2019, May 10). Solar Geometry Calculator. Retrieved from NOAA.gov: https://www.esrl.noaa.gov/gmd/grad/antuv/SolarCalc.jsp
US Environmental Protection Agency. (2019). Air Quality Daily Values Report. Retrieved from US EPA: https://www.epa.gov/outdoor-air-quality-data/air-quality-index-daily-values-report
Zulkifli, H. (2019, Mar 12). Multivariate Time Series Forecasting Using Random Forest. Retrieved from Towards Data Science: https://towardsdatascience.com/multivariate-time-series-forecasting-using-random-forest-2372f3ecbad1
... View more
10-24-2020
11:35 PM
1 Like
I tried the UNIVARIATE approach on a wide table of 7500 columns (35 rows), and UNIVARIATE ran in 300s. All other techniques I tried on this table ran much much slower. I think the Univariate approach is very powerful. Here is my code just as an example.
ods listing close; ods output frequencies=_afreq; proc univariate data=sashelp.cars NEXTRVAL=20 freq; var _numeric_; run; ods listing;
... View more
10-09-2018
10:51 AM
BDN participates in the SAS Web infrastructure platform, which means that the migration/promotion tools should work as well to move BDN content between environments. This is part of the SAS Deployment tools such as the SAS Deployment Manager (SDM). I found this powerpoint presentation that describes the tooling that exists to manage content in the 9.4 platform, which includes all of the BDN related content. https://www.sas.com/content/dam/SAS/no_no/doc/other1/FANS_Oslo_11mars_SASDeploymentBackupUtility.pdf. This should make it easy for you to move content around.
... View more
07-19-2018
03:57 PM
1 Like
Yes it is possible to use code based transformations in SAS Data Studio in SAS Viya.
... View more
03-09-2018
05:06 PM
The language that is used in the calculated columns transform is CAS datastep. You can find a function reference here:
SAS Functions and CALL Routines by Category
Note that not all datastep functions are supported in CAS but many of them are. Thanks for providing the feedback, and we do plan to provide an expression builder in a future release.
... View more
12-13-2017
11:08 AM
2 Likes
There are a couple of ways to do this using DM Studio and DM Server. The BASE driver in DM Studio/Server cannot read them directly, but there are a few options available to get this to work.
One way is to use SAS Federation Server to access the data and setup a connection to federation server from DM Studio/Server. Here are instructions on how to configure SAS Federation server connections to metadata bound libraries. http://support.sas.com/documentation/cdl/en/fedsrvag/70395/HTML/default/viewer.htm#p19idcme77fiomn11bj3ql0b1uhi.htm
Another way is to use SAS to reach through to the dataset. You can add a SAS code node to your Data jobs and in the code use SAS to reach through to the library. You would then have to process and land the data for DM Studio/Server to read it, but it could only be the required subset that you need for your data management jobs.
A third way is that you can call many of the data quality functions available directly from SAS such as dqparse, dqstandardize, and others. You can also call services in dm server directly from your SAS jobs that perform many data quality functions, which would not require you to move the data to a Data Management instance. For example, DI Studio has a transform available that will processrows of data in the DM Server.
Hope this helps.
... View more
08-30-2017
10:53 AM
3 Likes
Here are a few comments/suggestions on this topic:
1. @SASKiwi has a good suggestion to use SQL passthru. You can use SQL passthru with most SAS/Access engines, not only with ODBC. DI Studio will even generate SQL passthru for you in the SQL transforms if you select passthru=yes in the UI options. This will help get around some of the restrictions you are seeing, and has the bonus of being more performant as the SQL gets pushed to the database.
2. Tables and columns can have LABELs associated with them, which you can use for the longer text. Labels in many databases do not have length restrictions. We see this in other databases that have similar limits. Users build their column names shorter and use the labels to store longer and more descriptive variable information. SAS will capture both column name and label information when registering these tables in the SAS metadata server and pass that information around to all the SAS applications using that data.
3. We at SAS have heard the request and we are working on continuing to improve long variable name support in the future.
... View more
- Tags:
- fix please
08-30-2017
10:10 AM
The cluster node essentially does essentially the same thing and should run just as fast. GIve that a try and see if it meets your needs.
... View more
01-17-2017
11:32 AM
One additional comment, if you use the SAS in-database hadoop code accelerator to read/write the data, Avro, Parquet, and Sequence files are automatically compressed using Snappy, assuming it is enabled on your hadoop system. The DBCREATE_TABLE_OPTS setting is also available on the table properties in DI Studio.
Nancy
... View more
08-10-2016
04:02 PM
As Ron indicates, yes it is possible, and the syntax can be tricky. Here are a few things to try, and be aware of.
- Use the execute function. It is the one that you need to use to call some external program
- The command is running locally, so you have to be able to run commands on the host you are trying to execute the script on
- Check that you have permissions set, and that you can get to the code from the program
You can try this. Create some simple program, and just call that. Verify that your expression code is calling that program correctly. I would put it in the same location as Excel is to make sure you can get to the directory that you are trying to call out to with your more complex call.
If you have verified that and things still are not working, then there is a handy microsoft tool called procmon: https://technet.microsoft.com/en-us/sysinternals/bb795533.aspx.
Its a handy way on a windows machine to see all files and processes being used. It can help figure out what is happening on the host you are working on to see where things might be failing.
Hope this helps,
Nancy
... View more
02-25-2015
06:23 PM
A topic of possible confusion is how authentication works in an Hadoop system. Recently we had a question about which user to enter in the SAS Data Loader configuration screen, and in general how Hadoop handles users, so I thought I would share a bit about this topic.
First, there is a good article in the Cloudera documentation on this topic: Authorization and Authentication In Hadoop | Cloudera Engineering Blog . To summarize, Authentication is the process of determining whether someone is who they claim to be. Out of the box, if Hadoop is configured with all of its defaults, Hadoop doesn’t do any authentication of users. Hadoop has the ability to require authentication, in the form of Kerberos principals. Kerberos is an authentication protocol which uses “tickets” to support authentication.
SAS Data Loader supports both modes of authentication. The Kerberos mode is significantly more complicated, so I will not go into that in this article; I will save that topic for another time.
For non-Kerberos mode, Data Loader expects that the user provided in the configuration screen is one that exists on the cluster and has at least the following permissions:
1. Read/write/delete files in the HDFS directory (used for Oozie jobs)
2. Read/write/delete tables in Hive
Why are these permissions needed? Here is the explanation for each of them.
The first permission is required for the directives, Copy Data To and Copy Data From Hadoop to work. The way Copy Data to Hadoop works is to call Oozie to actually run the job on the Hadoop cluster using Sqoop. To do this, it creates a temporary directory in the HDFS, uploads some files to this directory, and then starts an Oozie job. It cleans all this up after the run. So the user who is configured in Data Loader has to be able to have enough permissions to do these steps.
Hive permissions are needed because SAS Data Loader will perform drop, recreate, and append actions on data when working with directives. The user has to have enough permissions to support these actions.
Hope this helps clarifies how permissions are needed for the user on the configuration screen.
... View more
- Find more articles tagged with:
- Production Software
Labels:
12-22-2011
04:09 PM
I usually start people off by directing them to the products pages off of http://support.sas.com, the same place you used to get to this communities page. You can navigate to the products section and there is a link to the DI Studio page which has doc, pointer information, training examples, etc. Here is the direct link: http://support.sas.com/software/products/etls/index.html . There are training classes that you can attend, and an online user guide that you can browse through. In addition, there are many papers available from SAS User Forums that can be helpful that discuss DI Studio and contain lots of information around best practices, tips and tricks, etc. You can access the papers from the same website, http://support.sas.com, click on Knowledge Base, Papers, and then select Conference Proceedings on the top RHS. Here is the direct link: http://support.sas.com/events/sasglobalforum/previous/online.html Hope this helps!
... View more
12-22-2011
04:02 PM
In 9.3, the answer to this question is yes, we have added a command line interface for deploying di jobs programmatically. You can reference the online help or di studio users guide to see the specifics, and there is a wrapper script that handles the details that you can call from a command line.
... View more