Looks so good!   Nice job team! ... View more

Hello Everyone. My name is Nancy, and I am a computer and data scientist here at SAS.  I did my University thesis on solar energy forecasting and am looking forward to assisting on this project.  Here are some findings from that research that you might find helpful, and I am happy to send along the paper.   I have also provided some additional references that you might find helpful.      - One important surprise predictor is air quality.  I found up to a 10% reduction in energy output at certain times of the year based on poor air quality.  Our area here in the east coast of the united states is bathed in particles from coal fired power plants.  In summer months the atmosphere mixes more readily and so the impact of these particles are not as pronounced.  However, during winter months, the atmosphere is more stratified, and the particles do not mix or dissipate as readily, particularly in early winter.  During this time, solar irradiation is decreased due to these particles.  Finally, this is very much a locality based predictor.  Information related to air quality can generally be collected from government sources;  in our country, daily numbers for most localities are freely available from the US Environmental Protection agency.    - You have already mentioned other useful predictors above including irradiance, date/time, temperature, solar azimuth, and windspeed.  Wind was particularly interesting as a predictor;  in our area, days of high wind produced less output, due to the turbulence and cooling effect associated with wind flowing over the panels.  These numbers are available from government weather data sites.  Note that we found it to be important to use meteorological data collected as close as possible to the installation site where the panels are located.    Another finding from our research into this area is the modeling techniques that can produce better accuracy with this type of data.  We found that Random Forest is a good method for this type of forecasting because it support highly non-linear data that exhibits multi-collinearity.  Deep learning methods that incorporate time series, such as RNN and Gradient Boost, are also quite efficient as predictive models, at the expense of more complexity and greater processing time.  Since solar energy output produces an abundance of data, it lends itself very well to deep learning techniques.     Here are a few references you might find helpful:   https://www.researchgate.net/profile/Zhenxing-Lei/publication/336184094_A_review_of_deep_learning_for_renewable_energy_forecasting/links/60014485299bf14088975ffd/A-review-of-deep-learning-for-renewable-energy-forecasting.pdf   Abdel-Nasser, M., & Mahmoud, K. (2017). Accurate photovoltaic power forecasting models using deep LSTM-RNN. Neural Computing and Applications , 1-21.   Chandler, D. (2018, August 18). Air pollution can put a dent in solar power. Retrieved from MIT news: http://news.mit.edu/2018/air-pollution-can-put-dent-solar-power-0829   Hickey, H. (2018, July 24). WHY WINTER AIR IN THE EASTERN U.S. IS STILL SO DIRTY. Retrieved from Futurity.org: https://www.futurity.org/winter-air-pollution-emissions-1819872/   NCSU. (2019, May 8). Weather and Climate Database. Retrieved from North Carolina Climate Office: https://climate.ncsu.edu/cronos/?station=REED NOAA. (2019, May 10). Solar Geometry Calculator. Retrieved from NOAA.gov: https://www.esrl.noaa.gov/gmd/grad/antuv/SolarCalc.jsp    US Environmental Protection Agency. (2019). Air Quality Daily Values Report. Retrieved from US EPA: https://www.epa.gov/outdoor-air-quality-data/air-quality-index-daily-values-report    Zulkifli, H. (2019, Mar 12). Multivariate Time Series Forecasting Using Random Forest. Retrieved from Towards Data Science: https://towardsdatascience.com/multivariate-time-series-forecasting-using-random-forest-2372f3ecbad1    ... View more

I tried the UNIVARIATE approach on a wide table of 7500 columns (35 rows), and UNIVARIATE ran in 300s.  All other techniques I tried on this table ran much much slower.  I think the Univariate approach is very powerful.  Here is my code just as an example.     ods listing close; ods output frequencies=_afreq; proc univariate data=sashelp.cars NEXTRVAL=20 freq; var _numeric_; run; ods listing; ... View more

BDN participates in the SAS Web infrastructure platform, which means that the migration/promotion tools should work as well to move BDN content between environments.  This is part of the SAS Deployment tools such as the SAS Deployment Manager (SDM).  I found this powerpoint presentation that describes the tooling that exists to manage content in the 9.4 platform, which includes all of the BDN related content.  https://www.sas.com/content/dam/SAS/no_no/doc/other1/FANS_Oslo_11mars_SASDeploymentBackupUtility.pdf.  This should make it easy for you to move content around.      ... View more

The cluster node essentially does essentially the same thing and should run just as fast.  GIve that a try and see if it meets your needs.   ... View more

One additional comment, if you use the SAS in-database hadoop code accelerator to read/write the data, Avro, Parquet, and Sequence files are automatically compressed using Snappy, assuming it is enabled on your hadoop system.  The DBCREATE_TABLE_OPTS setting is also available on the table properties in DI Studio.     Nancy   ... View more

As Ron indicates, yes it is possible, and the syntax can be tricky.  Here are a few things to try, and be aware of.    - Use the execute function.  It is the one that you need to use to call some external program - The command is running locally, so you have to be able to run commands on the host you are trying to execute the script on - Check that you have permissions set, and that you can get to the code from the program   You can try this.  Create some simple program, and just call that.  Verify that your expression code is calling that program correctly.  I would put it in the same location as Excel is to make sure you can get to the directory that you are trying to call out to with your more complex call.   If you have verified that and things still are not working, then there is a handy microsoft tool called procmon:  https://technet.microsoft.com/en-us/sysinternals/bb795533.aspx.    Its a handy way on a windows machine to see all files and processes being used.  It can help figure out what is happening on the host you are working on to see where things might be failing.    Hope this helps,   Nancy  ... View more

I usually start people off by directing them to the products pages off of http://support.sas.com, the same place you used to get to this communities page.  You can navigate to the products section and there is a link to the DI Studio page which has doc, pointer information, training examples, etc.  Here is the direct link:  http://support.sas.com/software/products/etls/index.html  .  There are training classes that you can attend, and an online user guide that you can browse through.  In addition, there are many papers available from SAS User Forums that can be helpful that discuss DI Studio and contain lots of information around best practices, tips and tricks, etc.  You can access the papers from the same website, http://support.sas.com, click on Knowledge Base, Papers, and then select Conference Proceedings on the top RHS.  Here is the direct link:  http://support.sas.com/events/sasglobalforum/previous/online.html Hope this helps! ... View more

In 9.3, the answer to this question is yes, we have added a command line interface for deploying di jobs programmatically.   You can reference the online help or di studio users guide to see the specifics, and there is a wrapper script that handles the details that you can call from a command line.  ... View more