DATA Step, Macro, Functions and more

Calculate MD5 digest for a file without xcmd

Reply
Super Contributor
Posts: 441

Calculate MD5 digest for a file without xcmd

Dear SAS friends,

 

I am challenged with a conversion of a program from an enviroment where XCMD is allowed to one where it is not. The program creates a zip file with data and a MD5 hash of that data for the validation of integrity of that data once it is being transferred.

 

I have considered a few options:

1) Start a fight to get XCMD allowed. May take forever and succes is not guaranteed.

2) Use CALL MODULExx() using the OpenSSL libcrypto.so library

3) Implement the entire algorithm in datastep or DS2.

4) Ask the community hoping someone has done this already.

 

Of course there may be more.

 

I understand an MD5 hash can be calculated by splitting a message (file) into chunks and adding up the checksums of the chunks. Most solutions seem to involve numeric operations on a 128 bits structure (unsigned int) where we only have 64 bits. So using the MD5() datastep function, however tempting, may not be the solution. Unless of course someone has cracked this.

 

I hope someone can steer me in the right direction.

 

Looking forward to your suggestions,

- Jan.

Super User
Super User
Posts: 7,994

Re: Calculate MD5 digest for a file without xcmd

Posted in reply to jklaverstijn

What version control system is on the new system?  SVN at least uses and calculates MD5 as part of its process.  If the version control software doesn't do it automatically, then they must have a import mechanism that does create MD5's to ensure that files uploaded to the new system match what was sent, just get it to store that MD5 and problem solved.  If not and you can't call system, then you have a bit of a problem.

Super Contributor
Posts: 441

Re: Calculate MD5 digest for a file without xcmd

Hi @RW9,

 

Thanks for chiming in. We do use SVN but it's not going to help here due to the way the existing process is designed. And SVN is only for the development of the software and not for keeping reports and othe output files. And no, we cannot call system; that's why I had to turn to you.

 

Regards Jan.

Frequent Contributor
Posts: 83

Re: Calculate MD5 digest for a file without xcmd

[ Edited ]
Posted in reply to jklaverstijn

This is kind of a shot in the dark since I've never personally tried this and I don't know what it's capabilities or limitations are, but could you use PROC GROOVY or PROC LUA?  There are libraries out there for both for calculating hash functions that you could use without having to code it yourself.  One concern would be if they are available or would have access to the filesystem if XCMD is disabled.  I've been wanting to explore the capabilities of both procs for some time, but haven't had a chance to get around to it.

Frequent Contributor
Posts: 83

Re: Calculate MD5 digest for a file without xcmd

Also, does everything need to be done from within SAS and what environment are you running from?  When I've dealt with stuff like this in the past I have usually relied on a shell script to call SAS and then do any post-SAS processing back in the calling script.  This also assumes you're able to run SAS batch jobs from command line rather than just EntGuide.

Community Manager
Posts: 2,955

Re: Calculate MD5 digest for a file without xcmd

Maybe this handy paper from Rick Langston?

 

Implementing Hashing Techniques in SAS®

 

http://support.sas.com/resources/papers/proceedings16/SAS2800-2016.pdf

Super Contributor
Posts: 441

Re: Calculate MD5 digest for a file without xcmd

Posted in reply to ChrisHemedinger

Very interesting @ChrisHemedinger. Thanks for that, I missed this paper. It does help if I choose to go the way of doing the entire MD5 calculation in SAS. And of course that's where Rick Langston comes out to play as well. At least it appears shows how to do it without the need for the long integers.

 

Regards Jan.

Super Contributor
Posts: 441

Re: Calculate MD5 digest for a file without xcmd

You're very right Chris, we may do this outside of SAS. It shows how SAS focussed we are that this is not a clearcut approach for us. But may in the end prove the silver bullet.

 

Regards Jan.

Super Contributor
Posts: 441

Re: Calculate MD5 digest for a file without xcmd

Yes @Sven111 this one caught me off guard. I had the code Groovy prepared, pressed F3 only to find out that proc groovy needs xcmd. I assume the same is true for proc lua. So good idea but no cigar.

 

Regards Jan.

Respected Advisor
Posts: 4,173

Re: Calculate MD5 digest for a file without xcmd

Posted in reply to jklaverstijn

@jklaverstijn

And if you want to start "the fight" for XCMD then this blog http://blogs.sas.com/content/sasdummy/2012/08/31/the-case-for-xcmd-privileges-in-sas-enterprise-guid... from @ChrisHemedinger could be helpful.

 

Super Contributor
Posts: 345

Re: Calculate MD5 digest for a file without xcmd

Posted in reply to jklaverstijn

jklaverstijn wrote:

Yes @Sven111 this one caught me off guard. I had the code Groovy prepared, pressed F3 only to find out that proc groovy needs xcmd. I assume the same is true for proc lua. So good idea but no cigar.

 

Regards Jan.


Good news: proc lua does not require xmcd, but i don't think that you can use any lib with lua in sas.

Frequent Contributor
Posts: 83

Re: Calculate MD5 digest for a file without xcmd

Posted in reply to andreas_lds

If you can't import libraries, could you just find the source code for one of the hash libraries and copy it directly into PROC LUA and use it that way?

Super Contributor
Posts: 441

Re: Calculate MD5 digest for a file without xcmd

Thanks for the suggestion. I have found some code in the mean time.
Ask a Question
Discussion stats
  • 12 replies
  • 248 views
  • 0 likes
  • 6 in conversation