For best results, as I am not familiar with the attached program for web crawling, I would try to use web crawling with our current production solutions. I would investigate using the Text Import node, which is part of the SAS Text Miner. If your company does not license SAS Text Miner, then another option would be to look at using SAS Information Retrieval Studio.