Thanks everyone for your help and support. The issue was resolved with the help of SAS Tech Support. Thanks SAS Tech Support for the solution. The issue was that I was using the jdbc engine whereas SAS Access Interface to Spark provides connectivity to Databricks. The details are given here SAS Help Center: Support for Spark on the SAS Viya Platform. Look for the section on Databricks. One would need the details about server, httpPath and PersonalAccessToken among other things.
The SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment.