skip to Main Content

I tried to save a PySpark dataframe in a SQL database in Synapse:

test = spark.createDataFrame([Row("Sarah", 28), Row("Anne", 5)], ["Name", "Age"])
test.write
    .format("jdbc")
    .option("url", "jdbc:sqlserver://XXXX.sql.azuresynapse.net:1433;database=azlsynddap001;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.sql.azuresynapse.net;loginTimeout=30;Authentication=ActiveDirectoryIntegrated")
    .option("forwardSparkAzureStorageCredentials", "true")
    .option("dbTable", "test_CP")
    .save()

I got the following error:

IllegalArgumentException: KrbException: Cannot locate default realm

Here is the detail of the error:

---------------------------------------------------------------------------
IllegalArgumentException                  Traceback (most recent call last)
/tmp/ipykernel_7675/2619697817.py in <module>
      1 test = spark.createDataFrame([Row("Sarah", 28), Row("Anne", 5)], ["Name", "Age"])
----> 2 test.write
      3     .format("jdbc")
      4     .option("url", "jdbc:sqlserver://XXXX.sql.azuresynapse.net:1433;database=XXXX;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.sql.azuresynapse.net;loginTimeout=30;Authentication=ActiveDirectoryIntegrated")
      5     .option("forwardSparkAzureStorageCredentials", "true")

/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py in save(self, path, format, mode, partitionBy, **options)
   1105             self.format(format)
   1106         if path is None:
-> 1107             self._jwrite.save()
   1108         else:
   1109             self._jwrite.save(path)

~/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py in __call__(self, *args)
   1302 
   1303         answer = self.gateway_client.send_command(command)
-> 1304         return_value = get_return_value(
   1305             answer, self.gateway_client, self.target_id, self.name)
   1306 

/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)
    115                 # Hide where the exception came from that shows a non-Pythonic
    116                 # JVM exception message.
--> 117                 raise converted from None
    118             else:
    119                 raise

IllegalArgumentException: KrbException: Cannot locate default realm

Where does this error come from? I googled everywhere but no way to find what I did wrong. I also see this post has the same problem as me.

2

Answers


  1. I got same error in my environment, I solved like this.

    Sample Data frame

    columns = ["language","users_count"]
    data = [("aa", "20"), ("bb", "10"), ("jk", "30")]
    

    Sample code

    df = spark.createDataFrame(data)
    df.write.jdbc('jdbc:sqlserver://<SQL_server>.database.windows.net:1433;database=vamsi', 'Table_name',
                      mode='overwrite',
                      properties={'user': 'User_name', 'password': 'User_password'})
    

    Ref1

    Output:

    Ref3

    Login or Signup to reply.
  2. Can you try using a dedicated synapse connector instead of JDBC, more details here

    Also if you are using databricks then you can try using databricks SQL DW connector details here

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search