I want to mount my ADLS2 storage into a Databricks workspace without sharing the data with every user in the workspace. We have multiple teams working on different clusters in the same workspace so we want to keep the data…
We want to enable the access for the users to create token. Now, I have logged in to the databricks account using the admin access, and I go to "Access Control" -> "Personal Access Tokens" and I want to enable…
in Azure databricks, need to connect to a table in odbc sql server and create a dataframe and create exactly the same table in databricks I have already completed the below codes and I can see as attached. How to…
In Pyspark, I have 2 dataframe. 1st dataframe say df1 is empty dataframe created from a schema. 2nd dataframe df2 is non-empty dataframe filled from a csv file. Now I want to merge such that all below scenarios are covered.…
Using autoloader, I am reading some continues data from storage to Databricks Delta Live table. The declaration of data pipeline is as follows. import dlt from pyspark.sql.functions import * from pyspark.sql.types import * sch = "StructType([StructField('Date', StringType(), True), StructField('machine', StringType(),…
I am trying to run the following code iter_max = 100 CC = 1 save_model = False with mlflow.start_run(run_name=run_name, experiment_id=experiment.experiment_id, nested=True) as run: text_clf = Pipeline([ ('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf',LogisticRegression(max_iter=iter_max, C=CC))]) text_clf.fit(X_train, y_train) run_id = mlflow.active_run().info.run_id` But I get…
I have below pyspark row type data: indv_msg = [Row(cbm_json_output=Row(country_code='USA', date='06-10-2023', date_epoch='1696550400', id='USA-001535-1696550400', interfaceVersion='1.0.0', opmode_car_door=Row(health_category='GREEN', msg_id='1', num_yellow_preds_in_last_14_days=0, reason=None, reasonDetail=None), opmode_landing_door=Row(health_category='GREEN', msg_id='1', reason=None, reasonDetail=None), sensor=Row(component_type=None, health_category=None, landing_priority=None, msg_id='1', num_yellow_preds_in_last_14_days=None, reason=None, reasonDetail=None), unit_id='001535')) While trying to convert to json string, it is…
I am trying to connect to sql server from databricks using pyodbc facing below issue import pyodbc sql_server_url='server_url' sqlDB='db-dev' SQL_DB_USER='user' SQL_DB_PASS='123' conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=' + sql_server_url +';DATABASE=' + sqlDB + ';UID='+SQL_DB_USER+';PWD='+SQL_DB_PASS) Error: ('01000', "[01000] [unixODBC][Driver…
I am trying to import python files from databricks notebook. I am able to import the function but it keep giving me error - NameError: name 'col' is not defined. I have a file /Workspace/Shared/Common/common.py. The code in that file…
Hi It seems I could not load data correctly from DBFS is Databricks using Auto loader, at least it is not displaying the data -- 'Query returned no results'. Any help is welcome!