I am trying to run the following code iter_max = 100 CC = 1 save_model = False with mlflow.start_run(run_name=run_name, experiment_id=experiment.experiment_id, nested=True) as run: text_clf = Pipeline([ ('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('clf',LogisticRegression(max_iter=iter_max, C=CC))]) text_clf.fit(X_train, y_train) run_id = mlflow.active_run().info.run_id` But I get…
Can a table be created in Azure Databricks pointed to azure storage account in delta format with fixed varchar(x) and timestamp_ntz (no time zone) data type ? I am using a script similar to below and it doesn't like the…
Hi It seems I could not load data correctly from DBFS is Databricks using Auto loader, at least it is not displaying the data -- 'Query returned no results'. Any help is welcome!
I'm working with a Delta table in Databricks that is partitioned by a Year column, and I have a timestamp column in the table. I want to optimize my queries to fetch data within a specific timestamp range (e.g., between…
The following bicep deployment has suddenly started failing and I do not understand why var managedResourceGroupName = 'rg-metadata-${dataBricksName}' resource databrickworkspace 'Microsoft.Databricks/workspaces@2018-04-01' = { name: dataBrickName location: location sku: { name: dataBrickPricingTier } properties: { managedResourceGroupId: managedResourceGroup.id parameters: { enableNoPublicIp: {…
I am using databricks for a specific workload. This workload involves an approx of 10 to 200 dataframes that are read and written to a storage location. This workload can benefit from parallelism. The constraint i have is cost optimization.…
I am using the Databricks VSCode extension for development in an IDE. The basic functionalities are all working well. I connected to an Azure Databricks workspace with Unity Catalog enabled, selected an active cluster (DBR 13.2) and configured the sync…
I have these following inputs. 1.Endpoint 2.Database 3.Collections Based on these inputs needs to establish a connection with Azure Cosmos Db from databricks Tried this approach: installed importted com.azure.cosmos.spark maven package in databricks cluster with the DBR of 11.3LTS and…
I have an array type column subscription in my table with columns Sample Table +-------+--------+----+----------+ |user_id| Subscription|Activity | +-------+-------------+----------+ |111 |[A,C,B] |20 | |-------+-------------+----------+ |222 |[C,A] |10 | |-------+-------------+----------+ |333 |[B] | 5 | +-------+-------------+----------+ I want to change my…
Question: When doing a Purview scan on Azure Databricks, I get the following error. What could be the cause of the error and how we can fix it? Remarks: For safety, I have replaced the 16 digits in the error…