Amazon web services – How to control databricks autoscaling from the driver node
I am using databricks for a specific workload. This workload involves an approx of 10 to 200 dataframes that are read and written to a storage location. This workload can benefit from parallelism. The constraint i have is cost optimization.…