I’m trying to run a simple code example to train and predict a ML algorithm using the Opensearch ML Commons plugin on a AWS managed Opensearch cluster. The cluster is on OS version 2.5, which definitely supports the plugin. The code example can be found here .
When I run the sample code in my console I get the following error message:
"type": "m_l_resource_not_found_exception",
"reason": "No eligible node found to execute this request. It's best practice to provision ML nodes to serve your models. You can disable this setting to serve the model on your data node for development purposes by disabling the "plugins.ml_commons.only_run_on_ml_node" configuration using the _cluster/setting api"
However after trying to update the settings I get _cluster/settings payload not allowed
, it doesn’t seem like the AWS OS version supports the cluster/settings API except for these
/_cluster/settings for several properties:
action.auto_create_index
action.search.shard_count.limit
indices.breaker.fielddata.limit
indices.breaker.request.limit
indices.breaker.total.limit
cluster.max_shards_per_node
A few questions:
- Am I correct in my assumption that I can’t update this cluster setting, meaning that I can’t run ML Commons on an AWS version of Opensearch
- Is there a way to configure an ML node on a AWS OS. All examples seen require you to update the opensearch.yml file, which I can’t do because I’m using a hosted version.
- Is there some security permission that I don’t have that could allow me to change these cluster settings?
Thanks!
2
Answers
Amazon OpenSearch Service supports ML Commons for versions >= 1.3
One of three things is likely occurring here:
Instance Types: certain operations (e.g. anomaly detection, ML Commons) are not supported on all instances. While AWS do not specify which instances are supported for ML Commons, it is likely that
t2.micro
andt2.small
nodes will not be supported — this is the case with anomaly detection.Access Control: you must enable fine-grained access control (FGAC) in order to use non-standard plugins. kNN, SQL, and other CRUD-oriented actions, of course, are supported without FGAC. Although there is no documentation to confirm this, I imagine that ML Commons, as with Anomaly Detection, requires FGAC as the plugin involves significant modifications to cluster/node configuration.
ml_full_access
backend role to your user, even if you already have theall_access
role. More on this here.Formatting of
_settings
API call: Amazon’s flavour of OpenSearch requires “PUT requests to the _cluster/settings API that use the "flat" settings form” [source]Notice Neural Search is an experimental feature, you can send an email to [email protected] to enable Neural Search feature for your account as explained in this link.