Deserializing JSON (without dynamic) where array has multiple possible depths based on value of another property?

I am trying to deserialize the following JSON without using dynamic as it is not supported by my system: https://raw.githubusercontent.com/vansha/Two10.CountryLookup/master/Two10.CountryLookup/region-list.json

Each line presents a region for geolocating with.

Using the JSON Converter here: https://json2csharp.com/

I get the suggestion for data classes as:

    // Root myDeserializedClass = JsonConvert.DeserializeObject<RegionJsonObject>(myJsonResponse);

    public class Geometry {
        public string type { get; set; }
        public List<List<List<List<double>>>> coordinates { get; set; }
    }

    public class Properties {
        public string name { get; set; }
        public string type { get; set; }
    }

    public class RegionJsonObject {
        public string type { get; set; }
        public Properties properties { get; set; }
        public Geometry geometry { get; set; }
        public string id { get; set; }
    }

The problem is that if you look at the JSON, there are actually two possible formats for the coordinates array. If the Geometry.type is Polygon we have three levels of depth [[[ ]]] and if it is MultiPolygon we get four levels of depth [[[[ ]]]]. So I am instead considering a format of the following with a custom JsonConverter:

    class GeometryConverter : JsonConverter {
        public override bool CanConvert(Type objectType) {
            return (objectType == typeof(Geometry));
        }

        public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer) {

            Geometry geometry = new Geometry();

            JObject jo = JObject.Load(reader);
            geometry.type = jo["type"].Value<string>();

            if (geometry.type == "Polygon") {
                geometry.coordinatesPolygon = jo["coordinates"].Value<List<List<List<double>>>>(); //DOESN'T WORK
            }
            else {
                geometry.coordinatesMulipolygon = jo["coordinates"].Value<List<List<List<List<double>>>>>(); //DOESN'T WORK
            }

            return geometry;
        }

        public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer) {
            // If you want to support serializing you could implement this method as well
            throw new NotImplementedException();
        }
    }

    public class Geometry {
        public string type { get; set; }
        public List<List<List<List<double>>>> coordinatesMulipolygon { get; set; } //multipolygon
        public List<List<List<double>>> coordinatesPolygon { get; set; } //polygon
    }

    public class Properties {
        public string type { get; set; }
        public string name { get; set; }
    }

    public class RegionJsonObject {
        public string type { get; set; }
        public string id { get; set; }
        public Properties properties { get; set; }

        [JsonProperty("geometry")]
        [JsonConverter(typeof(GeometryConverter))] 
        public Geometry geometry { get; set; }
    }

}

However, this is not working. Trying to set geometry.coordinatesPolygon or geometry.coordinatesMultipolygon in this manner gives the error:

InvalidCastException: Cannot cast Newtonsoft.Json.Linq.JArray to Newtonsoft.Json.Linq.JToken.

Am I on the right track with this approach? How do I get the arrays from the JSON into each of these two list objects as needed? Is there a better or more proper way?

Thanks for any help.