I looking for a solution to extract data from OCI object Storage. We have files from Oracle ERP and storing them in Object Storage. Wanted to extract this data and load it into Snowflake system. We have AWS s3 integration with Snowflake. Looking for options to build an integration from OCI object storage —> AWS S3 —> Snowflake. We have a requirement to load data into Snowflake as nightly batch and near real time data processing. Can please share your thoughts on building this integration?
Looking for help who are familiar with the process.
Snowflake doesn’t have direct integration with OCI object storage, S3 integration option did not find any AWS documentation.
2
Answers
Have you considered the use of something like RClone to sync Object Storage with S3? RClone blog post. You could use something like OCI Container Instances to keep the cost to a minimum.
There’s also Oracle Integration which may allow you to create a data feed from OCI Object Storage. I’ve not used it but it appears you may be able to feed data directly to Snowflake and bypass AWS S3 altogether.
OCI object storage API is compatible with AWS object storage API. Depending on how you are extracting the data in your pipeline, you would potentially only need to change the url and not even have to do the migration from OCI Object Storage to AWS S3. Here is an article from Oracle that discusses the pre-requisites of this approach.
https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm#usingAPI