I am still new to Redshift service and quite confused of when to use or what data to put into Spectrum.
Suppose I have star schema data warehouse on Redshift, should I put fact table or dim table into Spectrum(external tables from s3) for storage optimization?
Or typically data warehouse has different layers eg: landing, staging, or data vault. Should we put other layers of data into Spectrum only leave star schema data into Redshift.
And since data in S3 is appended only, do we need install apachi hudi or delta lake to work with Redshift Spectrum?
I found an aws article :https://aws.amazon.com/blogs/big-data/10-best-practices-for-amazon-redshift-spectrum/ states below, but still not clear.
A few good use cases are the following:
- Huge volume but less frequently accessed data Heavy scan- and
- aggregation-intensive queries Selective queries that can use
- partition pruning and predicate pushdown, so the output is fairly small
Anyone can provide real world example to explain? thanks
2
Answers
In general, put everything into ‘normal’ Amazon Redshift.
Redshift Spectrum is handy for accessing data stored in Amazon S3 without having to load it into the Redshift cluster, but it will not be as fast as accessing data stored in ‘normal’ Redshift.
Therefore, it is useful for rarely-accessed data or for one-off queries on a dataset without having to import the data into Redshift.
Do not use Spectrum as part of your normal ETL flow. One exception to this might be if you are receiving ‘landing’ data via Amazon S3 (eg Seed Files) — rather than importing the tables into Redshift, they could be referenced via Spectrum. However, normal loading tools such as Fivetran can load the data directly into Redshift, which is preferable to using Spectrum.
This is a broad topic but I’ll give a few thoughts.
First off Spectrum is a (often large) set of compute elements embedded in S3 that can do some aspect of the query plan. These part centered around applying WHERE conditions and performing aggregation (GROUP BY). There are also aspects of the query plan that cannot be perform in the S3 layer such as JOINs and advanced functions such as window functions.
The next thing to understand is that while these embedded compute elements are close to S3 in terms of access speed, the S3 service is far away from the Redshift cluster (network distance). If the large amount of data stored in S3 can be pared down to a small set that is shipped to Redshift then Spectrum can be a huge performance improvement. However, if the large amount of data stored in S3 needs to be moved to the Redshift cluster completely to perform the query then there can be a large hit to performance.
Spectrum can be a huge benefit; allowing for a very large amount of data to be filtered down quickly by a fleet of small compute elements. This can result in a big win in performance and in the amount of data that can be addressed.
With these in mind you will want to have data in Spectrum that your query plan will want to get a subset transferred from S3 to redshift. This in general will apply to your fact tables and not to your dim tables. However, if your queries aren’t going to apply a WHERE clause to the fact table or aggregate the data down then you won’t see the advantages. Also for this to work the WHERE clause needs to apply to a column in the fact table as JOINs cannot be done in S3 so filtering on dim columns won’t help. Similarly and GROUP BY needs to be applied only on the fact table columns or this won’t reduce the data coming to Redshift from S3.
So fact tables.
Data generally gets into Redshift through S3 and this can be done with the COPY command. You can also get data into Redshift from S3 using Spectrum. This can be a useful tool if other tools are also using S3 for this shared data. S3 can seem like a common data store for separate data systems. This can be useful for some data solutions.
You also bring up very large, infrequently used data. Like older historical data that is usually needed but is sometimes needed. This can be helpful in that older data can be offloaded from the Redshift cluster and the access time for this data isn’t important as it is very infrequently used. There is a potential issue – The Redshift cluster can only work on a certain size of data given it’s disk space and memory. So you can clog up your cluster if the amount of historical data is too large. This may mean that looking at the full set of historical data in one query may not be possible. Again if the data is aggregated or filtered in S3 this issue isn’t a problem.
Bottom line – Spectrum is a great tool but isn’t the right tool for every problem.