I have a bunch of device_id (roughly 300million), I want to create a full video resolution list [‘360p’, ‘480p’, ‘540p’, ‘720p’, ‘1080p’]for each device_id like
device_id
1
2
device_id video_resolution
1 360p
1 480p
1 540p
1 720p
1 1080p
2 360p
2 480p
2 540p
2 720p
2 1080p
I wonder if it's possible to create these record in sql?
I've tried to use spark, but I do not have that many resources and often run into OOM issue.
2
Answers
Try this: the cross join would give you all the combinations. I am not familiar with the internals of PostgreSQL but generally the
CREATE TABLE
statements are ‘logged’ less, thefore it may not run out of memory (you may run out of target disk space, of course).You don’t need the first two statement, of course, and you need to adjust the table names.