I would like to use some publicly available data from a government website as a source of data in an iOS app. But I am not sure what is the best / most polite / scalable way have a large number of users request data from this website with the least impact on their servers and best reliability for me.
- It is 1-50kb of static XML with a fixed URL scheme
- It updates with a new XML once a day
- New users would need to download past data
- It has a Last-Modified header but no caching headers
- It does not use compression or a CDN
- It’s a government website, so if someone even replies to my email I doubt they are going to change how they host it for me…
I’m thinking I could run a script on a server to download this file once a day and re-host for my app however my heart desires. But I don’t currently run a server which I could use for this and it seems like a lot just for this. My knowledge of web development is not great, so am I perhaps missing something obvious and I just don’t know what search terms I should be using to find the answer.
- Can I point a CDN at this static data somehow and use that?
- Is there something in CloudKit I could use?
- Should I run a script on AWS somehow to do the rehosting without needing a full server?
- Should I just not worry about it and access the data directly??
2
Answers
I think the best way for you is to create an intermediary database where you can store your data in a secure manner.
Also make sure you are not violating any data policies in the process.
I hope this helps.
You can use the AWS S3 service (Simple Storage Service).
The flow is somewhat like this:
If the file doesn’t exist on S3 yet, or, if the creation date of the file on S3 is yesterday, the iOS app downloads the XML from the gov site and stores it in S3.
If the file exists on S3 and is up to date, download it from S3.
After that, the data can be presented by the app without overloading to the site.