Question posted in 
I am working on a project that requires that we query a number of
different marketplace APIs (right now its just eBay and Etsy) and
store the contents in our DB – updating nightly.
Ideally, we would like to mirror the DB schemas on these sites
for convenience and consistency. I am wondering if anybody out
there has any thoughts about how to go about this? Can a schema
be inferred by what is provided from the web services (I am guessing
this is a big fat no). Short of that, are there any good examples
out there of what eBay and Etsy’s database structures might look like?
I know this is a long shot, but it’s worth a shot none-the-less.
Thanks for your time.
2
Answers
I would say that the schema would be very very similar to the web service, which in one respects answers your question.
But… I would suggest that you really don’t want to emulate the schema of eBay. eBay has been designed to work with a lot of data and has been de-optimized and has grown over time into a behomth. It is also not a good example of how a database should be structured and will cause you endless pain and tourment developing your solution based around it.
So if I were you I would only capture what information you require from both systems.
As one of the systems admins at Etsy I can tell you that you definitely do not want to duplicate our DB schema 🙂 At a high level, our DB infrastructure spans dozens of servers, and stores tons of data which isn’t exposed through the API.
There are also DB indexes and other architecture designs which exist to make fast DB lookups possible which you probably don’t want to duplicate either. For example, we split a lot of data across many MySQL servers (sharding), and then have another set of MySQL servers (indexing) which help us find data on the shards.
I would advise that you look at your own application needs and design your database schema around that.