I have stacked over the problem where I cannot understand how organisation secure theirs public apis from any person who collects data.
I know we use passport and other ways of auth tokens in order to protect private info from unauthorised user.
But there are things like public Search Engine that doesnt need user to be authenticated in order to find on Facebook a person via search or access Profile info page. Which means there are open public api’s that does not required authentication from user.
But going through couple of organisations I did not manage to get any public api request that I could access via Postman or simple via url.
So I am interested how do organisation protect their public api’s from requests.
How the frontend send requests to those public (kinda private api), or even if it has some sort of default api key for all sort of public api calls, how they secure those from people, if in our modern browsers we can access Local Storage or Cookies where we can extract that public api_token
I am confused over MERN STACK and Laravel + SPA React application.
Developing public routes for api calls they are all accessible from browser url or postman unless if the route is private and requires auth_token from passport or jsonwebtoken that is already requiring user to be registered. But I am trying to achieve in my application for users to be able to search and access Product Details without authentication.
But obviously I would not like if any sort of Big Data engineer would easily steal all public data from my web application unless if he is not lazy and doing html parsing for each public Product Details page.
So how do I secure those above described public api routes in my backend application.
And how does it do big organisations like Facebook, Google, LinkedIn and etc.
The reason why I am asking this question, because it is so easy to find some MERN Stack course and they will teach you how to handle authentication for authorised users and so one. Or even the LAMP technologies. But no one explains how to protect this data without requiring any user to be signed in.
Thanks you very much in advance to any person who is going to answer this long and very much confusing question.
2
Answers
A lot of the time I have seen APIs secured via Basic Auth or OAuth. When you use Basic Auth you send an Authorization header with a base64 encoded username and password. Headers are encrypted when you connect with HTTPS / SSL.
OAuth is a bit involved, but follows a similar idea. Your OAuth token is send via encrypted headers as well. You can read more about OAuth here: https://www.digitalocean.com/community/tutorials/an-introduction-to-oauth-2
When I was coding in PHP and working with Prestashop e-commerce I have used a Crawler/Bot similar to the one in this gist, but this can be easily spoofed, because it’s based on the
HTTP_USER_AGENT
. A better approach here is to use IP addresses to lookup the good know crawlers, aka the ones from searches engines like Google and Bing, but this will not work for keeping at bay the bad crawlers and bots. because they switch IP addresses very frequently.Companies like Facebook or even smaller ones, that have a good amount of resources at their disposal, use Artificial Intelligence(AI) to try to draw the line between Who is doing good and bad requests, and this type of software is known as User Behavior Analytics (UBA):
So this must be why you have such a difficulty in getting through the APIs of companies like Facebook, but that doesn’t mean it’s not possible, because Hackers have being do it, and the number of data breaches occurring every year in big companies are a proof of that.
Well that may be because of a common misconception among developers that don’t really understand the difference between Who vs What is doing the request to the API server.
The Difference Between WHO and WHAT is Accessing the API Server
I wrote a series of articles around API and Mobile security, and from the article Why Does Your Mobile App Need An Api Key? I will quote the following:
Think about the Who as the user your API server will be able to Authenticate and Authorize access to the data, and think about the What as the software making that request in behalf of the user.
So, in my opinion a lot of Developers are not aware of this difference between Who and What in a request, therefore they concentrate in solutions for the Who.
Possible Solutions
This organizations are using very sophisticated UBA solutions that may not be in the reach of every organization in terms of cost or because they are proprietary solutions, but other solutions exists, and you can read the section Defending the API Server on this other reply I gave to the question
secure api data from calls out of the app
to understand how you can gradually improve the security of an API server for a web app.If you also need to secure the API server for requests from a mobile app, then you can lock it down with a very high degree of confidence to your mobile app, by employing the Mobile App Attestation concept, and you can read more about in this reply I gave to the question How to secure an API REST for mobile app?.
Do you Want to Go the Extra Mile?
I cannot finish any response to a security question without give a reference to the excellent work from the OWASP foundation.
For Web Apps
OWASP Web Top 10 Risks
The Web Security Testing Guide:
For Mobile Apps
OWASP Mobile Security Project – Top 10 risks
OWASP – Mobile Security Testing Guide:
For APIS
OWASP API Security Top 10