All the data can be yours: reverse engineering APIs (jero.zone)

Article summary

The article discusses the process of reverse-engineering APIs to access data from various websites, including universities and other organizations. The author shares tips and techniques for finding and using these APIs, such as searching GitHub, looking for fetch requests, and using tools like GraphQL Playground. The author has used these methods to build unofficial software for their schools, including a university shuttle tracker and a course catalog. The article highlights the potential benefits of accessing data through APIs, including easier access to structured data and reduced need for web scraping.

Main themes

• API reverse-engineering
• Web scraping
• Data access
• Internet citizenship
• API security
• Adversarial interoperability

What commenters say

• Reverse-engineering APIs without permission is seen as unwanted and potentially abusive by some, who argue that it can impose unnecessary costs on website owners.
• Others argue that using APIs in a responsible manner, without imposing excessive traffic or harm, is a legitimate and beneficial practice that can coexist with website owners' interests.
• Some commenters believe that website owners have a responsibility to secure their APIs and prevent abuse, rather than relying on users to obtain permission or follow specific guidelines.
• There is disagreement over the concept of 'internet citizenship' and whether users have a right to access and use APIs in ways that website owners may not intend or explicitly permit.
• A few commenters argue that the ability to reverse-engineer APIs and access data is essential to the internet's value and that restricting it would stifle innovation and progress.
• Some believe that the onus is on website owners to clearly communicate their API usage policies and terms of service, rather than relying on implicit expectations or restrictions.
• Others suggest that the use of custom user agents and transparent communication can help to build trust and facilitate cooperation between API users and website owners.
• There is a concern that over-restricting API access could lead to a less open and less useful internet, where users are forced to rely on proprietary client apps and limited to specific use cases.