My name is Corey Lewis, I'm a 30 year old web dev living in Portland, OR. I started writing this blog to increase my web presence and hopefully help some new people with understanding concepts that had always confused me. I hope you find it helpful!
I recently started working for a small dev agency here, and the first item I was supposed to take over was a data scraping tool. If you don't know what that is, basically it's going through websites and targeting specific elements and grabbing the data off of them. How do you do that? Well, you can use something like Puppeteer to run an automated Chromium browser that can carry out tasks such as "navigate to this page" and "click the element with this CSS selector", etc. In this project we were grabbing XHR responses from the server (if you open your browser dev tools, go to 'network' and click 'XHR' you can see requests and responses, their headers, etc) and then saving the JSON in the responses to our database (a SQL database). Puppeteer has a handy method that places a listener for requests or responses. So any time the browser receives a response it would fire off a callback. It was within this callback that I had an async func
Comments
Post a Comment