Skip to main content
Untangling the web using the simplest language possible.


Posts

Showing posts from August, 2018

JavaScript Closures: A use case

I recently started working for a small dev agency here, and the first item I was supposed to take over was a data scraping tool.  If you don't know what that is, basically it's going through websites and targeting specific elements and grabbing the data off of them.  How do you do that? Well, you can use something like Puppeteer to run an automated Chromium browser that can carry out tasks such as "navigate to this page" and "click the element with this CSS selector", etc.  In this project we were grabbing XHR responses from the server (if you open your browser dev tools, go to 'network' and click 'XHR' you can see requests and responses, their headers, etc) and then saving the JSON in the responses to our database (a SQL database). Puppeteer has a handy method that places a listener for requests or responses.  So any time the browser receives a response it would fire off a callback.  It was within this callback that I had an async func