firecrawl

An open-source web scraping tool that converts websites into clean, LLM-ready data formats like markdown.

Firecrawl

Web scraping has just changed forever. Do you want to scrape data from the internet? Then you have to get to know Firecrawl. Firecrawl is a large language model based scraping tool, and it allows you to extract data from a website without knowing anything about the HTML of this website. 

The website that we will be scraping today is this one, and this is a website that I made specifically for this exercise. So you will see that we have hotels here. So we have hotel names, we have the location of the hotel, and we also have a rating from 1 to 5. 

Then, um, for this specific website, we have five pages. So you will see that we have five pages full of hotels, and I'm as well going to show you how you can use Firecrawl and OpenAI to scrape multiple pages. 

Then let's take a look at the website of Firecrawl. Firecrawl comes with a free plan, which allows you to scrape approximately 500 pages. As you can see here, the only thing you have to do is create a free account. Then you will get to this website, to the dashboard. 

Then you navigate to API keys, and then you can just copy here the API key, and you can use it in the script that I'm about to show you.

Started with installing the necessary libraries. So in requirements.txt you will see that you need to pip install file cr.

Pii open A.V pandas and Open PI XL. Then I also have a EnV file where I'm storing both the API key and as well the open AI um API key like like this. So just put them in here and then in main.py I have the full script. As you can see, I'm enabling all these libraries here. I'm loading EMV to connect with the EMV file where I've stored the API keys. 

I'm creating a list with items and this is where I'm going to store the items that I have scraped. Um, I'm in initiating app which is the file C application and this here is um page content. This is the actual data that we scrape from the web page. 

So the only thing that filec does is it takes data from the web page. Because if you scrape a web page, you will get HTML. But if you send this HTML directly to open AI to CH GPT, you will burn a lot of tokens. So the only thing that Firecrawl does is it takes the HTML that you have scraped from the website, it removes the HTML, it sends the content of the web page but without the HTML back to you. 

So you can then process it further and the benefit here is that if you send the HTML to open AI to J GPT, you'll burn a lot of tokens. So without the HTML you.

Keep about let's say like 30 40% of the um data on the website so you will save a lot. So I'm just going to put page content on the screen so you can see what we get back from Firecrawl. Let's run the script and this is the data that we get back from Firecrawl. 

So you will see that we have all the hotels here. So we have the name of the hotel, we have the location, we have the rating, etc. etc. So this is all the content of the website without the HTML. 

And as you can see here, I as well have an option here, page options only main content, which I set to true. This makes sure that you do not receive any information about the header of the or the footer of the page. You only receive the main content of the page and this is why Firecrawl is such an amazing tool to be used because it saves you a lot of tokens and as well a lot of money. 

Um then we're going to initiate open a because at the end we're still going to use a large language model to process whatever we get back from fir craw because we still have to put this in a structured form. So um here I'm initiating open AI then I'm providing the fields. So the fields that I want to get back is Hotel name.

Hotel location and hotel rating and this is of course matching with the data of this website here. So we have the name, the location, and the rating. So if you want to scrape any other page, the only thing you have to do is fill in the fields here that you want to get back. 

Then I have the system prompt and user prompt. So the system prompt is something like you're a helpful assistant. Uh, you receive a scrap web page and you extract the items and return them. Val Jason, you return a list with all fields and of course you can change this if you want. 

And then the user prompt is the extracted web page and this is the content that we receive back from Firecrawl and the fields that you return. And this is then referring to the fields that I have provided here. So this is where I provide the output that I want to get from OpenAI. 

So this is where I'm making the actual call to jgpt 40. Of course, you can change this for any other version of OpenAI. The response format, I have another video on the response format. The type is Json object, so I just want to get back the response in Json. Temperature is zero and this says that um, so but so zero is like.

Zero creativity and if you set this to two uh the model becomes very creative. Never set it to two. But if you wanted to like create a video ID or something like that you usually set it to 1.7. Here I don't want the model to have any creativity because I just wanted to provide me back um uh the structure data in the um formats that I provided this right. 

So creativity should be zero temperature zero and then it provide you the messages. So the system prompt and the user prompt then what I get back from uh the CH GPT API is this completion ver a zero message content. So I'm going to convert this Json string here to a valid python object. I'm doing that here and in this part here I'm checking if I'm getting back any dictionary. 

Then I'm going to remove the first key. So normally what you get back from jgpt is something like a dictionary with hotels and then as the value of this dictionary you get back a little list of all the hotels. Or if you scrape animals you will get back a dictionary with animals and then a list with all the animals as value. And I don't want this first dictionary I just want the list with hotels or animals or whatever.

You you guys are going to scrape. So that's why I'm removing this first key here. Then what I keep is just the list. Then I'm going to loop through every item in this list and I'm going to append this list item to um items. 

And then items is what I'm writing back to an Excel file or a CSV file. You can just uh comment out here what you don't need or just remove the uh line entirely. 

And these are the files that we created. Um I just run this script and you have seen the content that we got back from Firecrawl. And at the end uh if I open hotels.

Xlsx here you will see of course that I get back all the um hotel information here. So I get back the hotel name, the location and as well the rating. Um if I open the CSV file of course you see the same information like this. I've already seen an example of how to scrape one page with this script, but what most people want to do while web scraping is of course scrape multiple pages. 

So let's also see how we can change a script in order to script multiple pages. So I'm going to add um a list here with pages. So with range to six, um I'm basically creating a list with the items one to five. Um a script has five pages, so we can then iterate through that list and scrape every page. 

So pages is one to five and with range 1 to six we create a list with all the items between uh one and five. So 1, 2, 3, 4, five. And then let's also create a loop. So four page in pages and let's add some space here for everything till here. 

So for page in pages and then I'm going to add here and page is equal to make sure that you convert this integer to a string page. So we are now iterating to uh pages which um has the items 1, 2, 3, 4, 5. Of course if you want to scrape every.

Page until like let's say 100 you just add one and them one here and I will scate 1 200. But for now I'm just going to leave it to six to scrap one to five. Um let's see if we have everything okay. And of course um make sure that you modify the fields here. 

Uh just provide the fields that you want to extract from the page you are scraping. Uh and then you your script should work. Um let's try this out so let's run a script. So the script is finished of course it takes a bit longer than the previous script which only scraped one page. Um so let's open open hotels.

Xlsx and will'll see that we have all the hotel names. Hotel locations and as well the ratings. The first hotel is the Grand Plaza Hotel and the last hotel is the Grand Hotel. So let's make sure that that's correct. 

So the first hotel on the page is the Grand Plaza Hotel and the last one is the ground toel. So that's entirely correct. Now if you want to run the script yourself, you can find the entire script on my website and there is a link to this website in the description of this uh video.

Note, that there may be slight differences in text and video. Watch video for the original content.

Firecrawl Video