Tools for scraping dynamic sites

Crews
1

Hi everyone,

I am encountering some problems with the current tools scraping websites to retrieve possible tenders and bids in the US. I am looking for a three level tier: Federal (Sam.gov. fedconnect, federal state), state level and private entities.

A major problem the current tools are encountering (scraperwebsitetool, serper, etcs) is the dynamic nature of these sites. They always change as new tenders are put in and taken out.

Any ideas of relevant tools or even better, good libraries so I can build a custom tool?

Thanks,

Etai

2

Does something like firecrawl work or you are looking for a free alternative?

3

I prefer a free one.
Firecrawl is a second option

Etai Fiedelman

4

I have heard good things about craw4ai, I haven’t used it myself but you can try.

5

I will try.
Thanks very much!