Hi everyone,
I am working on an automated multi-agent workflow where one of my researcher agents is supposed to pull billing breakdowns and reference data directly from online portals. However, I am hitting a wall when directing the agent to read and extract information from this website. The web scraper tool frequently hangs during execution, or the agent ends up returning empty responses instead of actually capturing the rendered account details.
I think part of the problem stems from how the standard scraping tools handle dynamic form submissions and page layouts under the hood. On top of the extraction failures, I am also running into a couple of related headaches. First, the vector embeddings get clogged with useless navigation links and footer text rather than focusing solely on the core data container. Second, whenever the target host takes slightly longer to respond, the entire agent step hits a hard timeout and fails the overall task instead of retrying gracefully.
Has anyone here dealt with similar scraping or parsing hurdles when trying to integrate custom web tools into a crew? I am trying to figure out whether I should ditch the default web tools entirely in favor of an external browser automation library, or if there is a clean way to configure timeout parameters and element filters directly within the agent’s task definition.