I want to deploy agents to retrieve for me data on a weekly basis from federal as well as state level (USA). The output should be well structured with no place for hallucinations or at least minimize them.
The data retrieval will probably be via API and web-scraping of site (preferably the first but only if available).
I have two questions:
Would you opt for a flow or crew? I read a bit and see flows allow better control over the process which may be necessary here. The output will be very structured (name, agency, budget, etc.). Moreover I will be able to do a direct llm call. Or, should I opt for crews (which are easier (from my end). Another thing is maybe to insert a crew within a flow?
The second thing, I wont deploy it locally but in AWS, I was told that Lambda is a good option (mainly as the running time will be short). Any experience with that?
Great question.. I have two preferred options for controlling the goals of crews.
My first try
I put a a crew planner in place and give it the goal to , look through the data or loop through the files etc
Then tasks the other agents / tasks to work on this.
If this fails (you can never tell, as this also depends on you $ and LLM choice..)
The second is to break the data into chunks in a flow Build Your First Flow - CrewAI and then feed to the next crew in the flow.
For hosting I have had a lot of success with the CrewAI platform, you need to use standard crews (unless you have budget) but it works well and the support team are really helpful.
Thanks @Tony_Wood,
It can’t be in the CrewAI platform as I am doing it for my work and I am required by the OSS method.
The reason a flow is an option is the cruciality of getting very structured output and don’t let the llm hallucinate (as much as possible). I actually was able to build a custom tool that reduces the probability of it happening (retrieving much more digested data so it has less room for mistakes) so a flow it is
Thanks again,
Etai
