# Try to read all files in a folder **URL:** \ **Category:** CrewAI Community Support\ **Tags:** agent, crewai\ **Created:** [November 22, 2024, 3:58am UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557 "2024-11-22T03:58:13Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![AndresVettori](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/andresvettori/32/1140_2.png) [@AndresVettori](https://community.crewai.com/u/AndresVettori)\ **Post date:** [November 22, 2024, 3:58am UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/1 "2024-11-22T03:58:13Z")
Hi, I’m trying to create a Crew that reads all source code files found in a folder (playing with some legacy source code migration scenarios) and what i’m seeing is that only a few (sometimes one, sometimes two or three but no more) files are read and processed. I have an agent with only one task that so far the only thing is trying to do is to read those source code files and generate some YAML metadata (purpose, references, callers, etc.) extracted from the source code. What i’m doing wrong? ---
**Author:** ![rokbenko](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/rokbenko/32/492_2.png) [@rokbenko](https://community.crewai.com/u/rokbenko)\ **Post date:** [November 22, 2024, 7:15am UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/2 "2024-11-22T07:15:28Z")
@AndresVettori Just to make sure, you’re using the [`DirectoryReadTool`](https://docs.crewai.com/tools/directoryreadtool), right? ---
**Author:** ![AndresVettori](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/andresvettori/32/1140_2.png) [@AndresVettori](https://community.crewai.com/u/AndresVettori)\ **Post date:** [November 22, 2024, 12:58pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/3 "2024-11-22T12:58:43Z")
Exactly, I have configured that tool for the task, and that works perfectly, but somehow the agent decides not to process all files but only a few and I don’t know how to control that. Those files are legacy source code I’m trying to analyze to extract metadata and want to write all resutls from all analyzed files into one output file. ---
**Author:** ![rokbenko](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/rokbenko/32/492_2.png) [@rokbenko](https://community.crewai.com/u/rokbenko)\ **Post date:** [November 23, 2024, 2:05pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/4 "2024-11-23T14:05:53Z")
> [@AndresVettori](#): > > the agent decides not to process all files but only a few and I don’t know how to control that Two suggestions come to mind: 1. Have you tried adjusting the agent and task configurations to explicitly specify that all files in the directory should be processed? I’ve used the `DirectoryReadTool` tool before, and it worked flawlessly for me, recognizing all the files in the directory without any issues, but I make sure that I always set clear configurations. 2. Have you tried using a more capable LLM? Sometimes, upgrading to a more advanced LLM can resolve such issues by providing better handling and understanding of the task. ---
**Author:** ![LBrandt123](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/lbrandt123/32/1165_2.png) [@LBrandt123](https://community.crewai.com/u/LBrandt123)\ **Post date:** [November 24, 2024, 1:19am UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/5 "2024-11-24T01:19:34Z")
What about counting the number of files in the directory first and then either ensuring that it processed that number of files --or-- use the count of files as an index and ask for file[1], file[2], and so on? ---
**Author:** ![AndresVettori](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/andresvettori/32/1140_2.png) [@AndresVettori](https://community.crewai.com/u/AndresVettori)\ **Post date:** [November 25, 2024, 12:57pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/6 "2024-11-25T12:57:04Z")
Thanks for your response, I will try to clarify a little more my scenario regarding the two comments you kindly posted: 1. I think the DirectoryReadTool is not the problem, as I see it reading (and fetching the task) the whole contents of the directory (reading all file names correctly). So, because how that list of files (that is quite extensive, there are hundreds of files to process) is processed I think is the agent responsibility I have tried to be very explicit in the prompt (agent goal and task description) clearly stating that all files must be processed and no optimization should be done there, but no change in results. 2. I am usign gpt4-o and have also tried with other advanced models (gemini, claude, etc.) with the same results so I don’t think this is something related to the models. What I’m going to try now is to create some custom code that receives the directory name to process and explicitely read all files and process them. It seems a little counter intuitive to do that because I don’t need CrewAI to do it that way, but maybe there is something I can try and later could be adapted into a more native CrewAI way of working. Any ideas are welcome, Thanks again. ---
**Author:** ![AndresVettori](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/andresvettori/32/1140_2.png) [@AndresVettori](https://community.crewai.com/u/AndresVettori)\ **Post date:** [November 25, 2024, 12:57pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/7 "2024-11-25T12:57:54Z")
Thanks a lot for the suggestion, any pointers or sample on how to implement that in CrewAI? Agv ---
**Author:** ![rokbenko](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/rokbenko/32/492_2.png) [@rokbenko](https://community.crewai.com/u/rokbenko)\ **Post date:** [November 25, 2024, 1:07pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/8 "2024-11-25T13:07:37Z")
> [@AndresVettori](#): > > I think the DirectoryReadTool is not the problem, as I see it reading (and fetching the task) the whole contents of the directory (reading all file names correctly). So, because how that list of files (that is quite extensive, there are hundreds of files to process) is processed I think is the agent responsibility I have tried to be very explicit in the prompt (agent goal and task description) clearly stating that all files must be processed and no optimization should be done there, but no change in results. Got it! You can [force tool output as a result](https://docs.crewai.com/how-to/force-tool-output-as-result) by setting the `result_as_answer` parameter to `True`. As stated in the docs: > This parameter ensures that the tool output is captured and returned as the task result, without any modifications by the agent. Try the following: ```auto from crewai.agent import Agent from crewai_tools import DirectoryReadTool my_agent = Agent( ..., tools=[DirectoryReadTool(result_as_answer=True)], ) ``` ---
**Author:** ![AndresVettori](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/andresvettori/32/1140_2.png) [@AndresVettori](https://community.crewai.com/u/AndresVettori)\ **Post date:** [November 25, 2024, 2:04pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/9 "2024-11-25T14:04:01Z")
Hi, just tried the “force tool output as result” but is not what I need. It makes the result of the tool be returned by the agent without any processing, and I need the agent to process that file list and do something (analyze those file contents). What I’m seeing is this (output from the Agent): **# Agent:** **AI LLMs Expert Software Developer** ## Thought: Thought: The result of the file listing has shown a vast number of files in the directory, spread across various components such as models, views, controllers, services, and more. Given the vastness, I need to start with one category and one specific file to analyze and proceed towards extracting the necessary information. I’ll begin by looking at the app/models directory and analyze a file from there to capture the structure and functionality. To begin, I’ll read the content of one of the model files that might have complex business logic. Let’s start with `app/models/account.rb` since accounts are fundamental to ERP systems, and there’s a corresponding `accounts` table in the SQL DDL that has various fields and constraints. ## Using tool: Read a file’s content ## Tool Input: “{"file\_path": "/Users/andres.vettori/Projects/BonsaiERP/app/models/account.rb"}” ## Tool Output: The output here is perfect, it’s correct. BUT… It’s only FOR ONE FILE and the agent doesn’t continue processing more files. And the agent has looked into the file list and it’s “vast” so he decides to choose a few (or just one as in this case). What I also see now is that after processing a few files I get an error from the model: Error parsing LLM output, agent will retry: I did it wrong. Invalid Format: I missed the ‘Action:’ after ‘Thought:’. I will do right next, and don’t use a tool I have already used. If you don’t need to use any more tools, you must give your best complete final answer, make sure it satisfy the expect criteria, use the EXACT format below: Thought: I now can give a great answer Final Answer: my best complete final answer to the task. And so the agent returns the results it has compiled so far (for three files). This might be a case of context exhaustion (of some kind) in the model? ---
**Author:** ![rokbenko](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/rokbenko/32/492_2.png) [@rokbenko](https://community.crewai.com/u/rokbenko)\ **Post date:** [November 25, 2024, 2:09pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/10 "2024-11-25T14:09:26Z")
> [@AndresVettori](#): > > Hi, just tried the “force tool output as result” but is not what I need. It makes the result of the tool be returned by the agent without any processing, and I need the agent to process that file list and do something (analyze those file contents). Why not create an agent or task just to list all files in a directory and set the `result_as_answer` parameter to `True`? Then, use another agent or task to process them. This won’t fully solve your goal of processing all the files, but it makes the workflow easier to handle and debug by breaking it into smaller steps. ---
**Author:** ![AndresVettori](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/andresvettori/32/1140_2.png) [@AndresVettori](https://community.crewai.com/u/AndresVettori)\ **Post date:** [November 25, 2024, 6:50pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/11 "2024-11-25T18:50:09Z")
Ok, I have implemented this idea and now I’m testing it. Let’s see how it goes… Agv ---
**Author:** ![rokbenko](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/rokbenko/32/492_2.png) [@rokbenko](https://community.crewai.com/u/rokbenko)\ **Post date:** [November 25, 2024, 6:53pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/12 "2024-11-25T18:53:40Z")
> [@AndresVettori](#): > > Let’s see how it goes… Let me know the results! ---
**Author:** ![cmosguy](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/cmosguy/32/950_2.png) [@cmosguy](https://community.crewai.com/u/cmosguy)\ **Post date:** [November 28, 2024, 5:52pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/13 "2024-11-28T17:52:01Z")
@AndresVettori I am curious how your adventure went here? Did this yield the results you were looking for? ---
**Author:** ![AndresVettori](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/andresvettori/32/1140_2.png) [@AndresVettori](https://community.crewai.com/u/AndresVettori)\ **Post date:** [November 29, 2024, 4:23pm UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/14 "2024-11-29T16:23:14Z")
Hi and thanks for asking. The use of “result\_as\_answer” worked well for things deterministic, like reading a folder’s content. But, for other kind of things where an agent is told (via prompt) to do something over a large list of files (maybe hundreds) then in all cases the agent try to “deduce” what is the best subset of files to accomplish the goals, and so never process all files and most of the time it only process one file, sometimes three, you get it. So, in my case I need to move forward and then stick to just code calling an LLM with more reduced scope (like analyzing one file at the time) and that’s it. So, i have (for now) decided not to use CrewAI for my endeavor. Thanks a lot, Agv ---
**Author:** ![plume69](https://sea1.discourse-cdn.com/flex025/user_avatar/community.crewai.com/plume69/32/1198_2.png) [@plume69](https://community.crewai.com/u/plume69)\ **Post date:** [December 2, 2024, 11:32am UTC](https://community.crewai.com/t/try-to-read-all-files-in-a-folder/1557/15 "2024-12-02T11:32:54Z")
I don’t know if this can help, but I used CrewAI to analyze the code of my application. I know there are probably more relevant and more complete tools, but I wanted to have a summarized and functional view. The key point is that we don’t control the context size, so a specific LLM configuration must be used. My project being quite simple, this single parameter allows me to have my documentation, but if you exceed the LLM’s context window, it will be a bit trickier 🙂 I hope this can help you, it’s quite simple but it works for me : Here is my code : from crewai import Agent, Crew, Process, Task from crewai.project import CrewBase, agent, crew, task from lead\_score\_flow.models import CandidateScore from crewai import LLM from crewai\_tools import DirectoryReadTool,FileReadTool from pathlib import Path llm = LLM( model=“gpt-4o-mini”, temperature=0.4, max\_tokens=4000, top\_p=0.9, frequency\_penalty=0.1, presence\_penalty=0.1, ) @CrewBase class CodeAnalysis: “”“Code Analyzer”“” ``` agents_config = "config/agents.yaml" tasks_config = "config/tasks.yaml" @agent def expert_python_agent(self) -> Agent: # Remonter au répertoire 'lead_score_flow' directory_path = Path( __file__ ).parents[3] print(f"Répertoire cible : {directory_path}") directory_tool = DirectoryReadTool(directory=str(directory_path)) file_read_tool = FileReadTool() return Agent( config=self.agents_config["expert_python_agent"], verbose=True, llm=llm, tools=[directory_tool,file_read_tool], allow_code_execution=False, # Disable code execution ) @task def code_analysis(self) -> Task: return Task( config=self.tasks_config["code_analysis"], output_pydantic=CandidateScore, ) @crew def crew(self) -> Crew: """Creates the Code Analysis Crew""" return Crew( agents=self.agents, tasks=self.tasks, process=Process.sequential, verbose=True, ) ``` code\_analysis: description: \> ONLY READ Python and YAML files, analyze the code to deliver a comprehensive explanation of the global application functionality. You MUST read all Python and YAML files ``` Use your expertise to thoroughly examine the code, considering key aspects such as: - Functionality and logic ADDITIONAL INSTRUCTIONS ----------------------- Your final answer MUST include: - A resume explanation of what the application does **Important:** Do not ask any questions to the user and provide your final answer following the instructions above. **Important:** Do not explain each file. **Important:** Do not read file in code_analysis_crew. **Important:** Do not write technical terms and test or configuration information. ``` expected\_output: \> formatted md file. A thorough analysis of the provided application, including explanations of its functionality. agent: expert\_python\_agent expert\_python\_agent: role: \> Expert Python Programmer goal: \> Analyze the Python code to provide a precise explanation of how it works. backstory: \> As a Python programming expert, you have extensive experience in software development and the architecture of generative AI solutions using agents. You excel at providing simple explanations of how an application works, with content that is understandable by CEOs, IT teams, or business teams who do not have a strong knowledge of development or the functioning of AI agents.