Agent Tasks do not accurately process large JSON arrays (> 20 objects) in KnowledgeSource

General
1

Issue:
When using JSONKnowledgeSource, CrewAI agents and tasks are not accurately processing entire JSON arrays if the array contains a large number of objects.
In my tests, small arrays (4-5 objects) work fine — each object is processed as expected. But when the array size increases (20+ objects), the agent starts missing or incompletely processing objects.

What I tried so far:
:white_check_mark: Improved the Task config to explicitly mention to process each JSON object → no effect.
:white_check_mark: Split the large JSON array into multiple smaller files (e.g. 5 objects per file), added all files to KnowledgeSource → same issue persists. The agent still does not process all objects accurately.

Expectation:
Agents should reliably process large JSON arrays, either by chunking them properly or ensuring the retrieval covers the full array.

Request:
Please advise if this is a known limitation, and if so:

  • Is there a recommended way to configure tasks or knowledge sources to handle large JSON arrays correctly?
  • Should this be reported as a bug or feature improvement?
2

Welcome to the community Kushang

Great question. Knowledge source I find is best used as a look up. For large datasets I prefer to break them down into chunks and use flow to process each one. Build Your First Flow - CrewAI