Hi, I’m using a custom knowledge source built on a JSON for a crew I’m making, and I’m running into issues where a given knowledge source retrieval will occasionally result in repeated chunks being returned. I’ve verified that my chunking works as intended with a log (every JSON entry gets its own chunk and there are no duplicates within the logs). I’m also logging the knowledge retrieval events, which is where I found the duplicate chunks. Is this intended behavior? If so, is there a way to force unique chunks to be retrieved?
The listener i’m using to log:
@crewai_event_bus.on(knowledge_events.KnowledgeRetrievalCompletedEvent)
def on_knowledge_retrieved(source, event):
logger.info("Knowledge retrieval complete")
logger.debug(f"Knowledge query: {event.query}")
logger.debug(f"Knowledge retrieved:{str(event.retrieved_knowledge)}")
It’s possible that the repeated chunks are a byproduct of the logging code but the agents also only report info from the repeated chunks. They don’t themselves repeat the chunks but it means I get less results than I want from the JSON.