To try and keep my post shorter, I thought that I’d start this discussion with a video (apologies for it being unscripted )
The YouTube video is here (~8mins) I suggest you set quality in settings to 720p
Have you tried experimenting with things like lm studio which allow you to truncate the middle of context or use a rolling window? This is very coarse control but that along with setting context window size might be interesting.
On the crewai side specifically, you could reset different forms of the memory function dynamically, which would at least selectively forget things.
@moto,
What I’m talking about is real time context managment during run time, not static (LLM studio).
Re: CrewAI, I don’t think we have any official control of context from the code. Yes we can selectively forget, but NOT selectivley remember (bookmark)!
I’m not sure what you mean re lm studio not being real time. As far as I know they take each new prompt to the llm and if it is smaller than context size, let it go. If the prompt is larger than context size, they either roll the context window forward, discarding the earliest piece of the prompt or truncate the middke of the prompt, leaving a beginning portion and the ending portion.
On crewai memory, there is not a memory control except on or off but you could use the reset function of each memory type dynamically by finding that in the code. Not great but something.
Yes. What if you create a simple tool that copies short term memory up to a point then issue a memory reset. Then another tool that reads that memory to get it back in to context later.
That has potential, if we can tag it & recall by that tag. I need to know more about how crewai handles context/memory and a clear definition of what ‘short term memory’ is.
Short-Term Memory Temporarily stores recent interactions and outcomes using RAG, enabling agents to recall and utilize information relevant to their current context during the current executions.
If only we could ‘optionally’ tag/name instances of short-term memory, use these tags in our prompts later, or can we?
The facility of being able to indicate to a Task, or Agent that they should save their output with options to leave it in the context, or make it static recallable via tag later. This would also have the benefit of reducing context size, etc.
As an aside, an interesting article
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem.
)
