In case someone comes across the same problem, the solution was:
Put the openrouter key under the OPENAI_API_KEY, and append “openrouter/” to the model name. Like this:
OPENAI_API_BASE= Model Not Found | OpenRouter
OPENAI_MODEL_NAME= openrouter/mistralai/mistral-7b-instruct
OPENAI_API_KEY = (the openrouter api key)