What is a system prompt and what belongs in it?

I think of a system prompt as the highest-priority instruction block for a conversation. It tells the model who it is, what job it is handling, and which rules it must follow. I put the stable role, scope, behavioral rules, output style, and refusal behavior there. I also define the required output format, such as a schema or concise sections, so the task is explicit before the conversation starts.

One practical rule is to keep that block stable. A stable prefix can be prompt-cached, so the provider may reuse that computation across requests. I separate that shared policy from request-specific information. If a user supplies a document, question, or other request payload, I put it in the user turn rather than rebuilding the system prompt for every call.

I never put secrets in the system prompt. The prompt can be extracted, logged, cached, or shown to someone reviewing the application, so I assume any value there is visible and treat it as unsafe. The same applies to credentials and sensitive data. In an interview, I would explain that the system prompt sets policy, but it is not a security boundary. I would enforce authorization and data protection in the application around the model.