# InferenceGateway Source: /reference/inferencegateways/ An InferenceGateway is the front door for inference requests: the only address a caller sees. It speaks the OpenAI and Anthropic APIs, authenticates callers, resolves the model a request names to a ModelService, and forwards to whichever of that service's endpoints should serve it, translating the request for the backend that won. A Modelplane can run several, each on an InferenceCluster of its own. Run one per region to keep a caller's traffic in its jurisdiction, or two in a region to survive losing a cluster. Modelplane runs no global load balancer: distributing callers across gateways is yours to configure, whether by geo DNS, an anycast address, or an edge of your own with these gateways as origins. Apply instances as `apiVersion: modelplane.ai/v1alpha1`, `kind: InferenceGateway`. [Concept guide: Set Up the Gateway](/platform/inference-gateway/index.md) ## Example ```yaml apiVersion: modelplane.ai/v1alpha1 kind: InferenceGateway metadata: name: eu spec: # The InferenceCluster this gateway runs on, which decides its region and its # address. The cluster needs no GPU pools. clusterName: gw-gcp-eu # The name the gateway answers on. Point it at status.address. hostname: eu.example.com tls: certificateRefs: - name: eu-example-com-tls auth: # Each key in a selected Secret is one caller: the entry's name is the # caller's identity, its value is the key. Adding a caller means writing a # Secret, not editing this gateway. secretSelector: matchLabels: modelplane.ai/inference-keys: "true" # The ModelServices this gateway serves. Absent, it serves every one. Scoped # here to a region, which is how residency is expressed. serviceSelector: matchLabels: example.org/region: eu ``` ## Definition The CompositeResourceDefinition this reference is generated from, with the complete OpenAPI schema, validation rules, and defaults: ```yaml apiVersion: apiextensions.crossplane.io/v2 kind: CompositeResourceDefinition metadata: name: inferencegateways.modelplane.ai spec: group: modelplane.ai names: categories: [crossplane, modelplane] kind: InferenceGateway plural: inferencegateways shortNames: [ig] scope: Cluster versions: - name: v1alpha1 served: true referenceable: true additionalPrinterColumns: - name: CLUSTER type: string jsonPath: .spec.clusterName - name: HOSTNAME type: string jsonPath: .spec.hostname - name: ADDRESS type: string jsonPath: .status.address schema: openAPIV3Schema: description: >- An InferenceGateway is the front door for inference requests: the only address a caller sees. It speaks the OpenAI and Anthropic APIs, authenticates callers, resolves the model a request names to a ModelService, and forwards to whichever of that service's endpoints should serve it, translating the request for the backend that won. A Modelplane can run several, each on an InferenceCluster of its own. Run one per region to keep a caller's traffic in its jurisdiction, or two in a region to survive losing a cluster. Modelplane runs no global load balancer: distributing callers across gateways is yours to configure, whether by geo DNS, an anycast address, or an edge of your own with these gateways as origins. type: object required: [spec] properties: spec: type: object required: [clusterName] x-kubernetes-validations: # Serving HTTPS needs somewhere to get a certificate, and a # certificate is only meaningful for a name. - rule: "!has(self.tls) || has(self.hostname)" message: spec.hostname is required when spec.tls is set. properties: clusterName: type: string description: >- The InferenceCluster this gateway runs on, which decides its region and its address. A gateway doesn't move: unlike a ModelDeployment, whose replicas re-place when their cluster goes away, a gateway stays where it was put. Availability comes from running more of them, because failing over would change the address callers use and could move traffic out of the jurisdiction the gateway exists to hold. The cluster needs no GPU pools. A cluster with none is a gateway and nothing else, which is what a region with callers but no accelerators wants. A cluster that serves models can host a gateway too, and does so at most once. minLength: 1 maxLength: 253 hostname: type: string description: >- The name this gateway answers on. Point it at status.address once the gateway has one. Omit it and the gateway answers on its address alone, over plain HTTP. That is the getting-started shape, and also the shape for anyone terminating TLS on an edge of their own in front of the gateway. minLength: 1 maxLength: 253 x-kubernetes-validations: - rule: "!self.contains('/') && !self.contains(':')" message: spec.hostname must be a bare DNS name, with no scheme, port or path. tls: type: object description: >- Serves callers over HTTPS. Without it the caller's hop is unencrypted, so anything reachable from an untrusted network wants this or an edge that terminates TLS in front. required: [certificateRefs] properties: certificateRefs: type: array description: >- Secrets holding the gateway's certificate, of type kubernetes.io/tls, in the same namespace as this Modelplane's other gateway Secrets. Modelplane copies them to the gateway's cluster. minItems: 1 maxItems: 8 x-kubernetes-list-type: map x-kubernetes-list-map-keys: [name] items: type: object required: [name] properties: name: type: string minLength: 1 maxLength: 253 auth: type: object description: >- Authenticates callers against keys this gateway holds. Omit it and the gateway authenticates nobody, so anything that can reach the address can invoke any ModelService it serves. That is only appropriate behind something that has already established who is calling. Modelplane authenticates callers; it does not authorize them. Every accepted key can reach every ModelService this gateway serves, and /v1/models lists them all regardless of key. To narrow what a key can reach, narrow the gateway with serviceSelector or run a separate gateway with its own keys. required: [secretSelector] properties: secretSelector: type: object description: >- Selects Secrets holding caller API keys. Each key in a selected Secret is one caller: the entry's name is the caller's identity and its value is the key. So adding a caller means writing a Secret, not editing this gateway. The gateway stamps the resolved identity onto every request and every usage record, and never forwards the caller's key. Ranking one caller above another is not Modelplane's decision to make, so it publishes the identity and leaves acting on it to whatever does decide. required: [matchLabels] properties: matchLabels: type: object additionalProperties: type: string maxLength: 63 minProperties: 1 maxProperties: 16 serviceSelector: type: object description: >- Selects the ModelServices this gateway serves, by their labels. Absent, it serves every one. This is how a gateway is scoped: to a region, so an EU service is only reachable through EU gateways; to your public services on an internet-facing front door; or to a named set on a dedicated gateway. These are your labels, under your own prefix. Modelplane matches them and never interprets them, so a region means no more to it than any other label. required: [matchLabels] properties: matchLabels: type: object additionalProperties: type: string maxLength: 63 minProperties: 1 maxProperties: 16 status: type: object properties: address: type: string description: >- The address this gateway answers on, and what spec.hostname should point at. It is also the target to health check, at /healthz, to decide whether this gateway is in rotation. /healthz answers 200 whenever this gateway's proxy is running and serving. It says nothing about whether any ModelService is reachable through it, so a gateway with no healthy backend stays in rotation and answers requests with a 503. Read each ModelService's RoutingReady for that. clientCACertificate: type: string maxLength: 16384 description: >- PEM certificate of the CA that signs this gateway's client certificate. Every InferenceCluster accepts client certificates from it, which is how this gateway proves itself to a cluster gateway and how anything else is refused. One CA per gateway rather than one per Modelplane, so that no private key has to be distributed: each is generated on the cluster that uses it and only its certificate travels. Note that every cluster gateway trusts every fleet gateway's CA and checks the signing CA rather than the subject, so this bounds where the keys live, not what one of them can reach. endpoints: type: object description: The paths this gateway serves. properties: openAI: type: string description: >- Base URL for the OpenAI API. A caller sets its SDK's base_url to this and names a ModelService as the model. anthropic: type: string description: Base URL for Anthropic's Messages API. ```