LLM ProxyGoogle Gemini

You are viewing the docs for Archestra v1.4. View the latest docs

Google Gemini

Use Gemini through Google AI Studio with an API key, or through Vertex AI with no key. On GKE, Archestra signs in to Vertex AI with Workload Identity.

Adding a Provider

Go to Model Providers → Add API Key, select Google Gemini, and configure the credentials described below. Click Test & Create. A successful test adds the key and makes its models available under Models.

Archestra supports both the Google AI Studio (Gemini Developer API) and Vertex AI implementations of the Gemini API.

Gemini Connection Details

  • Base URL: https://<archestra-host>/v1/gemini/v1beta
  • Authentication:

Using Vertex AI

To use Vertex AI instead of Google AI Studio, configure these environment variables:

VariableRequiredDescription
ARCHESTRA_GEMINI_VERTEX_AI_ENABLEDYesSet to true to enable Vertex AI mode
ARCHESTRA_GEMINI_VERTEX_AI_PROJECTYesYour GCP project ID
ARCHESTRA_GEMINI_VERTEX_AI_LOCATIONNoGCP region (default: us-central1)
ARCHESTRA_GEMINI_VERTEX_AI_CREDENTIALS_FILENoPath to service account JSON key file

Vertex AI mode also gives Knowledge access to Vertex's multimodal embedding model (multimodalembedding@001) — see Image Embedding.

Priority PayGo

When Archestra uses Vertex AI, request Priority PayGo by sending these headers to its Gemini generateContent or streamGenerateContent endpoint:

http
X-Vertex-AI-LLM-Shared-Request-Type: priority
X-Vertex-AI-LLM-Request-Type: shared

Archestra forwards both values to Vertex. Omit X-Vertex-AI-LLM-Request-Type: shared to use available Provisioned Throughput before spilling over to Priority PayGo.

Check usageMetadata.trafficType in the response or streamed usage chunk: ON_DEMAND_PRIORITY confirms priority service; ON_DEMAND means standard service. Eligibility, pricing, and possible downgrades follow Vertex AI's Priority PayGo rules.

Missing Models and Nano Banana

Some Vertex AI models, including Nano Banana 2.1, require the global endpoint.

  1. Set ARCHESTRA_GEMINI_VERTEX_AI_ALLOW_GLOBAL_ENDPOINT=true and restart Archestra.
  2. Click Refresh Models, then search for Nano Banana.

Leave global access disabled if requests must stay in your configured region.

For GKE deployments, we recommend using Workload Identity which provides secure, keyless authentication. This eliminates the need for service account JSON key files.

Setup steps:

  1. Create a GCP service account with Vertex AI permissions:

    bash
    gcloud iam service-accounts create archestra-vertex-ai \
      --display-name="Archestra Vertex AI"
    
    gcloud projects add-iam-policy-binding PROJECT_ID \
      --member="serviceAccount:archestra-vertex-ai@PROJECT_ID.iam.gserviceaccount.com" \
      --role="roles/aiplatform.user"
    
  2. Bind the GCP service account to the Kubernetes service account:

    bash
    gcloud iam service-accounts add-iam-policy-binding \
      archestra-vertex-ai@PROJECT_ID.iam.gserviceaccount.com \
      --role="roles/iam.workloadIdentityUser" \
      --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
    

    Replace NAMESPACE with your Helm release namespace and KSA_NAME with the Kubernetes service account name (defaults to archestra-platform).

  3. Configure Helm values to annotate the service account:

yaml
archestra:
  orchestrator:
    kubernetes:
      serviceAccount:
        annotations:
          iam.gke.io/gcp-service-account: archestra-vertex-ai@PROJECT_ID.iam.gserviceaccount.com
  env:
    ARCHESTRA_GEMINI_VERTEX_AI_ENABLED: "true"
    ARCHESTRA_GEMINI_VERTEX_AI_PROJECT: "PROJECT_ID"
    ARCHESTRA_GEMINI_VERTEX_AI_LOCATION: "us-central1"

With this configuration, Application Default Credentials (ADC) will automatically use the bound GCP service account—no credentials file needed.

Other Environments

For non-GKE environments, Vertex AI supports several authentication methods through Application Default Credentials (ADC):

  • Service account key file: Set ARCHESTRA_GEMINI_VERTEX_AI_CREDENTIALS_FILE to the path of a service account JSON key file
  • Local development: Use gcloud auth application-default login to authenticate with your user account
  • Cloud environments: Attached service accounts on Compute Engine, Cloud Run, and Cloud Functions are automatically detected
  • AWS/Azure: Use workload identity federation to authenticate without service account keys

See the Vertex AI authentication guide for detailed setup instructions for each environment.

Prompt Caching

Cache a long system prompt or tool list once, and every request that references it bills its input at Gemini's cached rate.

  1. Create the cache through Archestra with your Google SDK's caches API, such as client.caches.create(...). Use the same base URL and key as for generation.
  2. Pass the returned cache name in cachedContent when calling generateContent or streamGenerateContent.

Archestra records provider-reported cached input tokens, including implicit cache hits. Cache lifetime, model eligibility, and storage charges follow Gemini’s caching rules.

The Model Router also accepts cachedContent alongside Chat Completions or Responses input when targeting Gemini. Reference a cache created for that Gemini model, and follow Google's restrictions on supplying system instructions or tools alongside cached content.

Anthropic-style cache_control markers and OpenAI-specific cache options cannot be translated to Gemini; the router rejects them with a 400 error instead of silently ignoring them.