Announcing OpenAI-compatible API support for Amazon SageMaker AI endpoints

2026-05-20 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, long

Summary

Amazon SageMaker AI now offers OpenAI-compatible API support for real-time inference endpoints, enabling users to invoke models on SageMaker by simply updating their endpoint URL. This integration allows existing applications built with the OpenAI SDK, LangChain, or Strands Agents to seamlessly connect without custom clients or code rewrites. SageMaker endpoints expose an "/openai/v1" path for Chat Completions requests, including streaming, and utilize time-limited bearer tokens for authentication. This feature supports agentic workflows on owned infrastructure, multi-model hosting via inference components, and serving fine-tuned models, such as Qwen3-4B on an ml.g6.2xlarge instance, all through a unified OpenAI-compatible interface.

Key takeaway

For AI Engineers deploying large language models, this SageMaker AI update simplifies integrating your existing OpenAI SDK or LangChain applications. You can now run agentic workflows or serve fine-tuned models on dedicated AWS infrastructure with minimal code changes, enhancing control over data residency and GPU resources. Ensure your IAM roles for token generation are narrowly scoped to specific endpoint ARNs to maintain robust security.

Key insights

SageMaker AI now offers OpenAI API compatibility, simplifying integration for existing LLM applications.

Principles

• Standardize LLM access via OpenAI API.
• Isolate model inference on dedicated GPUs.
• Scope IAM permissions for token security.

Method

Deploy models on SageMaker AI real-time endpoints, generate a time-limited bearer token using the SageMaker Python SDK, then configure your OpenAI-compatible client with the SageMaker endpoint URL and token.

In practice

• Run LangChain agents on SageMaker.
• Host multiple models on one endpoint.
• Serve fine-tuned models with existing apps.

Topics

• Amazon SageMaker AI
• OpenAI API Compatibility
• LLM Inference
• Bearer Token Authentication
• Multi-Model Hosting
• AI Agents

Code references

• aws-samples/sagemaker-genai-hosting-examples

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Building custom model provider for Strands Agents with LLMs hosted on SageMaker AI endpoints — Custom model parsers bridge LLM response format mismatches between SageMaker endpoints and Strands agents.
• OpenAI frontier models and Codex are now available on AWS — OpenAI models are now on AWS, simplifying enterprise AI integration.
• Amazon SageMaker AI in 2025, a year in review part 1: Flexible Training Plans and improvements to price performance for inference workloads — SageMaker AI's 2025 updates enhance generative AI inference with predictable capacity, improved resilience, and optimized performance.
• OpenAI and Amazon announce strategic partnership — OpenAI and Amazon partner to integrate advanced AI models and infrastructure, expanding enterprise AI capabilities and compute capacity.
• OpenAI models now available on Amazon Web Services — OpenAI's GPT-5.5, GPT-5.4, and Codex are now available on AWS Bedrock, integrating with existing AWS infrastructure and security.
• An Interview with OpenAI CEO Sam Altman and AWS CEO Matt Garman About Bedrock Managed Agents — AI agent development requires deep integration of models with cloud-native runtime environments for enterprise adoption.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.