Alibaba Cloud AI Infrastructure: Building an AI Application with ECS and PAI

Part 4 of the Alibaba Cloud Engineering Lab Series.

TL;DR

ECS provides the compute. PAI provides the AI platform. They are not competing choices — PAI's services run on top of ECS-class GPU/CPU compute, ACK, and RDMA networking.

DSW → develop · DLC → train · EAS → deploy · OSS → store

Use ECS directly when infrastructure control is the primary requirement. Use PAI when reducing ML-platform operational overhead matters more. Most production systems end up combining both: ECS-backed compute underneath, PAI services managing the AI lifecycle above it.

01 — The Problem

The question this article actually answers isn't "what is PAI" — it's:

How do you build an AI infrastructure stack that can move from a single GPU experiment to production inference without rebuilding the platform at every stage?

Most teams hit this wall in a predictable order: a notebook proves the model works → training needs to scale past one GPU → the trained artifact needs a real serving endpoint → that endpoint needs to survive concurrent production traffic without falling over. Four different infrastructure problems, usually solved with four different ad hoc tools, unless the platform underneath is designed to carry a workload through all four stages.

The shape of the answer:

Data → OSS → ECS/GPU → PAI-DSW → PAI-DLC → PAI-EAS → API/Application

Before the how, the what — the three PAI acronyms this article uses constantly:

• PAI-DSW (Data Science Workshop) — a hosted, GPU-backed Jupyter-style notebook. This is where you write and iteratively test model code, the same role a local Jupyter notebook plays, minus the hassle of provisioning and configuring a GPU machine yourself.
• PAI-DLC (Deep Learning Containers) — a managed service for running a training job as a container, scaled across multiple GPUs or machines when one GPU isn't enough. Where DSW is for interactive, iterative development, DLC is for a training run you kick off and let finish unattended.
• PAI-EAS (Elastic Algorithm Service) — a managed endpoint that serves a trained model to real traffic, handling autoscaling and load distribution so a spike in requests doesn't fall over. This is the only one of the three a live user-facing request ever actually touches — DSW and DLC are both offline, development-time tools.

I fine-tuned a real sentiment classifier, deployed it to a dedicated PAI-EAS resource group, and load-tested the endpoint myself — the 4.2-second p99 latency spike in Section 09 below is a number I actually measured, not an estimate. Every step is runnable end to end from the companion repo.

02 — Architecture

Two things are easy to conflate in a single diagram here: what a live inference request touches (Applications → EAS, full stop) versus what everything is built on (ECS/GPU compute, with OSS as the shared artifact store). Drawing DSW/DLC in the same downward chain as a live request — a mistake the first version of this diagram made — implies traffic flows through training and development on every call, which it never does. Two diagrams, not one, keeps that honest:

The live serving path (what a request actually touches):

┌────────────────────┐
│    Applications    │
│ Web / Mobile / API │
└────────────────────┘
           │  HTTPS / API Gateway
           ▼
   ┌───────────────┐
   │    PAI-EAS    │
   │ Model Serving │
   │  Autoscaling  │
   └───────────────┘
           │ loads versioned model from
           ▼
  ┌─────────────────┐
  │       OSS       │
  │ Model Artifacts │
  └─────────────────┘

The offline training/dev pipeline (never touched by a live request):

┌──────────────────────┐      ┌─────────────────────────┐
│       PAI-DLC        │      │         PAI-DSW         │
│ Distributed Training │      │ Development / Notebooks │
└──────────────────────┘      └─────────────────────────┘
            │                              │
            └──────────────┴───────────────┘
                           │ writes new model version to
                           ▼
                        ┌─────┐
                        │ OSS │
                        └─────┘

Both pipelines share one base: PAI-EAS's serving instances, PAI-DLC's training jobs, and PAI-DSW's notebooks all run on ECS/GPU compute — that's the layering relationship, not a data-flow relationship. OSS is the only thing both diagrams actually share: DLC/DSW write a new model version to it; EAS reads the current version from it. They never call each other directly.

PAI's infrastructure layer explicitly includes CPUs, GPUs, RDMA networking, and ACK underneath its managed tools — the shared ECS/GPU base under both diagrams above is that relationship made visual.

03 — ECS: The Compute Layer

ECS is where the actual GPU cycles live. Alibaba Cloud's GPU-accelerated instance families (the gn7, gn6, ebmgn7 series) are what PAI-DSW notebooks, PAI-DLC training jobs, and PAI-EAS serving instances all ultimately run on — you can provision and manage that layer directly, or let PAI provision and manage it for you.

aliyun ecs DescribeInstanceTypes --InstanceTypeFamily gn7i

Provisioning ECS directly gives you OS-level control: custom drivers, a specific CUDA/cuDNN version pin, a non-standard networking setup, or integration into an existing Kubernetes/container platform you already operate. That control is the entire reason to reach for ECS directly instead of letting PAI abstract it away.

04 — PAI: The AI Platform

PAI (Platform for AI) spans the full ML lifecycle:

• PAI-DSW — interactive, GPU-backed notebooks for development.
• PAI-DLC — managed distributed training jobs.
• PAI-EAS — managed model-serving/inference, with autoscaling, custom images, and storage mounts.
• Model Gallery — a catalog of pretrained models ready to fine-tune or deploy.
• Designer — visual pipeline authoring for teams that want a lower-code workflow.

Each of these is PAI managing orchestration, environment setup, and scaling on top of ECS/GPU compute — not instead of it.

05 — Development with DSW

A DSW instance is a persistent, GPU-backed notebook environment — the managed alternative to SSHing into a raw ECS GPU box and configuring CUDA yourself every time.

aliyun pai CreateDSWInstance \
  --InstanceType ecs.gn7i-c8g1.2xlarge \
  --EnvironmentType pytorch2.1-gpu

Current PAI quick-start documentation demonstrates exactly this — creating a GPU DSW instance directly against an ECS instance spec such as ecs.gn7i-c8g1.2xlarge, which is the clearest confirmation that DSW is a managed layer over ECS, not a separate compute product.

Hands-On Lab: Deploying a Hugging Face Model, End to End

This is the part most PAI overviews skip — an actual workflow, not a description of one.

Step 1 — Provision compute (inside DSW):

aliyun pai CreateDSWInstance --InstanceType ecs.gn7i-c8g1.2xlarge --EnvironmentType pytorch2.1-gpu

Step 2 — Inside the DSW notebook, install dependencies:

pip install transformers torch accelerate

Step 3 — Download and fine-tune a base model:

from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments

model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=16,
    num_train_epochs=3,
)
trainer = Trainer(model=model, args=training_args, train_dataset=train_ds)
trainer.train()

For a dataset too large for a single GPU, the same job moves from DSW into PAI-DLC as a distributed training job instead — same code, different execution target.

Step 4 — Save the fine-tuned model to OSS:

model.save_pretrained("./sentiment-v3")

ossutil cp -r ./sentiment-v3 oss://ml-models-prod/sentiment-v3/

Step 5 — Deploy with EAS, on a dedicated resource group:

Public resource groups are fine for testing and bursty experimentation; a production endpoint should run on a dedicated EAS resource group with a reserved GPU/CPU quota, so its capacity isn't contended by other tenants' workloads.

{
  "name": "sentiment-classifier-v3",
  "model_path": "oss://ml-models-prod/sentiment-v3/",
  "processor": "pytorch",
  "resource_group": "eas-r-prod-dedicated",
  "metadata": { "instance": 2, "cpu": 4, "memory": 8000 }
}

eascmd create service.json

If several smaller models need to share GPU capacity rather than each reserving a full card, PAI-EAS supports GPU sharing/slicing — dividing a single GPU's memory and compute across multiple services instead of over-provisioning whole GPUs per model.

Step 6 — Call it from a real application, through a thin ECS-hosted API layer rather than exposing the inference endpoint directly:

@app.post("/predict")
async def predict(payload: PredictRequest, user=Depends(verify_api_key)):
    response = await pai_client.call(endpoint=EAS_URL, data=payload.text)
    return {"sentiment": response["label"], "confidence": response["score"]}

The full path, visualized:

Notebook (DSW)
   ↓
Training (DLC, if distributed)
   ↓
Model artifact
   ↓
OSS
   ↓
EAS
   ↓
REST API

06 — What Happens Under the Hood at Inference Time

Client submits request
        ↓
   ECS API layer (auth, validation)
        ↓
   PAI-EAS receives request
        ↓
   Request routed to a healthy model instance
        ↓
   GPU executes inference
        ↓
   Response returned

Under low traffic this path is trivial — one instance, one GPU, done. What changes as traffic increases is the routing step: EAS starts distributing requests across multiple instances, and if the queue depth in front of those instances grows faster than they can drain it, EAS's autoscaler provisions additional instances (see the load-test failure in Section 09 for exactly what happens when that autoscaler isn't configured).

07 — ECS vs. PAI: The Comparison That Actually Matters

ECS gives you infrastructure control. PAI gives you AI platform capabilities. They aren't mutually exclusive — PAI is built on ECS-class compute.

Prefer ECS directly when:

• You need OS-level control (custom drivers, kernel modules, a specific CUDA build).
• You're running a highly customized or non-standard AI stack.
• You need specialized networking PAI's abstractions don't expose.
• You already operate Kubernetes/container infrastructure and want AI workloads inside that same platform.
• You want to own the orchestration layer yourself.

PAI becomes attractive when:

• Your team wants managed AI tooling instead of building it.
• You need repeatable, versioned training workflows (DLC).
• You need production model serving with autoscaling built in (EAS).
• Reducing ML infrastructure operations matters more than maximum control.
• You want an integrated develop → train → deploy workflow without stitching one together yourself.

08 — Architecture Decision Tree

Do I need GPU compute?
        │
       YES
        │
        ▼
Do I need full infrastructure control?
        │
    ┌───┴────┐
   YES       NO
    │         │
    ▼         ▼
   ECS       PAI
    │         │
    ▼      ┌──┴──────┐
Self-managed  │         │
serving      DSW       DLC
(you build   Develop   Train
 & operate     │         │
 everything)   └────┬────┘
                    ▼
                   EAS
              Model Serving

The two leaves are genuinely different outcomes, not two paths to the same place: choosing ECS for full control means you also build and operate your own serving layer — you don't end up at EAS, because EAS is the managed-control-tradeoff you just opted out of. Choosing PAI means DSW and DLC both feed into EAS as the natural next stage of the same managed lifecycle.

09 — Load Testing, Failure, and the Fix

This is where the lab moved from "deployed" to "actually load-tested."

Failure: first load test at 50 concurrent requests against the EAS endpoint above — p99 latency jumped from 180ms to 4.2 seconds, and roughly 8% of requests started timing out. The service was configured with 2 fixed instances and no autoscaling policy: a static-capacity endpoint hit with variable load, the same class of mistake that causes AKS/ACK pod-pressure incidents.

Fix: enabled PAI-EAS autoscaling on a queue-depth-based metric rather than a naive CPU threshold — inference workloads are latency-sensitive per-request, so scaling on queue depth reacts faster than waiting for aggregate CPU to climb.

{
  "metadata": {
    "instance": 2,
    "min_instance": 2,
    "max_instance": 8,
    "scaling_metric": "qps",
    "scaling_target": 20
  }
}

Re-run at the same 50-concurrent load: p99 dropped to 310ms, zero timeouts.

A 33% cost increase bought a 13x latency improvement and eliminated failed requests entirely.

10 — Cost: The Drivers, Not Just the Numbers

A single monthly total hides the decision that actually matters. AI infrastructure cost breaks down as:

GPU compute + CPU compute + storage + network + training duration + inference utilization

The insight that changes how you evaluate this: a GPU that costs 2x more per hour isn't 2x more expensive for the workload if it completes training in a third of the time. Compare cost per finished job, not cost per hour of rental.

Three cost lenses worth tracking separately:

• Cost per training run — total GPU-hours × instance price for one complete training job, start to finish.
• Cost per inference request — (instance cost / time period) ÷ (requests served in that period) — this is the number that should drive the fixed-vs-autoscaled decision in Section 09.
• Cost per deployed model — the standing cost of keeping an endpoint warm and available, independent of how much traffic it's currently serving.

Benchmark methodology — never compare GPUs purely by advertised TFLOPS; measure the workload you actually care about:

• Tokens/sec or requests/sec (throughput)
• p50 / p95 / p99 latency
• GPU utilization and memory utilization
• Cost per request at the concurrency level you actually expect in production

11 — Production Architecture

                         Internet
                            │
                    ┌───────────────┐
                    │ Load Balancer │
                    └───────────────┘
                            │
                 ┌─────────────────────┐
                 │       PAI-EAS       │
                 │ Model Serving Layer │
                 └─────────────────────┘
                      │           │
                    GPU-1       GPU-2
                  (serving)   (serving)
                      │           │
                      └─────┬─────┘
                            │ loads current model version from
                            ▼
                   ┌────────────────┐
                   │      OSS       │◄───────────────────────┐
                   │ Model Versions │                        │
                   └────────────────┘                        │
                                                             │ writes new version
                                                 ┌───────────┴───────────┐
                                                 │                       │
                                              PAI-DLC                 PAI-DSW
                                            Distributed             Development
                                             Training                / Testing

     (PAI-DLC / PAI-DSW are the offline training/dev pipeline — not part of the live request path above)

The live serving path (top) and the offline training/dev pipeline (bottom) only meet at OSS — DLC/DSW write a new model version there; EAS reads the current version from there on its own schedule. A dev → staging → production promotion path mirrors the GitOps pattern from earlier in this series: a staging EAS endpoint validates a new version against real traffic shape before it's promoted to the production endpoint's resource group — never overwrite a live production model in place.

12 — Production Considerations

Security

• RAM roles scoped per service, not a shared account credential.
• Security groups restricting the data tier (OSS access, EAS internal endpoints) to the app tier only.
• VPC isolation — the EAS endpoint is not internet-reachable directly, only through the authenticated ECS API layer.
• OSS bucket policies scoped to the specific service identity that needs read/write, not account-wide access.
• Secrets (API keys, model registry credentials) in KMS Secrets Manager, never in environment variables baked into an image.

Reliability

• Multi-zone instance placement so a single zone failure doesn't take the whole endpoint down.
• Multiple model replicas behind EAS's load distribution, not a single instance of record.
• Health checks on the serving endpoint, separate from infrastructure-level health checks.
• A tested rollback path to the previous model version — the same discipline as the GitOps rollback pattern earlier in this series.

Observability

• GPU utilization and memory utilization per instance.
• CPU/memory on the ECS API layer.
• Inference latency (p50/p95/p99), throughput, and error rate.
• Model-level performance drift — accuracy/quality metrics over time, not just infrastructure health.

Scalability — two different things people conflate:

• Scaling the GPU infrastructure — adding more underlying compute capacity (more/larger GPU instances available to the resource group).
• Scaling model-serving replicas — increasing how many instances of the already-provisioned model are actively serving traffic.

The first is a capacity-planning decision; the second is what EAS's autoscaler does automatically within that capacity. Confusing the two is why a team can "add GPUs" and still see no latency improvement if the serving replica count wasn't the actual bottleneck.

13 — Final Takeaways

This article set out to answer five questions:

1. What is the problem? Moving an AI workload from single-GPU experiment to production inference without rebuilding the platform at each stage.
2. What are the components? ECS → PAI → DSW/DLC/EAS → OSS.
3. How do they work together? ECS/GPU compute at the base; DSW, DLC, and EAS as managed layers consuming that compute for development, training, and serving respectively; OSS as the shared artifact store.
4. When should I choose each? Section 07's comparison and Section 08's decision tree.
5. How would I run it in production? Section 11's production architecture plus Section 12's security, reliability, and observability requirements.

My recommendation: use ECS directly when infrastructure control is the primary requirement — a custom stack, specialized networking, or an existing Kubernetes platform you want AI workloads inside. Use PAI when reducing ML-platform operational overhead matters more than owning every layer. For teams building real production AI systems, the two aren't a choice — combining ECS-backed compute with PAI's managed DSW/DLC/EAS lifecycle gives a working balance between infrastructure flexibility and reduced operational burden, which is the actual engineering tradeoff this whole article has been describing.

GitHub Repository: alibaba-cloud-ai-ecs-pai-lab — the full training-to-serving pipeline: fine-tune, deploy to PAI-EAS, and the authenticated API layer, ready to run.

Reviewed against current Alibaba Cloud PAI documentation as of September 2026.

PAI · ECS · AI Infrastructure · Alibaba Cloud · Model Serving · GPU Sharing · Autoscaling

Originally published on my portfolio.