AWS SAM Troubleshooting - Fixing pip/runtime and AWS CLI Issues

🔧 AWS SAM Troubleshooting - Fixing pip/runtime and AWS CLI Issues

If you're deploying AWS Lambda functions with SAM (Serverless Application Model), you may have encountered frustrating build errors. This guide explains the two most common issues and how to fix them permanently.

Note: This guide uses generic examples (<YOUR_STACK_NAME>) and is safe to share publicly.

🚨 The Two Common Problems

Problem A — sam build fails with pip/runtime error

You may see this error:

Error: PythonPipBuilder:ResolveDependencies - Failed to find a Python runtime containing pip on the PATH.

What this means: SAM is trying to build for a specific Lambda runtime (like python3.11), but your shell has:

• python from one location (e.g., conda environment)
• pip from another location (e.g., ~/.local/bin or /usr/bin)

SAM requires a matching pair - the pip must belong to the same Python interpreter that matches your Lambda runtime version.

Problem B — AWS CLI crashes with botocore conflicts

You may see errors like:

KeyError: 'opsworkscm'
ModuleNotFoundError: No module named 'dateutil'

What this means: Your system AWS CLI (/usr/bin/aws) is accidentally importing incompatible botocore or boto3 packages from ~/.local/lib/python..., causing version conflicts.

🔍 Quick Diagnosis (5 Commands)

Run these from your SAM project directory to diagnose the issue:

# 1. What Python runtime does your template.yaml require?
grep "Runtime: python" template.yaml

# 2. What python are you using?
which python
python -V

# 3. What pip are you using?
which pip
pip -V

# 4. Does pip belong to this python?
python -m pip -V

🚩 Red flag: If pip -V and python -m pip -V show different paths or Python versions, your PATH is contaminated.

✅ The Fix: Dedicated Environment + Clean PATH

The solution is to create an isolated environment that matches your Lambda runtime and force clean PATH ordering.

Step 1: Create an environment matching your Lambda runtime

If your template.yaml specifies Runtime: python3.11, create a Python 3.11 environment:

# Using conda (recommended)
conda create -n aws-sam-py311 python=3.11 pip -y
conda activate aws-sam-py311

# Or using venv
python3.11 -m venv ~/.virtualenvs/aws-sam-py311
source ~/.virtualenvs/aws-sam-py311/bin/activate

Step 2: Install SAM CLI and AWS CLI inside the environment

# Upgrade pip first
python -m pip install --upgrade pip

# Install SAM CLI
python -m pip install aws-sam-cli

# Optional: Install AWS CLI v2 (avoids system aws/botocore conflicts)
# Using conda-forge:
conda install -c conda-forge awscli -y

# Or using pip:
python -m pip install awscli

Step 3: Disable user-site imports and fix PATH ordering

This is the critical step that prevents ~/.local contamination:

# Disable user site-packages (~/.local)
export PYTHONNOUSERSITE=1

# Force clean PATH (conda/venv bin first, then system)
export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin"

# Or for venv:
# export PATH="$VIRTUAL_ENV/bin:/usr/bin:/bin"

# Clear shell hash table
hash -r

Step 4: Verify the fix

# All should point to your environment
which python
which pip
which sam
which aws

# Verify versions match
python -V      # Should be 3.11.x
pip -V         # Should show python 3.11
sam --version  # Should work without errors
aws --version  # Should work without errors

Step 5: Build and deploy

sam build --cached --parallel
sam deploy --no-confirm-changeset --stack-name <YOUR_STACK_NAME> --region <YOUR_AWS_REGION>

🐳 Alternative: Container Build (Docker)

If you have Docker installed, you can avoid all Python toolchain issues by building in a container:

sam build --use-container
sam deploy --no-confirm-changeset --stack-name <YOUR_STACK_NAME> --region <YOUR_AWS_REGION>

Pros:

• ✅ No need to match local Python version
• ✅ Builds in environment identical to Lambda
• ✅ Most reproducible approach

Cons:

• ❌ Slower than native builds
• ❌ Requires Docker installed and running

⚡ Quick Fix for Broken AWS CLI (Emergency)

If you need to use system AWS CLI right now and it's broken:

# Force it to ignore user-site packages
PYTHONNOUSERSITE=1 /usr/bin/aws --version
PYTHONNOUSERSITE=1 /usr/bin/aws sts get-caller-identity
PYTHONNOUSERSITE=1 /usr/bin/aws s3 ls

But the proper fix is: Install AWS CLI inside your dedicated environment (see Step 2 above).

🤔 Why Lambda is python3.11 but my machine uses python3.12?

This is a common source of confusion. They are different things:

Component	What It Is	Where It's Defined
Lambda Runtime	Python version AWS runs in production	template.yaml → Runtime: python3.11
Your Local Python	Python version for development/training/scripts	Your system default or conda environment

Key point: When SAM builds your Lambda functions, it must build dependencies compatible with the Lambda runtime, even if your system default is Python 3.12.

Example from template.yaml:

AnprDeviceLicenseValidateFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: anpr_device_license_validate/
    Handler: app.lambda_handler
    Runtime: python3.11          # ← Lambda uses 3.11
    Architectures:
      - x86_64

So you have three options:

1. Match local environment to Lambda (recommended) - Create python3.11 env for SAM work
2. Use container build - Let Docker handle it with sam build --use-container
3. Upgrade Lambda runtime - Change template.yaml to python3.12 (requires testing)

📋 Complete Example: Deploy Script

Here's a complete bash script that implements all the fixes:

#!/usr/bin/env bash
set -euo pipefail

# Activate conda environment (matches Lambda runtime)
source ~/anaconda3/etc/profile.d/conda.sh
conda activate aws-sam-py311

# Critical: Clean PATH and disable user-site
export PYTHONNOUSERSITE=1
export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin"
hash -r

echo "Environment ready:"
echo "  Python: $(python -V)"
echo "  SAM: $(sam --version | head -1)"
echo "  AWS: $(aws --version)"

# Build and deploy
sam build --cached --parallel
sam deploy --no-confirm-changeset

🎯 Troubleshooting Checklist

Issue	Check	Fix
sam build fails	which pip vs python -m pip -V	Create dedicated env, fix PATH
aws command crashes	echo $PYTHONNOUSERSITE	Set PYTHONNOUSERSITE=1
Wrong Python version	python -V vs Lambda runtime	Create env matching Lambda
Multiple pip versions	which -a pip	Fix PATH ordering
Conda conflicts	conda env list	Create separate env for SAM

🔒 Security Best Practices

⚠️ When sharing code publicly:

• ❌ Never publish template.yaml with secrets (API keys, tokens, webhook URLs)
• ✅ Use AWS Secrets Manager or SSM Parameter Store for secrets
• ✅ Redact from logs:
  • AWS account IDs
  • API Gateway URLs
  • Stack names and ARNs
  • Any access keys/tokens

💡 Pro Tips

1. Create a deployment script

Instead of remembering all these environment variables, create a deploy.sh script:

#!/usr/bin/env bash
set -euo pipefail

# Activate environment
source ~/anaconda3/etc/profile.d/conda.sh
conda activate aws-sam-py311

# Clean environment
export PYTHONNOUSERSITE=1
export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin"
hash -r

# Build and deploy
sam build --cached --parallel
sam deploy --no-confirm-changeset

echo "✅ Deployment complete!"

Make it executable: chmod +x deploy.sh

2. Use SAM build cache for faster builds

# First build (slow)
sam build

# Subsequent builds (much faster!)
sam build --cached --parallel

3. Test locally before deploying

# Invoke function locally
sam local invoke MyFunction --event events/test.json

# Start local API
sam local start-api

4. Skip changeset confirmation in CI/CD

# Manual deployment - shows changes
sam deploy

# CI/CD deployment - no prompts
sam deploy --no-confirm-changeset

📊 Before vs After

❌ Before (Broken)
$ sam build Error: Failed to find Python runtime containing pip $ aws --version KeyError: 'opsworkscm' $ which pip /home/user/.local/bin/pip # Wrong location! $ pip -V pip 24.0 (python 3.12) # Wrong version!

✅ After (Fixed)
$ conda activate aws-sam-py311 $ export PYTHONNOUSERSITE=1 $ export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin" $ sam build Build Succeeded ✨ $ aws --version aws-cli/2.32.26 Python/3.11.14 $ which pip /home/user/anaconda3/envs/aws-sam-py311/bin/pip # Correct! $ pip -V pip 25.3 (python 3.11) # Matches Lambda runtime!

🎓 Summary

The root cause of most SAM build failures is PATH contamination - your shell mixes Python versions and pip locations from different sources (~/.local, /usr/bin, conda environments).

The complete fix:

1. ✅ Create dedicated environment matching Lambda runtime (python3.11)
2. ✅ Install SAM CLI and AWS CLI inside that environment
3. ✅ Set PYTHONNOUSERSITE=1 to disable user-site packages
4. ✅ Fix PATH ordering: export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin"
5. ✅ Run hash -r to clear shell cache

After this, sam build and aws commands will work reliably! 🚀

🔗 Additional Resources

• 📚 AWS SAM Official Documentation
• 🐍 Conda Environment Management
• 🐳 SAM Container Builds
• ⚙️ AWS CLI v2 Installation

---

Tags: AWS, SAM, Lambda, Python, DevOps, Deployment, Troubleshooting, ServerlessFramework, CICD, CloudComputing