🔧 AWS SAM Troubleshooting - Fixing pip/runtime and AWS CLI Issues
If you're deploying AWS Lambda functions with SAM (Serverless Application Model), you may have encountered frustrating build errors. This guide explains the two most common issues and how to fix them permanently.
Note: This guide uses generic examples (<YOUR_STACK_NAME>) and is safe to share publicly.
🚨 The Two Common Problems
Problem A — sam build fails with pip/runtime error
You may see this error:
Error: PythonPipBuilder:ResolveDependencies - Failed to find a Python runtime containing pip on the PATH.
What this means: SAM is trying to build for a specific Lambda runtime (like python3.11), but your shell has:
pythonfrom one location (e.g., conda environment)pipfrom another location (e.g.,~/.local/binor/usr/bin)
SAM requires a matching pair - the pip must belong to the same Python interpreter that matches your Lambda runtime version.
Problem B — AWS CLI crashes with botocore conflicts
You may see errors like:
KeyError: 'opsworkscm'
ModuleNotFoundError: No module named 'dateutil'
What this means: Your system AWS CLI (/usr/bin/aws) is accidentally importing incompatible botocore or boto3 packages from ~/.local/lib/python..., causing version conflicts.
🔍 Quick Diagnosis (5 Commands)
Run these from your SAM project directory to diagnose the issue:
# 1. What Python runtime does your template.yaml require?
grep "Runtime: python" template.yaml
# 2. What python are you using?
which python
python -V
# 3. What pip are you using?
which pip
pip -V
# 4. Does pip belong to this python?
python -m pip -V
🚩 Red flag: If pip -V and python -m pip -V show different paths or Python versions, your PATH is contaminated.
✅ The Fix: Dedicated Environment + Clean PATH
The solution is to create an isolated environment that matches your Lambda runtime and force clean PATH ordering.
Step 1: Create an environment matching your Lambda runtime
If your template.yaml specifies Runtime: python3.11, create a Python 3.11 environment:
# Using conda (recommended)
conda create -n aws-sam-py311 python=3.11 pip -y
conda activate aws-sam-py311
# Or using venv
python3.11 -m venv ~/.virtualenvs/aws-sam-py311
source ~/.virtualenvs/aws-sam-py311/bin/activate
Step 2: Install SAM CLI and AWS CLI inside the environment
# Upgrade pip first
python -m pip install --upgrade pip
# Install SAM CLI
python -m pip install aws-sam-cli
# Optional: Install AWS CLI v2 (avoids system aws/botocore conflicts)
# Using conda-forge:
conda install -c conda-forge awscli -y
# Or using pip:
python -m pip install awscli
Step 3: Disable user-site imports and fix PATH ordering
This is the critical step that prevents ~/.local contamination:
# Disable user site-packages (~/.local)
export PYTHONNOUSERSITE=1
# Force clean PATH (conda/venv bin first, then system)
export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin"
# Or for venv:
# export PATH="$VIRTUAL_ENV/bin:/usr/bin:/bin"
# Clear shell hash table
hash -r
Step 4: Verify the fix
# All should point to your environment
which python
which pip
which sam
which aws
# Verify versions match
python -V # Should be 3.11.x
pip -V # Should show python 3.11
sam --version # Should work without errors
aws --version # Should work without errors
Step 5: Build and deploy
sam build --cached --parallel
sam deploy --no-confirm-changeset --stack-name <YOUR_STACK_NAME> --region <YOUR_AWS_REGION>
🐳 Alternative: Container Build (Docker)
If you have Docker installed, you can avoid all Python toolchain issues by building in a container:
sam build --use-container
sam deploy --no-confirm-changeset --stack-name <YOUR_STACK_NAME> --region <YOUR_AWS_REGION>
Pros:
- ✅ No need to match local Python version
- ✅ Builds in environment identical to Lambda
- ✅ Most reproducible approach
Cons:
- ❌ Slower than native builds
- ❌ Requires Docker installed and running
⚡ Quick Fix for Broken AWS CLI (Emergency)
If you need to use system AWS CLI right now and it's broken:
# Force it to ignore user-site packages
PYTHONNOUSERSITE=1 /usr/bin/aws --version
PYTHONNOUSERSITE=1 /usr/bin/aws sts get-caller-identity
PYTHONNOUSERSITE=1 /usr/bin/aws s3 ls
But the proper fix is: Install AWS CLI inside your dedicated environment (see Step 2 above).
🤔 Why Lambda is python3.11 but my machine uses python3.12?
This is a common source of confusion. They are different things:
| Component | What It Is | Where It's Defined |
|---|---|---|
| Lambda Runtime | Python version AWS runs in production | template.yaml → Runtime: python3.11 |
| Your Local Python | Python version for development/training/scripts | Your system default or conda environment |
Key point: When SAM builds your Lambda functions, it must build dependencies compatible with the Lambda runtime, even if your system default is Python 3.12.
Example from template.yaml:
AnprDeviceLicenseValidateFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: anpr_device_license_validate/
Handler: app.lambda_handler
Runtime: python3.11 # ← Lambda uses 3.11
Architectures:
- x86_64
So you have three options:
- Match local environment to Lambda (recommended) - Create python3.11 env for SAM work
- Use container build - Let Docker handle it with
sam build --use-container - Upgrade Lambda runtime - Change
template.yamltopython3.12(requires testing)
📋 Complete Example: Deploy Script
Here's a complete bash script that implements all the fixes:
#!/usr/bin/env bash
set -euo pipefail
# Activate conda environment (matches Lambda runtime)
source ~/anaconda3/etc/profile.d/conda.sh
conda activate aws-sam-py311
# Critical: Clean PATH and disable user-site
export PYTHONNOUSERSITE=1
export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin"
hash -r
echo "Environment ready:"
echo " Python: $(python -V)"
echo " SAM: $(sam --version | head -1)"
echo " AWS: $(aws --version)"
# Build and deploy
sam build --cached --parallel
sam deploy --no-confirm-changeset
🎯 Troubleshooting Checklist
| Issue | Check | Fix |
|---|---|---|
| sam build fails | which pip vs python -m pip -V |
Create dedicated env, fix PATH |
| aws command crashes | echo $PYTHONNOUSERSITE |
Set PYTHONNOUSERSITE=1 |
| Wrong Python version | python -V vs Lambda runtime |
Create env matching Lambda |
| Multiple pip versions | which -a pip |
Fix PATH ordering |
| Conda conflicts | conda env list |
Create separate env for SAM |
🔒 Security Best Practices
⚠️ When sharing code publicly:
- ❌ Never publish template.yaml with secrets (API keys, tokens, webhook URLs)
- ✅ Use AWS Secrets Manager or SSM Parameter Store for secrets
- ✅ Redact from logs:
- AWS account IDs
- API Gateway URLs
- Stack names and ARNs
- Any access keys/tokens
💡 Pro Tips
1. Create a deployment script
Instead of remembering all these environment variables, create a deploy.sh script:
#!/usr/bin/env bash
set -euo pipefail
# Activate environment
source ~/anaconda3/etc/profile.d/conda.sh
conda activate aws-sam-py311
# Clean environment
export PYTHONNOUSERSITE=1
export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin"
hash -r
# Build and deploy
sam build --cached --parallel
sam deploy --no-confirm-changeset
echo "✅ Deployment complete!"
Make it executable: chmod +x deploy.sh
2. Use SAM build cache for faster builds
# First build (slow)
sam build
# Subsequent builds (much faster!)
sam build --cached --parallel
3. Test locally before deploying
# Invoke function locally
sam local invoke MyFunction --event events/test.json
# Start local API
sam local start-api
4. Skip changeset confirmation in CI/CD
# Manual deployment - shows changes
sam deploy
# CI/CD deployment - no prompts
sam deploy --no-confirm-changeset
📊 Before vs After
| ❌ Before (Broken) |
|---|
|
| ✅ After (Fixed) |
|---|
|
🎓 Summary
The root cause of most SAM build failures is PATH contamination - your shell mixes Python versions and pip locations from different sources (~/.local, /usr/bin, conda environments).
The complete fix:
- ✅ Create dedicated environment matching Lambda runtime (
python3.11) - ✅ Install SAM CLI and AWS CLI inside that environment
- ✅ Set
PYTHONNOUSERSITE=1to disable user-site packages - ✅ Fix PATH ordering:
export PATH="$CONDA_PREFIX/bin:/usr/bin:/bin" - ✅ Run
hash -rto clear shell cache
After this, sam build and aws commands will work reliably! 🚀
🔗 Additional Resources
- 📚 AWS SAM Official Documentation
- 🐍 Conda Environment Management
- 🐳 SAM Container Builds
- ⚙️ AWS CLI v2 Installation
Tags: AWS, SAM, Lambda, Python, DevOps, Deployment, Troubleshooting, ServerlessFramework, CICD, CloudComputing
