sync local dir with s3 bucket , code for Jupyter

 refer to code:

.

local_directory = "/path/to/your/local/directory"

s3_bucket = "your-s3-bucket-name"

s3_folder = "your-s3-folder-name"

# Sync S3 bucket folder to local directory

!aws s3 sync s3://$s3_bucket/$s3_folder $local_directory

# Sync local directory to S3 bucket folder

!aws s3 sync $local_directory s3://$s3_bucket/$s3_folder

..

Replace /path/to/your/local/directory, your-s3-bucket-name, and your-s3-folder-name with your specific values. The first aws s3 sync command downloads the S3 folder's contents to the local directory, and the second one uploads the local directory's contents to the S3 folder. You can use either of these commands as needed.

Note that the aws s3 sync command does not support excluding or including specific files like rsync, but it will only copy new and updated files by default.

Thank you.

🙇🏻‍♂️

www.marearts.com

To save a JSON object (stored in a Python variable) to an Amazon S3 bucket

 

refer to code:

.

import boto3

import json

# Initialize the S3 client

s3 = boto3.client('s3')

# Specify the S3 bucket and JSON object key

bucket_name = 'your-bucket-name'

object_key = 'path/to/your/object.json'

# Your JSON data

json_data = {

"key1": "value1",

"key2": "value2",

"key3": "value3"

}

# Convert the JSON data to a string

json_content = json.dumps(json_data)

# Save the JSON content to the S3 bucket

s3.put_object(Bucket=bucket_name, Key=object_key, Body=json_content)

print(f"Saved JSON data to '{bucket_name}/{object_key}'")

..

This code will convert the JSON data to a string, and then save it to the specified S3 bucket and key.

Thank you.

🙇🏻‍♂️

www.marearts.com

Folder existing check in s3 bucket, python example code

Check folder exist in s3 bucket

refer to code:

.

import boto3

import botocore.exceptions

s3 = boto3.client('s3')

bucket_name = 'your-bucket-name'

folder_path = 'f1/f2/f3/f4/'

try:

response = s3.list_objects_v2(Bucket=bucket_name, Prefix=folder_path)

if 'Contents' in response or 'CommonPrefixes' in response:

print(f"Folder {folder_path} exists in bucket {bucket_name}")

else:

print(f"Folder {folder_path} does not exist in bucket {bucket_name}")

except botocore.exceptions.ClientError as e:

if e.response['Error']['Code'] == "NoSuchBucket":

print(f"Bucket {bucket_name} does not exist")

else:

raise

..

thank you

🙇🏻‍♂️

www.marearts.com

AWS S3, Get object list in Subfolder by python code using s3_client.list_objects function

This is my s3 folder structure

This is code to get file list in certain subfolder.

#get boto3 instance
s3_client = boto3.client(
        's3',
        aws_access_key_id=ACCESS_KEY,
        aws_secret_access_key=SECRET_KEY,
    )

#get object list
contents = s3_client.list_objects(Bucket='test-can-delete-anyone', Prefix='folder1/subfolder1')['Contents']
for object in contents:
     print(object['Key'])

result
folder1/subfolder1/
folder1/subfolder1/4_kitchen.jpg
folder1/subfolder1/5_bathroom.jpg
folder1/subfolder1/5_bedroom.jpg
folder1/subfolder1/5_frontal.jpg
folder1/subfolder1/5_kitchen.jpg
folder1/subfolder1/6_bathroom.jpg

another example

#get object list
contents = s3_client.list_objects(Bucket='test-can-delete-anyone', Prefix='folder1/')['Contents']
for object in contents:
        print(object['Key'])

result
folder1/
folder1/1_kitchen.jpg
folder1/2_bathroom.jpg
folder1/2_bedroom.jpg
folder1/2_frontal.jpg
folder1/2_kitchen.jpg
folder1/subfolder1/
folder1/subfolder1/4_kitchen.jpg
folder1/subfolder1/5_bathroom.jpg
folder1/subfolder1/5_bedroom.jpg
folder1/subfolder1/5_frontal.jpg
folder1/subfolder1/5_kitchen.jpg
folder1/subfolder1/6_bathroom.jpg

AWS s3 bucket - check folder exist or not by python code

Check certain folder exist in s3 bucket by python ncode


-

#create boto3 instance
s3_client = boto3.client(
        's3',
        aws_access_key_id=ACCESS_KEY,
        aws_secret_access_key=SECRET_KEY,
    )

#check folder exist
try:
        s3_client.get_object(Bucket='s3-bucket-name', Key='folder-name/')
        print('folder exist')
except botocore.exceptions.ClientError as e:
        print('no folder exist')

-

Thank you.


function type is here:

def check_folder_exist(s3_client, bucket_name, folder_name):
    
    try:
        s3_client.get_object(Bucket=bucket_name, Key=folder_name)
        return True
    except botocore.exceptions.ClientError as e:
        return False

If not working above code then try this one as well.

def check_folder_exist(bucket_name, folder_name):

try:

print(bucket_name, folder_name)

result = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=folder_name )

if 'Contents' in result:

return True

else:

return False

except botocore.exceptions.ClientError as e:

print(e)

return False

AWS S3 bucket, folder creation in python code

Basically s3 bucket doesn't have folder concept.
But this code create folder by key, and it doesn't have any object.


--

#create boto3 instance
s3_client = boto3.client(
        's3',
        aws_access_key_id=ACCESS_KEY,
        aws_secret_access_key=SECRET_KEY,
    )

#create folder by key    
s3_client.put_object(Bucket='s3-bucket-name', Key=('folder-name'+'/'))

--

Thank you.

64base image to pil image and upload s3 bucket on aws lambda

This article is about how to convert base64 string image to byte and invoke to PIL image.
Then we can handle image whatever we want to ex) image resize ..

base64 image come from img_base64 = event['base64Image']
and then convert to byte data imgdata = base64.b64decode(img_base64)

then it can save to image file and load to Image.open(filename)
or
invoke to pil image directly : img2 = Image.open(io.BytesIO(imgdata))

In the source code, there is also example how to upload image to s3 bucket.
refer to below code.
//

import sys
sys.path.append("/opt")

import json
import boto3
import os
import io
from PIL import Image
import base64



ACCESS_KEY = os.environ.get('ACCESS_KEY')
SECRET_KEY = os.environ.get('SECRET_KEY')

def uploadToS3(bucket, s3_path, local_path):
    s3_client = boto3.client(
        's3',
        aws_access_key_id=ACCESS_KEY,
        aws_secret_access_key=SECRET_KEY,
    )
    s3_client.upload_file(local_path, bucket, s3_path)
    
    
    
def lambda_handler(event, context):
    
    #base64 image string
    img_base64 = event['base64Image']
    #string to byte
    imgdata = base64.b64decode(img_base64)
    
    #save some image file
    with open("/tmp/imageToSave.png", "wb") as fh:
        fh.write(imgdata)
    
    #open("/tmp/imageToSave.png",'rb')
    uploadToS3("input-image1", "imageToSave.png", "/tmp/imageToSave.png")
    
    #load image file to pil
    img = Image.open("/tmp/imageToSave.png")
    width, height = img.size
    
    #load 
    img2 = Image.open(io.BytesIO(imgdata))
    width2, height2 = img2.size
    
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello leon it is from Lambda!'),
        'width file': width,
        'heihgt file' : height,
        'width pil': width2,
        'heihgt pil' : height2
    }

//

These method is referenced by:
https://stackoverflow.com/questions/16214190/how-to-convert-base64-string-to-image
https://stackoverflow.com/questions/11727598/pil-image-open-working-for-some-images-but-not-others
https://stackoverflow.com/questions/16214190/how-to-convert-base64-string-to-image
https://stackoverflow.com/questions/6444548/how-do-i-get-the-picture-size-with-pil

Thanks for effort.