sync local dir with s3 bucket , code for Jupyter

 refer to code:

.

local_directory = "/path/to/your/local/directory"

s3_bucket = "your-s3-bucket-name"

s3_folder = "your-s3-folder-name"

# Sync S3 bucket folder to local directory

!aws s3 sync s3://$s3_bucket/$s3_folder $local_directory

# Sync local directory to S3 bucket folder

!aws s3 sync $local_directory s3://$s3_bucket/$s3_folder

..

Replace /path/to/your/local/directory, your-s3-bucket-name, and your-s3-folder-name with your specific values. The first aws s3 sync command downloads the S3 folder's contents to the local directory, and the second one uploads the local directory's contents to the S3 folder. You can use either of these commands as needed.

Note that the aws s3 sync command does not support excluding or including specific files like rsync, but it will only copy new and updated files by default.

Thank you.

🙇🏻‍♂️

www.marearts.com

To save a JSON object (stored in a Python variable) to an Amazon S3 bucket

 

refer to code:

.

import boto3

import json

# Initialize the S3 client

s3 = boto3.client('s3')

# Specify the S3 bucket and JSON object key

bucket_name = 'your-bucket-name'

object_key = 'path/to/your/object.json'

# Your JSON data

json_data = {

"key1": "value1",

"key2": "value2",

"key3": "value3"

}

# Convert the JSON data to a string

json_content = json.dumps(json_data)

# Save the JSON content to the S3 bucket

s3.put_object(Bucket=bucket_name, Key=object_key, Body=json_content)

print(f"Saved JSON data to '{bucket_name}/{object_key}'")

..

This code will convert the JSON data to a string, and then save it to the specified S3 bucket and key.

Thank you.

🙇🏻‍♂️

www.marearts.com

copy s3 object to another bucket

refer to code: 

.

import boto3

# Initialize the S3 client

s3 = boto3.client('s3')

# Specify the source and destination S3 buckets and object keys

source_bucket = 'source-bucket-name'

source_key = 'path/to/source/object'

destination_bucket = 'destination-bucket-name'

destination_key = 'path/to/destination/object'

# Copy the object from the source bucket to the destination bucket

s3.copy_object(

CopySource={'Bucket': source_bucket, 'Key': source_key},

Bucket=destination_bucket,

Key=destination_key

)

print(f"Copied object from '{source_bucket}/{source_key}' to '{destination_bucket}/{destination_key}'")

..

Replace the placeholder values for source_bucket, source_key, destination_bucket, and destination_key with your actual bucket names and object keys. This code will copy the specified object from the source bucket to the destination bucket.

Thank you

🙇🏻‍♂️

www.marearts.com

load json file in memory form s3 bucket object (python example code)

 refer to code:

.

import boto3

import json

# Initialize the S3 client

s3 = boto3.client('s3')

# Specify the S3 bucket and JSON file key

bucket_name = 'your-bucket-name'

file_key = 'path/to/your/file.json'

# Download the JSON file from the S3 bucket

response = s3.get_object(Bucket=bucket_name, Key=file_key)

content = response['Body'].read()

# Parse the JSON content

data = json.loads(content)

# Print the JSON data

print(data)

..

Thank you.

🙇🏻‍♂️

www.marearts.com

get first item in dict (python sample code)

 refer to code:

.

my_dict = {'key1': 'value1', 'key2': 'value2', 'key3': 'value3'}

# Get the first key-value pair from the dictionary

first_key, first_value = next(iter(my_dict.items()))

print("First key:", first_key)

print("First value:", first_value)

..

Thank you.

www.marearts.com🙇🏻‍♂️

split dict item as ratio

 sperate dict item as ratio.

refer to code:

.

import random

def split_dict(d, train_ratio=0.9):

# Convert the dictionary to a list of tuples and shuffle it

items = list(d.items())

random.shuffle(items)

# Calculate the indexes for the split

split_idx = int(train_ratio * len(items))

# Split the list into two lists containing train_ratio and (1 - train_ratio) of the items

train_items = items[:split_idx]

test_items = items[split_idx:]

# Convert the two lists back to dictionaries

train_dict = {k: v for k, v in train_items}

test_dict = {k: v for k, v in test_items}

return train_dict, test_dict

my_dict = {'apple': 2, 'banana': 3, 'orange': 1, 'kiwi': 4, 'pineapple': 5}

train_dict, test_dict = split_dict(my_dict, train_ratio=0.9)

print(train_dict)

print(test_dict)

..

Thank you.

python dict shuffle

shuffle dict order 

.

import random

my_dict = {'apple': 2, 'banana': 3, 'orange': 1, 'kiwi': 4}

# Convert the dictionary to a list of tuples and shuffle it

items = list(my_dict.items())

random.shuffle(items)

# Convert the shuffled list back to a dictionary

shuffled_dict = {k: v for k, v in items}

print(shuffled_dict)

..

Thank you.