3/16/2023

sync local dir with s3 bucket , code for Jupyter

 refer to code:


.

local_directory = "/path/to/your/local/directory"
s3_bucket = "your-s3-bucket-name"
s3_folder = "your-s3-folder-name"

# Sync S3 bucket folder to local directory
!aws s3 sync s3://$s3_bucket/$s3_folder $local_directory

# Sync local directory to S3 bucket folder
!aws s3 sync $local_directory s3://$s3_bucket/$s3_folder

..

Replace /path/to/your/local/directory, your-s3-bucket-name, and your-s3-folder-name with your specific values. The first aws s3 sync command downloads the S3 folder's contents to the local directory, and the second one uploads the local directory's contents to the S3 folder. You can use either of these commands as needed.

Note that the aws s3 sync command does not support excluding or including specific files like rsync, but it will only copy new and updated files by default.


Thank you.

🙇🏻‍♂️

www.marearts.com



3/15/2023

To save a JSON object (stored in a Python variable) to an Amazon S3 bucket

 

refer to code:

.

import boto3
import json

# Initialize the S3 client
s3 = boto3.client('s3')

# Specify the S3 bucket and JSON object key
bucket_name = 'your-bucket-name'
object_key = 'path/to/your/object.json'

# Your JSON data
json_data = {
"key1": "value1",
"key2": "value2",
"key3": "value3"
}

# Convert the JSON data to a string
json_content = json.dumps(json_data)

# Save the JSON content to the S3 bucket
s3.put_object(Bucket=bucket_name, Key=object_key, Body=json_content)

print(f"Saved JSON data to '{bucket_name}/{object_key}'")

..

This code will convert the JSON data to a string, and then save it to the specified S3 bucket and key.


Thank you.

🙇🏻‍♂️

www.marearts.com



copy s3 object to another bucket

refer to code: 


.

import boto3

# Initialize the S3 client
s3 = boto3.client('s3')

# Specify the source and destination S3 buckets and object keys
source_bucket = 'source-bucket-name'
source_key = 'path/to/source/object'

destination_bucket = 'destination-bucket-name'
destination_key = 'path/to/destination/object'

# Copy the object from the source bucket to the destination bucket
s3.copy_object(
CopySource={'Bucket': source_bucket, 'Key': source_key},
Bucket=destination_bucket,
Key=destination_key
)

print(f"Copied object from '{source_bucket}/{source_key}' to '{destination_bucket}/{destination_key}'")

..

Replace the placeholder values for source_bucket, source_key, destination_bucket, and destination_key with your actual bucket names and object keys. This code will copy the specified object from the source bucket to the destination bucket.


Thank you

🙇🏻‍♂️

www.marearts.com

load json file in memory form s3 bucket object (python example code)

 refer to code:

.

import boto3
import json

# Initialize the S3 client
s3 = boto3.client('s3')

# Specify the S3 bucket and JSON file key
bucket_name = 'your-bucket-name'
file_key = 'path/to/your/file.json'

# Download the JSON file from the S3 bucket
response = s3.get_object(Bucket=bucket_name, Key=file_key)
content = response['Body'].read()

# Parse the JSON content
data = json.loads(content)

# Print the JSON data
print(data)

..


Thank you.

🙇🏻‍♂️

www.marearts.com

get first item in dict (python sample code)

 refer to code:


.

my_dict = {'key1': 'value1', 'key2': 'value2', 'key3': 'value3'}

# Get the first key-value pair from the dictionary
first_key, first_value = next(iter(my_dict.items()))

print("First key:", first_key)
print("First value:", first_value)

..


Thank you.

www.marearts.com🙇🏻‍♂️

3/14/2023

split dict item as ratio

 sperate dict item as ratio.

refer to code:



.

import random

def split_dict(d, train_ratio=0.9):
# Convert the dictionary to a list of tuples and shuffle it
items = list(d.items())
random.shuffle(items)

# Calculate the indexes for the split
split_idx = int(train_ratio * len(items))

# Split the list into two lists containing train_ratio and (1 - train_ratio) of the items
train_items = items[:split_idx]
test_items = items[split_idx:]

# Convert the two lists back to dictionaries
train_dict = {k: v for k, v in train_items}
test_dict = {k: v for k, v in test_items}

return train_dict, test_dict




my_dict = {'apple': 2, 'banana': 3, 'orange': 1, 'kiwi': 4, 'pineapple': 5}
train_dict, test_dict = split_dict(my_dict, train_ratio=0.9)
print(train_dict)
print(test_dict)


..


Thank you.


python dict shuffle

shuffle dict order 


.

import random

my_dict = {'apple': 2, 'banana': 3, 'orange': 1, 'kiwi': 4}

# Convert the dictionary to a list of tuples and shuffle it
items = list(my_dict.items())
random.shuffle(items)

# Convert the shuffled list back to a dictionary
shuffled_dict = {k: v for k, v in items}

print(shuffled_dict)

..


Thank you.