AWS, find all file list in s3 bucket

 Find all file list in s3 bucket

import boto3

BUCKET_INPUT = 'bucket_name'

PREFIX_INPUT = 'some_prefix_folder_name'

item_list_all = []

paginator = s3_client.get_paginator('list_objects')

operation_parameters = {'Bucket': BUCKET_INPUT} #,'Prefix': PREFIX_INPUT}

page_iterator = paginator.paginate(**operation_parameters)

for page in page_iterator:

item_list_all.append(page['Contents'])

key_all = []

for page in item_list_all:

for i in range(len(page)):

key_all.append(page[i]["Key"])

print("The total number of files in the bucket:", len(key_all) )

Thank you.

www.marearts.com

ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

I met this error when I install simpletransformers or transformers.

I did many try but no luck.

But this was solution to me.

Install Rust before transformer installation.

so..

curl https://sh.rustup.rs -sSf | bash -s -- -y

PATH="/root/.cargo/bin:${PATH}"

and install transformers

Thank you.

www.marearts.com

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /var/lib/snapd/void/Dockerfile: no such file or directory

 

if you install docker using snap, it can be happen.

so remove docker from snap installation

> sudo snap remove docker

And install docker again using apt

> sudo apt update
> sudo apt upgrade
> sudo apt install docker.io
> sudo systemctl enable --now docker

Thank you.

www.marearts.com

labelme2yolo, yolo2labelme format converter

 

original page is here:

https://marearts.notion.site/Labelme2Yolo-Yolo2Labelme-3564dd886ac64b5499d6f2784a8a4be8

yolo2labelme main code

if __name__ == "__main__":

#parameters

paser = argparse.ArgumentParser()

args = paser.parse_args("")

##### set params #####

#input : yolo image, label path

args.yolo_images_path = './yolo_data/images/'

args.yolo_labels = './yolo_data/labels/'

#class count

args.yolo_nc = 80

#class names

args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']

#output : labelme output path

args.labelme_path = './output_yolo2labelme/'

args.labelme_save_image = True #False #create jpg images in output path

#######################

#######################

#start processing

yolo2labelme_main(args)

#######################

labelme2yolo main code

#parameters

paser = argparse.ArgumentParser()

args = paser.parse_args("")

##### set params #####

#input : labelme output path

args.labelme_path = './output_yolo2labelme/'

args.labelme_save_image = True #False #create jpg images in output path

#output : yolo image, label path

args.yolo_images_path = './output_yolo_data/images/'

args.yolo_labels = './output_yolo_data/labels/'

#class count

args.yolo_nc = 80

#class names

args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']

#######################

#######################

#start processing

labelme2yolo_main(args)

#######################

Thank you.

www.marearts.com

🙇🏻‍♂️

make python package zip for lambda layer using docker

 Here is good reference for this subject:

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-layer-simulated-docker/

https://dev.to/matthewvielkind/creating-python-aws-lambda-layers-with-docker-4376

If we use docker then it's very simple.

Let's go through step by step

1. make Main folder & python library folder

ex) In this tutorial, we suppose to need flask package

mkdir flask-layer

cd flask-layer

mkdir -pv python/lib/python3.6/site-packages

or

mkdir -pv python/lib/python3.8/site-packages

2. make requirements.txt and put packages to install by pip

flask==1.1.1

So folder structure looks like this:

├── requirements.txt └── python/ └── lib/ ├── python3.6/ │ └── site-packages/ └── python3.8/ └── site-packages/

3. Run docker to install packages

note, change python version properly

docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"

 

4. compress package to zip

zip -r yourpack.zip python > /dev/null

5. use zip file for your lambda

That's it all.

Thank you.

www.marearts.com

🙇🏻‍♂️