12/06/2019

using specific gpu device for TensorFlow

setting first or second gpu machine
CUDA_VISIBLE_DEVICES=0 python script_one.py
CUDA_VISIBLE_DEVICES=1 python script_two.py
another way
use first (it didn't work for me)
export CUDA_VISIBLE_DEVICE=0
./train.py
use second (it didn't work for me)
export CUDA_VISIBLE_DEVICE=1
./train.py
use both (it didn't work for me)
export CUDA_VISIBLE_DEVICE=0,1
./train.py

refer to here:
https://stackoverflow.com/questions/44135538/tensorflow-using-2-gpu-at-the-same-time

12/05/2019

monitoring gpu status in command (terminal)



pip install gpustat
>gpustat -cp




monitoring continuously
>watch -n 0.5 -c gpustat -cp --color


12/03/2019

find pdf file (or some exe file) in directories and copy it to another directory, python sample code


import os
import glob
from shutil import copyfile


files = []
start_dir = '/Volumes/input/'
output_path = '/Volumes/output/'
pattern = "*.pdf"

total = 0
for dir,_,_ in os.walk(start_dir):
files.extend(glob.glob(os.path.join(dir,pattern)))
for i,v in enumerate(files):
#found pdf files
print(total,i,v)
#extract filename only
filename = v.split('/')[-1]
#make new filename and output path
output_filename = output_path + str(total) + '_' + filename
#if file exist? then no copy
exist = glob.glob(output_filename)
#if not copy
if len(exist) == 0:
copyfile(v, output_filename)
#print out copied filename
print('copy! : ', output_filename)
#increase global count
total += 1

11/29/2019

download zip file from url, python sample code




from bs4 import BeautifulSoup
import requests
import os
import sys

def downloadZip(url, prefix_url, outpath):
mbyte=1024*1024
html = requests.get(url).text
soup = BeautifulSoup(html, features='lxml')


for name in soup.findAll('a', href=True):
#find A tag
zipurl = name['href']
#find file extension
if( zipurl.endswith('.zip') ):
#make download path
outfname = outpath +'/'+ zipurl.split('/')[-1]
#make url
zipurl = prefix_url+zipurl #http://aaa.com/ + 'abc.zip'
print(zipurl)
r = requests.get(zipurl, stream=True)
if( r.status_code == requests.codes.ok ) :
fsize = int(r.headers['content-length'])
print('Downloading %s (%sMb)'%(outfname, fsize/mbyte))
with open(outfname, 'wb') as fd:
for chunk in r.iter_content(chunk_size=1024): # chuck size can be larger
if chunk: # ignore keep-alive requests
fd.write(chunk)
fd.close()

base_path = os.getcwd()
path_join = os.path.join(base_path, 'data_download_pdf')
sys.path.append(path_join)

# point to output directory
outpath = path_join
url = 'https://www.gsa.gov/real-estate/real-estate-services/leasing-policy-procedures/lease-documents/lease-documents-region-1-new-england/'
prefix_url = 'https://www.gsa.gov/cdnstatic'

downloadZip(url, prefix_url, outpath)








11/24/2019

python website url verification code


import requests
import urllib.request

url = 'http://www.marearts.com'

try:
resp = requests.get(url, verify=False)
print(url, resp.status_code)
except:
print('fail to access')

try:
resp = urllib.request.urlopen(url)
print(url, resp.getcode())
except:
print('fail to access')

11/20/2019

All configured authentication methods failed -> vscode sftp setting

Try to use this script on sftp.json file.

{
"name": "test",
"protocol": "sftp",
"host": "ec2-114-24-120-84.eu-west-4.compute.amazonaws.com",
"remotePath": "/home/ubuntu/ABCD",
"privateKeyPath": "/Users/ABD/ec2_aws.pem",
"username": "ubuntu",
"port": 22,
"secure": true,
"uploadOnSave": true,
"passive": false,
"debug": true,
"ignore": [
"\\.vscode",
"\\.git",
"\\.DS_Store"
],
"generatedFiles": {
"uploadOnSave": true,
"extensionsToInclude": [],
"path": "./"
}
}

11/16/2019

yum install -y https://centos7.iuscommunity.org/ius-release.rpm -> yum install -y https://centos7.iuscommunity.org/ius-release.rpm

yum install -y https://centos7.iuscommunity.org/ius-release.rpm
but when you get this error :

Error:
 Problem: conflicting requests
  - nothing provides epel-release = 7 needed by ius-release-2-1.el7.ius.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)


Try to install this one first.

yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
then
yum install -y https://centos7.iuscommunity.org/ius-release.rpm


Thank you.