7/07/2020

extract year, month, day from file on Ubuntu, python example


...
import os, time
date_created_obj = time.localtime(os.path.getctime(full_path))
print('Year: {:4d}'.format(date_created_obj.tm_year)) # Year: 2020
print('Month: {:2d}'.format(date_created_obj.tm_mon)) # Month: 2
print('Day: {:2d}'.format(date_created_obj.tm_mday)) # Day: 10

...


7/06/2020

how to merge two csr_matrix, example python source code

let's see the code.

..
from scipy.sparse import csr_matrix
import numpy as np

#first matrix
row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 2, 2, 0, 1, 2])
data = np.array([1, 1, 1, 1, 1, 1])
mtx = csr_matrix((data, (row, col)), shape=(3, 3))

#second matrix
row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 1, 2, 0, 1, 2])
data = np.array([1, 1, 1, 1, 1, 1])
mtx2 = csr_matrix((data, (row, col)), shape=(3, 3))

#merge two matrix
mtx3 = merge_two_csr_mtx(mtx, mtx2)

#check
print('1st\n',mtx)
print('2nd\n',mtx2)
print('merge\n',mtx3)
..

result
1st
   (0, 0) 1
  (0, 2) 1
  (1, 2) 1
  (2, 0) 1
  (2, 1) 1
  (2, 2) 1
2nd
   (0, 0) 1
  (0, 1) 1
  (1, 2) 1
  (2, 0) 1
  (2, 1) 1
  (2, 2) 1
merge
   (0, 0) 2.0
  (0, 1) 1.0
  (0, 2) 1.0
  (1, 2) 2.0
  (2, 0) 2.0
  (2, 1) 2.0
  (2, 2) 2.0

How to convert a scipy csr_matrix back into lists of row, col and data?

refer to code


..
Define matrix & check values
from scipy.sparse import csr_matrix
import numpy as np
row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 1, 2, 0, 1, 2])
data = np.array([1, 1, 1, 1, 1, 1])
mtx2 = csr_matrix((data, (row, col)), shape=(3, 3))
print(mtx2) #matrix print out
print(mtx2.toarray()) #print out by array

>
(0, 0) 1
  (0, 1) 1
  (1, 2) 1
  (2, 0) 1
  (2, 1) 1
  (2, 2) 1
>
[[1 1 0]
 [0 0 1]
 [1 1 1]]
..


...
get back the row, col and data value from matrix
c = mtx2.tocoo()
print(c.row)
print(c.col)
print(c.data)

>
[0 0 1 2 2 2]
[0 1 2 0 1 2]
[1 1 1 1 1 1]
...

6/09/2020

sentence embedding, sentence to vector using bert

refer to source code

.
#pip install -U sentence-transformers
#https://github.com/UKPLab/sentence-transformers
from sentence_transformers import SentenceTransformer, LoggingHandler

# Load Sentence model (based on BERT) from URL
model = SentenceTransformer('bert-base-nli-mean-tokens')

# Embed a list of sentences
sentences = ['This framework generates embeddings for each input sentence',
'Sentences are passed as a list of string.',
'The quick brown fox jumps over the lazy dog.']
sentence_embeddings = model.encode(sentences)

# The result is a list of sentence embeddings as numpy arrays
for sentence, embedding in zip(sentences, sentence_embeddings):
print("Sentence:", sentence)
print("Embedding:", embedding.shape, type(embedding))
print("")
.

result is like this:
Sentence: This framework generates embeddings for each input sentence
Embedding: (768,) <class 'numpy.ndarray'>

Sentence: Sentences are passed as a list of string.
Embedding: (768,) <class 'numpy.ndarray'>

Sentence: The quick brown fox jumps over the lazy dog.
Embedding: (768,) <class 'numpy.ndarray'>

5/25/2020

install poppler in ubuntu

Try to this command:

sudo apt-get update -y
sudo apt-get install -y poppler-utils

😁

5/19/2020

Ways to sort list of dictionaries by values in Python – Using lambda function


.
#example list
dict_list = [{ "idx":1, "value1":32.44, "value2":123.2}, { "idx":2, "value1":32.414, "value2":133.2}, { "idx":3, "value1":32.244, "value2":113.2}]

#sort by ascending order
sorted_dict_list = sorted(dict_list, key = lambda i: i['value1'])
#sort by descending order
r_sorted_dict_list = sorted(dict_list, key = lambda i: i['value1'],reverse=True)

#show result
print(sorted_dict_list)
# [{'idx': 3, 'value1': 32.244, 'value2': 113.2}, {'idx': 2, 'value1': 32.414, 'value2': 133.2}, {'idx': 1, 'value1': 32.44, 'value2': 123.2}]

print(r_sorted_dict_list)
# [{'idx': 1, 'value1': 32.44, 'value2': 123.2}, {'idx': 2, 'value1': 32.414, 'value2': 133.2}, {'idx': 3, 'value1': 32.244, 'value2': 113.2}]
.


5/15/2020

multi-thread example python source code

The code generate 10 multi threads for running single_function.
If you have look the pid in result, thread is finished by quickly proceeded.

..
import queue
from concurrent.futures import ThreadPoolExecutor

#function for thread
def single_function(input, pid, out_queue):
total = 0
for i in range(0,input):
for j in range(0, input):
for k in range(0, input):
total = total + 1

out_queue.put( {'index':pid, 'result':total })
#run thread
my_queue = queue.Queue()
with ThreadPoolExecutor(max_workers=10) as executor:
for pid in range(0, 10):
executor.submit(single_function, 100, pid, my_queue)
#get result of each thread
result = {}
while not my_queue.empty():
get = my_queue.get()
print(get)

#finish all thread
..

result

{'index': 1, 'result': 1000000}
{'index': 3, 'result': 1000000}
{'index': 2, 'result': 1000000}
{'index': 0, 'result': 1000000}
{'index': 5, 'result': 1000000}
{'index': 4, 'result': 1000000}
{'index': 8, 'result': 1000000}
{'index': 6, 'result': 1000000}
{'index': 9, 'result': 1000000}
{'index': 7, 'result': 1000000}