7/29/2020

ROC & AUC example code in face detector model case



..

#https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html
import numpy as np
from sklearn import metrics
import matplotlib.pyplot as plt

#model #1
y = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0])
scores = np.array([0.64, 0.47, 0.46, 0.77, 0.72, 0.9, 0.85, 0.7, 0.87, 0.92, 0.89, 0.93, 0.85, 0.81, 0.88, 0.48, 0.1, 0.35, 0.68, 0.47])
fpr, tpr, thresholds = metrics.roc_curve(y, scores)
roc_auc = metrics.auc(fpr, tpr)

# plot
plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

..


7/28/2020

Example model metrics using sklearn in face detector case


..

from sklearn.metrics import classification_report
#model 1
y_true = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
y_pred = [0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
target_names = ['Non Face', 'Face']
print(classification_report(y_true, y_pred, target_names=target_names, digits=3))
..



..

#model 2
y_true = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
y_pred = [0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0]
target_names = ['Non Face', 'Face']
print(classification_report(y_true, y_pred, target_names=target_names, digits=3))
..


7/07/2020

extract year, month, day from file on Ubuntu, python example


...
import os, time
date_created_obj = time.localtime(os.path.getctime(full_path))
print('Year: {:4d}'.format(date_created_obj.tm_year)) # Year: 2020
print('Month: {:2d}'.format(date_created_obj.tm_mon)) # Month: 2
print('Day: {:2d}'.format(date_created_obj.tm_mday)) # Day: 10

...


7/06/2020

how to merge two csr_matrix, example python source code

let's see the code.

..
from scipy.sparse import csr_matrix
import numpy as np

#first matrix
row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 2, 2, 0, 1, 2])
data = np.array([1, 1, 1, 1, 1, 1])
mtx = csr_matrix((data, (row, col)), shape=(3, 3))

#second matrix
row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 1, 2, 0, 1, 2])
data = np.array([1, 1, 1, 1, 1, 1])
mtx2 = csr_matrix((data, (row, col)), shape=(3, 3))

#merge two matrix
mtx3 = merge_two_csr_mtx(mtx, mtx2)

#check
print('1st\n',mtx)
print('2nd\n',mtx2)
print('merge\n',mtx3)
..

result
1st
   (0, 0) 1
  (0, 2) 1
  (1, 2) 1
  (2, 0) 1
  (2, 1) 1
  (2, 2) 1
2nd
   (0, 0) 1
  (0, 1) 1
  (1, 2) 1
  (2, 0) 1
  (2, 1) 1
  (2, 2) 1
merge
   (0, 0) 2.0
  (0, 1) 1.0
  (0, 2) 1.0
  (1, 2) 2.0
  (2, 0) 2.0
  (2, 1) 2.0
  (2, 2) 2.0

How to convert a scipy csr_matrix back into lists of row, col and data?

refer to code


..
Define matrix & check values
from scipy.sparse import csr_matrix
import numpy as np
row = np.array([0, 0, 1, 2, 2, 2])
col = np.array([0, 1, 2, 0, 1, 2])
data = np.array([1, 1, 1, 1, 1, 1])
mtx2 = csr_matrix((data, (row, col)), shape=(3, 3))
print(mtx2) #matrix print out
print(mtx2.toarray()) #print out by array

>
(0, 0) 1
  (0, 1) 1
  (1, 2) 1
  (2, 0) 1
  (2, 1) 1
  (2, 2) 1
>
[[1 1 0]
 [0 0 1]
 [1 1 1]]
..


...
get back the row, col and data value from matrix
c = mtx2.tocoo()
print(c.row)
print(c.col)
print(c.data)

>
[0 0 1 2 2 2]
[0 1 2 0 1 2]
[1 1 1 1 1 1]
...

6/09/2020

sentence embedding, sentence to vector using bert

refer to source code

.
#pip install -U sentence-transformers
#https://github.com/UKPLab/sentence-transformers
from sentence_transformers import SentenceTransformer, LoggingHandler

# Load Sentence model (based on BERT) from URL
model = SentenceTransformer('bert-base-nli-mean-tokens')

# Embed a list of sentences
sentences = ['This framework generates embeddings for each input sentence',
'Sentences are passed as a list of string.',
'The quick brown fox jumps over the lazy dog.']
sentence_embeddings = model.encode(sentences)

# The result is a list of sentence embeddings as numpy arrays
for sentence, embedding in zip(sentences, sentence_embeddings):
print("Sentence:", sentence)
print("Embedding:", embedding.shape, type(embedding))
print("")
.

result is like this:
Sentence: This framework generates embeddings for each input sentence
Embedding: (768,) <class 'numpy.ndarray'>

Sentence: Sentences are passed as a list of string.
Embedding: (768,) <class 'numpy.ndarray'>

Sentence: The quick brown fox jumps over the lazy dog.
Embedding: (768,) <class 'numpy.ndarray'>

5/25/2020

install poppler in ubuntu

Try to this command:

sudo apt-get update -y
sudo apt-get install -y poppler-utils

😁