【Kaggle微课程】Natural Language Processing - 3. Word Vectors
文章目錄
- 1. 詞嵌入 Word Embeddings
- 2. 分類模型
- 3. 文檔相似度
- 練習:
- 1. 使用文檔向量訓練模型
- 2. 文本相似度
learn from https://www.kaggle.com/learn/natural-language-processing
1. 詞嵌入 Word Embeddings
參考博文:05.序列模型 W2.自然語言處理與詞嵌入 https://michael.blog.csdn.net/article/details/108886394
類似的詞語有著類似的向量表示,向量間可以相減作類比
- 加載模型
- 提取單詞向量
- 合并單詞向量為文檔向量,最簡單的做法是,平均每個單詞的向量
2. 分類模型
有了文檔向量,你可以使用 sklearn 模型、XGB模型等進行建模
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(doc_vectors, spam.label, test_size=0.1, random_state=1)- SVM 的例子
3. 文檔相似度
cosine similarity 余弦相似度 cos?θ=a?b∥a∥∥b∥\cos \theta=\frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}cosθ=∥a∥∥b∥a?b?
def cosine_similarity(a, b):return a.dot(b)/np.sqrt(a.dot(a) * b.dot(b)) a = nlp("REPLY NOW FOR FREE TEA").vector b = nlp("According to legend, Emperor Shen Nung discovered tea when leaves from a wild tree blew into his pot of boiling water.").vector cosine_similarity(a, b)輸出:
0.7030031練習:
試試你為餐館建立的情緒分析模型。在給定的一些示例文本的數據集中找到最相似的評論。
%matplotlib inlineimport matplotlib.pyplot as plt import numpy as np import pandas as pd import spacy# Set up code checking from learntools.core import binder binder.bind(globals()) from learntools.nlp.ex3 import * print("\nSetup complete")- 加載模型、數據
- 為了節省時間,加載已經處理好的所有評論詞向量
1. 使用文檔向量訓練模型
- SVM
輸出:
Model test accuracy: 93.847%- KNN
輸出:
Model test accuracy: 86.998%2. 文本相似度
- Centering the Vectors
有時在計算相似性時,人們會計算所有文檔的平均向量,然后每個文檔的向量減去這個向量。為什么你認為這有助于相似性度量?
有時候你的文檔已經相當相似了。例如,這個數據集是對企業的所有評論,這些文檔之間有很強的相似度,與新聞文章、技術手冊和食譜相比。最終你得到0.8和1之間的所有相似性,并且沒有反相似文檔(相似性<0)。當中心化向量時,您將比較數據集中的文檔,而不是所有可能的文檔。
- 找到最相似的評論
輸出:
After purchasing my final christmas gifts at the Urban Tea Merchant in Vancouver, I was surprised to hear about Teopia at the new outdoor mall at Don Mills and Lawrence when I went back home to Toronto for Christmas. Across from the outdoor skating rink and perfect to sit by the ledge to people watch, the location was prime for tea connesieurs... or people who are just freezing cold in need of a drinK! Like any gourmet tea shop, there were large tins of tea leaves on the walls, and although the tea menu seemed interesting enough, you can get any specialty tea as your drink. We didn't know what to get... so the lady suggested the Goji Berries... it smelled so succulent and juicy... instantly SOLD! I got it into a tea latte and watched the tea steep while the milk was steamed, and surprisingly, with the click of a button, all the water from the tea can be instantly drained into the cup (see photo).. very fascinating!The tea was aromatic and tasty, not over powering. The price was also very reasonable and I recommend everyone to get a taste of this place :)- 評論1
- 與評論1最相似的評論
- 看看相似的評論
如果你看看其他類似的評論,你會看到很多咖啡店。為什么你認為咖啡評論和只提到茶的例子評論相似?
咖啡店的評論也將類似于我們的茶館評論,因為咖啡和茶在語義上是相似的。大多數咖啡館都提供咖啡和茶,所以你會經常看到這兩個詞同時出現。
刷完了課程,獲得鼓勵證書,繼續加油!
我的CSDN博客地址 https://michael.blog.csdn.net/
長按或掃碼關注我的公眾號(Michael阿明),一起加油、一起學習進步!
總結
以上是生活随笔為你收集整理的【Kaggle微课程】Natural Language Processing - 3. Word Vectors的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: TensorFlow 2.0 - CNN
- 下一篇: 05.序列模型 W1.循环序列模型(作业