[Python] 머신러닝 완벽가이드 - 08. 텍스트 분석[문서 군집화]

Updated:

파이썬 머신러닝 완벽가이드 교재를 토대로 공부한 내용입니다.

실습과정에서 필요에 따라 내용의 누락 및 추가, 수정사항이 있습니다.


기본 세팅

import numpy as np
import pandas as pd

import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

import warnings
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

mpl.rc('font', family='NanumGothic') # 폰트 설정
mpl.rc('axes', unicode_minus=False) # 유니코드에서 음수 부호 설정

# 차트 스타일 설정
sns.set(font="NanumGothic", rc={"axes.unicode_minus":False}, style='darkgrid')
plt.rc("figure", figsize=(10,8))

warnings.filterwarnings("ignore")

5. 문서 군집화

문서 군집화는 비슷한 텍스트 구성의 문서를 군집화 하는 방법이다.

앞서 공부했던 텍스트 분류와 비슷하지만 문서 군집화는 비지도학습 기반이다.

5.1 Opinion Review 실습

데이터는 UCI Machine Learning Repository에서 다운 받을 수 있는 Opinion Review 데이터를 사용한다.

이 데이터는 전자기기, 자동차, 호텔에 대한 리뷰 파일로 구성되어 있다.

전자기기는 다시 내비게이션, 아이팟, 킨들, 넷북 등 세부 요소로 나뉜다.

5.1.1 데이터 불러오기

import glob, os

# 경로 지정 (r string으로 탈출문자 그대로 인식)
path = r'C:\Users\ekzm3\Desktop\Github_kkd\Python_Study_ML\08.텍스트분석\OpinosisDataset1.0\topics'

# path에 존재하는 .data 파일들의 파일명을 리스트로 취합
all_files = glob.glob(os.path.join(path, "*.data"))

filename_list = []
opinion_text = []

for file in all_files:
    # 경로 등 제거 후 순수 파일명만 저장
    filename_ = file.split('\\')[-1]
    filename = filename_.split('.')[0]
    filename_list.append(filename)
    
    # 각 파일 데이터 프레임으로 생성 후 to_string으로 text화
    df = pd.read_table(file, index_col=None, header=0, encoding='latin1')
    opiniontext = df.to_string().replace("   ", "") # 첫 공백 제거
    opinion_text.append(opiniontext)
    
# 파일명, 파일내용을 데이터 프레임으로 생성
document_df = pd.DataFrame({'filename':filename_list, 'opinion_text':opinion_text})
document_df.head()
filename opinion_text
0 accuracy_garmin_nuvi_255W_gps , and is very, very accurate .\n0but for the m...
1 bathroom_bestwestern_hotel_sfo The room was not overly big, but clean and ve...
2 battery-life_amazon_kindle After I plugged it in to my USB hub on my com...
3 battery-life_ipod_nano_8gb short battery life I moved up from an 8gb ....
4 battery-life_netbook_1005ha 6GHz 533FSB cpu, glossy display, 3, Cell 23Wh...
  • Opinion Review 데이터는 총 51개의 데이터로 구성되어 있다.

  • 각 파일명과 해당 파일 내용을 데이터 프레임으로 생성하였다.

  • 이 파일을 이용해서 군집화를 진행해보자.

5.1.2 피처 벡터화

tokenizer 함수

from nltk.stem import WordNetLemmatizer
import nltk
import string

# 단어 원형 추출 함수
lemmar = WordNetLemmatizer()
def LemTokens(tokens):
    return [lemmar.lemmatize(token) for token in tokens]

# 특수 문자 사전 생성: {33: None ...}
# ord(): 아스키 코드 생성
remove_punct_dict = dict((ord(punct), None) for punct in string.punctuation)

# 특수 문자 제거 및 단어 원형 추출
def LemNormalize(text):
    # 텍스트 소문자 변경 후 특수 문자 제거
    text_new = text.lower().translate(remove_punct_dict)
    
    # 단어 토큰화
    word_tokens = nltk.word_tokenize(text_new)
    
    # 단어 원형 추출
    return LemTokens(word_tokens)
  • string.punctuation은 느낌표, 물음표, 더하기 등의 문자들이다.

  • dict((ord(punct), None) for punct in string.punctuation)로 해당 문자 사전을 생성하였다.

  • text.lower().translate(remove_punct_dict)로 문자 사전에 따라 None으로 변환한다.

  • 그 후 단어 토큰화 후에 토큰별로 원형을 추출한다.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vect = TfidfVectorizer(stop_words='english' , ngram_range=(1,2), 
                             tokenizer = LemNormalize, min_df=0.05, max_df=0.85)

# 피처 벡터화: TF-IDF
feature_vect = tfidf_vect.fit_transform(document_df['opinion_text'])
  • 피처 벡터화로 TF-IDF를 사용했고 tokenizer로 앞서 만든 함수를 적용하였다.

  • 앞선 포스팅에서 공부하였듯이 피처 벡터화에서 Stemmer, Lemmatize는 tokenizer로 수행한다.

5.1.3 K-Means(5)

from sklearn.cluster import KMeans

# KMeans: 5
km_cluster = KMeans(n_clusters=5, max_iter=10000, random_state=0)
km_cluster.fit(feature_vect)

# cluster 및 중심 좌표 정보
cluster_label = km_cluster.labels_
cluster_centers = km_cluster.cluster_centers_

# cluster 라벨 추가
document_df['cluster_label'] = cluster_label
document_df.head()
filename opinion_text cluster_label
0 accuracy_garmin_nuvi_255W_gps , and is very, very accurate .\n0but for the m... 4
1 bathroom_bestwestern_hotel_sfo The room was not overly big, but clean and ve... 0
2 battery-life_amazon_kindle After I plugged it in to my USB hub on my com... 1
3 battery-life_ipod_nano_8gb short battery life I moved up from an 8gb .... 1
4 battery-life_netbook_1005ha 6GHz 533FSB cpu, glossy display, 3, Cell 23Wh... 1
  • 군집화는 K-Means를 이용하며 군집 수는 5개로 설정하였다.

  • 각 군집별로 유사한 주제가 구성되었는지 확인해보자.

document_df[document_df['cluster_label']==0].sort_values(by='filename')
filename opinion_text cluster_label
1 bathroom_bestwestern_hotel_sfo The room was not overly big, but clean and ve... 0
13 food_holiday_inn_london The room was packed to capacity with queues ... 0
14 food_swissotel_chicago The food for our event was delicious .\n0 Th... 0
15 free_bestwestern_hotel_sfo The wine reception is a great idea as it is ... 0
20 location_bestwestern_hotel_sfo Good Value good location , ideal choice .\n... 0
21 location_holiday_inn_london Great location for tube and we crammed in a f... 0
24 parking_bestwestern_hotel_sfo Parking was expensive but I think this is comm... 0
28 price_holiday_inn_london All in all, a normal chain hotel on a nice lo... 0
32 room_holiday_inn_london We arrived at 23,30 hours and they could not r... 0
30 rooms_bestwestern_hotel_sfo Great Location , NiceRooms , Helpless Conci... 0
31 rooms_swissotel_chicago The Swissotel is one of our favorite hotels in... 0
38 service_bestwestern_hotel_sfo Both of us having worked in tourism for over 1... 0
39 service_holiday_inn_london not customer, oriented hotelvery low service ... 0
40 service_swissotel_hotel_chicago Mediocre room and service for a very extravaga... 0
45 staff_bestwestern_hotel_sfo Staff are friendly and helpful.\n0The staff i... 0
46 staff_swissotel_chicago The staff at Swissotel were not particularly... 0
  • 군집 0의 파일명을 보면 호텔에 대한 리뷰로 군집화되어 있다.
document_df[document_df['cluster_label']==1].sort_values(by='filename')
filename opinion_text cluster_label
2 battery-life_amazon_kindle After I plugged it in to my USB hub on my com... 1
3 battery-life_ipod_nano_8gb short battery life I moved up from an 8gb .... 1
4 battery-life_netbook_1005ha 6GHz 533FSB cpu, glossy display, 3, Cell 23Wh... 1
11 features_windows7 I had to uninstall anti, virus and selected ot... 1
19 keyboard_netbook_1005ha , I think the new keyboard rivals the great... 1
26 performance_netbook_1005ha The Eee Super Hybrid Engine utility lets user... 1
34 screen_garmin_nuvi_255W_gps It is easy to read and when touching the scre... 1
35 screen_ipod_nano_8gb As always, the video screen is sharp and brigh... 1
36 screen_netbook_1005ha Keep in mind that once you get in a room full ... 1
41 size_asus_netbook_1005ha A few other things I'd like to point out is th... 1
42 sound_ipod_nano_8gb headphone jack i got a clear case for it and ... 1
49 video_ipod_nano_8gb I bought the 8, gig Ipod Nano that has the bu... 1
  • 군집 1은 킨들, 아이팟, 넷북 등 전자기기에 대한 리뷰로 군집화되어 있다.
document_df[document_df['cluster_label']==2].sort_values(by='filename')
filename opinion_text cluster_label
6 comfort_honda_accord_2008 Drivers seat not comfortable, the car itself ... 2
7 comfort_toyota_camry_2007 Ride seems comfortable and gas mileage fairly ... 2
16 gas_mileage_toyota_camry_2007 Ride seems comfortable and gas mileage fairly... 2
17 interior_honda_accord_2008 I love the new body style and the interior is ... 2
18 interior_toyota_camry_2007 First of all, the interior has way too many ch... 2
22 mileage_honda_accord_2008 It's quiet, get good gas mileage and looks cle... 2
25 performance_honda_accord_2008 Very happy with my 08 Accord, performance is... 2
29 quality_toyota_camry_2007 I previously owned a Toyota 4Runner which had... 2
37 seats_honda_accord_2008 Front seats are very uncomfortable .\n0 No mem... 2
47 transmission_toyota_camry_2007 After slowing down, transmission has to be ... 2
  • 군집 2는 토요타, 혼다 등 자동차에 대한 리뷰로 군집화되어 있다.
document_df[document_df['cluster_label']==3].sort_values(by='filename')
filename opinion_text cluster_label
5 buttons_amazon_kindle I thought it would be fitting to christen my ... 3
10 eyesight-issues_amazon_kindle It feels as easy to read as the K1 but doesn't... 3
12 fonts_amazon_kindle Being able to change the font sizes is awesom... 3
23 navigation_amazon_kindle In fact, the entire navigation structure has... 3
27 price_amazon_kindle If a case was included, as with the Kindle 1,... 3
44 speed_windows7 Windows 7 is quite simply faster, more stable... 3
  • 군집 3은 킨들에 대한 리뷰로 군집화되어 있다.
document_df[document_df['cluster_label']==4].sort_values(by='filename')
filename opinion_text cluster_label
0 accuracy_garmin_nuvi_255W_gps , and is very, very accurate .\n0but for the m... 4
8 directions_garmin_nuvi_255W_gps You also get upscale features like spoken di... 4
9 display_garmin_nuvi_255W_gps 3 quot widescreen display was a bonus .\n0 ... 4
33 satellite_garmin_nuvi_255W_gps It's fast to acquire satellites .\n0 If you'v... 4
43 speed_garmin_nuvi_255W_gps Another feature on the 255w is a display of t... 4
48 updates_garmin_nuvi_255W_gps Another thing to consider was that I paid $50 ... 4
50 voice_garmin_nuvi_255W_gps The voice prompts and maps are wonderful espe... 4
  • 군집 4는 네비게이션에 대한 리뷰로 군집화되어 있다.

5.1.4 K-Means(3)

앞서 5개의 군집을 3개로 낮춰 결과를 확인해보자.

from sklearn.cluster import KMeans

# KMeans: 3
km_cluster = KMeans(n_clusters=3, max_iter=10000, random_state=0)
km_cluster.fit(feature_vect)

# cluster 및 중심 좌표 정보
cluster_label = km_cluster.labels_
cluster_centers = km_cluster.cluster_centers_

# cluster 라벨 추가
document_df['cluster_label'] = cluster_label
document_df[document_df['cluster_label']==0].sort_values(by='filename')
filename opinion_text cluster_label
0 accuracy_garmin_nuvi_255W_gps , and is very, very accurate .\n0but for the m... 0
2 battery-life_amazon_kindle After I plugged it in to my USB hub on my com... 0
3 battery-life_ipod_nano_8gb short battery life I moved up from an 8gb .... 0
4 battery-life_netbook_1005ha 6GHz 533FSB cpu, glossy display, 3, Cell 23Wh... 0
5 buttons_amazon_kindle I thought it would be fitting to christen my ... 0
8 directions_garmin_nuvi_255W_gps You also get upscale features like spoken di... 0
9 display_garmin_nuvi_255W_gps 3 quot widescreen display was a bonus .\n0 ... 0
10 eyesight-issues_amazon_kindle It feels as easy to read as the K1 but doesn't... 0
11 features_windows7 I had to uninstall anti, virus and selected ot... 0
12 fonts_amazon_kindle Being able to change the font sizes is awesom... 0
19 keyboard_netbook_1005ha , I think the new keyboard rivals the great... 0
23 navigation_amazon_kindle In fact, the entire navigation structure has... 0
26 performance_netbook_1005ha The Eee Super Hybrid Engine utility lets user... 0
27 price_amazon_kindle If a case was included, as with the Kindle 1,... 0
33 satellite_garmin_nuvi_255W_gps It's fast to acquire satellites .\n0 If you'v... 0
34 screen_garmin_nuvi_255W_gps It is easy to read and when touching the scre... 0
35 screen_ipod_nano_8gb As always, the video screen is sharp and brigh... 0
36 screen_netbook_1005ha Keep in mind that once you get in a room full ... 0
41 size_asus_netbook_1005ha A few other things I'd like to point out is th... 0
42 sound_ipod_nano_8gb headphone jack i got a clear case for it and ... 0
43 speed_garmin_nuvi_255W_gps Another feature on the 255w is a display of t... 0
44 speed_windows7 Windows 7 is quite simply faster, more stable... 0
48 updates_garmin_nuvi_255W_gps Another thing to consider was that I paid $50 ... 0
49 video_ipod_nano_8gb I bought the 8, gig Ipod Nano that has the bu... 0
50 voice_garmin_nuvi_255W_gps The voice prompts and maps are wonderful espe... 0
  • 군집 0은 전자기기에 대한 리뷰만으로 잘 군집화되어 있다.
document_df[document_df['cluster_label']==1].sort_values(by='filename')
filename opinion_text cluster_label
1 bathroom_bestwestern_hotel_sfo The room was not overly big, but clean and ve... 1
13 food_holiday_inn_london The room was packed to capacity with queues ... 1
14 food_swissotel_chicago The food for our event was delicious .\n0 Th... 1
15 free_bestwestern_hotel_sfo The wine reception is a great idea as it is ... 1
20 location_bestwestern_hotel_sfo Good Value good location , ideal choice .\n... 1
21 location_holiday_inn_london Great location for tube and we crammed in a f... 1
24 parking_bestwestern_hotel_sfo Parking was expensive but I think this is comm... 1
28 price_holiday_inn_london All in all, a normal chain hotel on a nice lo... 1
32 room_holiday_inn_london We arrived at 23,30 hours and they could not r... 1
30 rooms_bestwestern_hotel_sfo Great Location , NiceRooms , Helpless Conci... 1
31 rooms_swissotel_chicago The Swissotel is one of our favorite hotels in... 1
38 service_bestwestern_hotel_sfo Both of us having worked in tourism for over 1... 1
39 service_holiday_inn_london not customer, oriented hotelvery low service ... 1
40 service_swissotel_hotel_chicago Mediocre room and service for a very extravaga... 1
45 staff_bestwestern_hotel_sfo Staff are friendly and helpful.\n0The staff i... 1
46 staff_swissotel_chicago The staff at Swissotel were not particularly... 1
  • 군집 1은 호텔에 대한 리뷰만으로 잘 군집화되어 있다.
document_df[document_df['cluster_label']==2].sort_values(by='filename')
filename opinion_text cluster_label
6 comfort_honda_accord_2008 Drivers seat not comfortable, the car itself ... 2
7 comfort_toyota_camry_2007 Ride seems comfortable and gas mileage fairly ... 2
16 gas_mileage_toyota_camry_2007 Ride seems comfortable and gas mileage fairly... 2
17 interior_honda_accord_2008 I love the new body style and the interior is ... 2
18 interior_toyota_camry_2007 First of all, the interior has way too many ch... 2
22 mileage_honda_accord_2008 It's quiet, get good gas mileage and looks cle... 2
25 performance_honda_accord_2008 Very happy with my 08 Accord, performance is... 2
29 quality_toyota_camry_2007 I previously owned a Toyota 4Runner which had... 2
37 seats_honda_accord_2008 Front seats are very uncomfortable .\n0 No mem... 2
47 transmission_toyota_camry_2007 After slowing down, transmission has to be ... 2
  • 군집 2는 자동차에 대한 리뷰만으로 잘 군집화되어 있다.

5.1.5 군집별 핵심 단어 추출

KMeans 포스팅에서 공부하였듯이 KMeans의 cluster_centers_ 속성은 각 군집별 피처의 중심점 좌표를 가지고 있다.

여기선 각 군집을 구성하는 word 피처가 군집의 중심을 기준으로 얼마나 가까운지로 해석 가능할 것이다.

cluster_centers = km_cluster.cluster_centers_

print('cluster_centers shape :', cluster_centers.shape)
print(cluster_centers)
cluster_centers shape : (3, 4613)
[[0.00760308 0.00777633 0.         ... 0.0067567  0.         0.        ]
 [0.00263248 0.         0.0017299  ... 0.         0.00190972 0.00146615]
 [0.00334841 0.         0.         ... 0.         0.         0.        ]]
  • 3개의 군집에 4,613개의 word 피처가 개별 군집 중심과 얼마나 가까운지 확인 가능하다.

  • 이를 이용해서 어떤 단어가 핵심 단어인지 추출해보자.

군집별 핵심 단어 추출 함수

def get_cluster_details(cluster_model, cluster_data, cluster_nums, 
                        feature_names, top_n_features=10):
    
    # 핵심 단어 등 정보를 담을 사전 생성
    cluster_details = {}
    
    # word 피처 중심과의 거리 내림차순 정렬시 값들의 index 반환
    center_info = cluster_model.cluster_centers_        # 군집 중심 정보
    center_descend_ind = center_info.argsort()[:, ::-1] # 행별(군집별)로 역순 정렬
    
    # 군집별 정보 담기
    for i in range(cluster_nums):
        # 군집별 정보를 담을 데이터 초기화
        cluster_details[i] = {} # 사전 안에 사전
        
        # 각 군집에 속하는 파일명
        filenames = cluster_data[cluster_data["cluster_label"] == i]["filename"]
        filenames = filenames.values.tolist()
        
        # 군집별 중심 정보
        top_feature_values = center_info[i, :top_n_features].tolist()

        # 군집별 핵심 단어 피처명
        top_feature_indexes = center_descend_ind[i, :top_n_features]
        top_features = [feature_names[ind] for ind in top_feature_indexes]
        
        # 각 군집별 정보 사전에 담기
        cluster_details[i]["cluster"] = i                              # i번째 군집
        cluster_details[i]["top_features"] = top_features              # 군집별 핵심 단어
        cluster_details[i]["top_feature_values"] = top_feature_values  # 군집별 중심 정보
        cluster_details[i]["filenames"] = filenames                    # 군집 속 파일명
        
    return cluster_details
  • 각 군집별로 핵심 단어, 중심 정보, 해당 군집에 속하는 파일명을 사전으로 반환한다.
# TF-IDF 객체의 전체 word 명칭
feature_names = tfidf_vect.get_feature_names()

# 함수 적용
cluster_details = get_cluster_details(cluster_model=km_cluster, cluster_data=document_df, cluster_nums=3,
                                  feature_names=feature_names, top_n_features=10 )
cluster_details.keys()
dict_keys([0, 1, 2])
  • 사전에는 K-Means에서 설정한 3개의 군집이 key 값으로 잘 저장되어있다.
cluster_details[0]
{'cluster': 0,
 'top_features': ['screen',
  'battery',
  'keyboard',
  'battery life',
  'life',
  'kindle',
  'video',
  'direction',
  'size',
  'voice'],
 'top_feature_values': [0.007603082102281897,
  0.007776334714273187,
  0.0,
  0.0008932639588010036,
  0.0,
  0.0,
  0.0005330648826557408,
  0.0,
  0.0006005763034298585,
  0.0010427607182775505],
 'filenames': ['accuracy_garmin_nuvi_255W_gps',
  'battery-life_amazon_kindle',
  'battery-life_ipod_nano_8gb',
  'battery-life_netbook_1005ha',
  'buttons_amazon_kindle',
  'directions_garmin_nuvi_255W_gps',
  'display_garmin_nuvi_255W_gps',
  'eyesight-issues_amazon_kindle',
  'features_windows7',
  'fonts_amazon_kindle',
  'keyboard_netbook_1005ha',
  'navigation_amazon_kindle',
  'performance_netbook_1005ha',
  'price_amazon_kindle',
  'satellite_garmin_nuvi_255W_gps',
  'screen_garmin_nuvi_255W_gps',
  'screen_ipod_nano_8gb',
  'screen_netbook_1005ha',
  'size_asus_netbook_1005ha',
  'sound_ipod_nano_8gb',
  'speed_garmin_nuvi_255W_gps',
  'speed_windows7',
  'updates_garmin_nuvi_255W_gps',
  'video_ipod_nano_8gb',
  'voice_garmin_nuvi_255W_gps']}
  • 첫 번째 군집의 정보를 보면 원하는 정보가 잘 저장되어 있다.
for cluster_num, cluster_detail in cluster_details.items():
        print(f"####### Cluster {cluster_num}")
        print(f"Top features: {cluster_detail['top_features']}")
        print(f"Reviews 파일명: {cluster_detail['filenames'][:3]}")
        print("-"*120)
####### Cluster 0
Top features: ['screen', 'battery', 'keyboard', 'battery life', 'life', 'kindle', 'video', 'direction', 'size', 'voice']
Reviews 파일명: ['accuracy_garmin_nuvi_255W_gps', 'battery-life_amazon_kindle', 'battery-life_ipod_nano_8gb']
------------------------------------------------------------------------------------------------------------------------
####### Cluster 1
Top features: ['room', 'hotel', 'service', 'staff', 'food', 'location', 'bathroom', 'clean', 'price', 'parking']
Reviews 파일명: ['bathroom_bestwestern_hotel_sfo', 'food_holiday_inn_london', 'food_swissotel_chicago']
------------------------------------------------------------------------------------------------------------------------
####### Cluster 2
Top features: ['interior', 'seat', 'mileage', 'comfortable', 'gas', 'gas mileage', 'car', 'transmission', 'performance', 'quality']
Reviews 파일명: ['comfort_honda_accord_2008', 'comfort_toyota_camry_2007', 'gas_mileage_toyota_camry_2007']
------------------------------------------------------------------------------------------------------------------------
  • 군집 0은 전자기기에 관한 리뷰로 screen, battery 등이 핵심 단어로 사용됐다.

  • 군집 1은 호텔에 관한 리뷰로 room, hotel, service 등 방과 서비스에 대한 단어가 핵심 단어로 사용됐다.

  • 군집 2도 마찬가지로 핵심 단어 파악이 가능하고, 이를 통해 각 군집별 주요 관심사를 직접 확인 가능하다.

Leave a comment