PySpark MLlib

What is Machine Learning?

• Machine learning is a methos of data analysis that automates analytical model building.
• Using algorithms that iteratively learn from data, machine learning allows computers to find hidden insights without being explicitly programmed where to look.

 

What is it used for?

• Fraud detection. 부정 적발
• Web search results. 웹 검색 결과
• Real-time ads on web pages. 웹페이지 실시간 광고
• Credit scoring and next-best offers. 
• Prediction of equipment failures. 장비 고장 예측
• New pricing models. 가격 모델
• Network intrusion detection. 네트워크 보안 탐지
• Recommendation Engines. 추천 엔진
• Customer Segmentation. 고객 세분화
• Text Sentiment Analysis. 텍스트 감성 분석 
• Predicting Customer Churn 고객 이탈 예측
• Pattern and image recognition. 패턴 및 이미지 분류
• Email spam filtering. 이메일 스팸 필터링
• Financial Modeling. 재무 모델링

 

Spark MLlib

• One of the main "quirks" of using MLlib is that you need to format your data so that eventually it just has one or two columns
  • Supervised(지도학습) : Features, Labels
  • Unsupervised(비지도학습) : Features
• This requires a littel more data processing work than some other machine learing libraries, but the big upside is that this exact same syntax works with distrubuted data, which is no small feat for what is going on "under the hood"

스파크의 머신러닝 라이브러리는 전반적으로 다른 머신 러닝 라이브러리보다 약간 더 많은 데이터 처리 작업이 필요하지만 정확히 동일한 구문이 분산된 데이터에서 작동한다는 점이 큰 장점이다. 

 

https://spark.apache.org/docs/latest/ml-guide.html

MLlib: Main Guide - Spark 3.3.0 Documentation