情景：在機器學習中構(gòu)建模型更重要的是特征，特征又分為很多種，原始基本特征、統(tǒng)計特征、偏好特征等等，今天想記錄一下統(tǒng)計特征，假如給你一個需求讓你統(tǒng)計某一天內(nèi)這些人一天內(nèi)看過別人朋友圈的次數(shù)，估計你會統(tǒng)計，但是讓你統(tǒng)計這一天內(nèi)的人在三天、七天、十五天、三十天內(nèi)看過別人朋友圈的數(shù)量作為幾個特征，想過怎么弄么，給大家分享一個我做過的方法

1、把每一天的數(shù)據(jù)，例如這一天這些人看過別人朋友圈多少次

2、建立hive表，按天分區(qū)

3、把每一天的數(shù)據(jù)load到hive中，其它的辦法也可以

4、之后按照正常的統(tǒng)計邏輯操作就可以啦，分享一下自己寫的部分代碼吧，寫的有點low~

object ownLookownphoto {def main(args: Array[String]): Unit = {val date = List("20190601","20190602","20190603","20190604","20190605")for(item <- date){merge(item)}}def merge(date:String): Unit ={val statis_feature = acquire_statis_feature(date)val next_date = acquireDay(date,1)statis_feature.write.format("csv").option("header","true").save("file_path")}def acquire_statis_feature(date:String): DataFrame ={val result_3 = acquire_day_feature(date,3)val result_7 = acquire_day_feature(date,7)val result_15 = acquire_day_feature(date,15)val result_30 = acquire_day_feature(date,30)val result = result_3.join(result_7,Seq("uid"),"outer").join(result_15,Seq("uid"),"outer").join(result_30,Seq("uid"),"outer")result}def acquireDay(dayStr:String,num:Int): String ={val sdf: SimpleDateFormat = new SimpleDateFormat("yyyyMMdd")val date = sdf.parse(dayStr)val cal: Calendar = Calendar.getInstance()cal.setTime(date)cal.add(Calendar.DATE, num)sdf.format(cal.getTime)}def acquire_day_feature(date:String,inter:Int): DataFrame ={val last_day = acquireDay(date,-inter)val data = spark.sql(s"select * from table_name where day > $last_day and day < $date")val statisresult = data.groupBy("uid").sum("count").withColumnRenamed("sum(count)",s"count_$inter")statisresult}}

?

總結(jié)

以上是生活随笔為你收集整理的特征工程－统计数据特征的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。