In this Post , we will learn about from_unixtime in pyspark with example .
Sample program
Inorder to pass the date parameter into a column in the dataframe , we will go with this option .
Using lit () we can pass any value into the dataframe . But the date values passed through can’t be retrieved properly .
Here , unix_timestamp() and from_unixtime() helps us to do the above easily.
import findspark
findspark.init()
from pyspark import SparkContext,SparkConf
from pyspark.sql.functions import *
sc=SparkContext.getOrCreate()
df.select(from_unixtime(unix_timestamp(lit('2018-09-30 00:00:00')),'yyyy-MM-dd')).alias('months_add')
Print("Printing df below")
df.show()Output
Printing df below
+-----------------------------------------------------------------------------------+
|from_unixtime(unix_timestamp(2018-09-30 00:00:00, yyyy-MM-dd HH:mm:ss), yyyy-MM-dd)|
+-----------------------------------------------------------------------------------+
| 2018-09-30|
+-----------------------------------------------------------------------------------+Reference
https://spark.apache.org/docs/2.2.0/api/python/pyspark.sql.html#pyspark.sql.functions.from_unixtime