BeginnersBug

pyspark

When otherwise in pyspark with examples

Post author By Usha Nandhini
Post date February 25, 2020
No Comments on When otherwise in pyspark with examples

In this post , We will learn about When otherwise in pyspark with examples

when otherwise used as a condition statements like if else statement

In below examples we will learn with single,multiple & logic conditions

Sample program – Single condition check

In Below example, df is a dataframe with three records .

df1 is a new dataframe created from df by adding one more column named as First_Level .

import findspark 
findspark.init() 
from pyspark import SparkContext,SparkConf 
from pyspark.sql import Row 
from pyspark.sql.functions import * 

sc=SparkContext.getOrCreate() 
#creating dataframe with three records
df=sc.parallelize([Row(name='Gokul',Class=10,marks=480,grade='A'),Row(name='Usha',Class=12,marks=450,grade='A'),Row(name='Rajesh',Class=12,marks=430,grade='B')]).toDF() 
print("Printing df dataframe below ")
df.show() 
df1=df.withColumn("First_Level",when(col("grade") =='A',"Good").otherwise("Average")) 
print("Printing df1 dataframe below ")
df1.show()

Output

print("printing df")
+-----+-----+-----+------+
|Class|grade|marks|  name|
+-----+-----+-----+------+
|   10|    A|  480| Gokul|
|   12|    A|  450|  Usha|
|   12|    B|  430|Rajesh|
+-----+-----+-----+------+
print("printing df1")
+-----+-----+-----+------+-----------+
|Class|grade|marks|  name|First_Level|
+-----+-----+-----+------+-----------+
|   10|    A|  480| Gokul|       Good|
|   12|    A|  450|  Usha|       Good|
|   12|    B|  430|Rajesh|    Average|
+-----+-----+-----+------+-----------+

Sample program – Multiple checks

We can check multiple conditions using when otherwise as like below

import findspark 
findspark.init() 
from pyspark import SparkContext,SparkConf 
from pyspark.sql import Row 
from pyspark.sql.functions import * 

sc=SparkContext.getOrCreate() 
#creating dataframe with three records
df=sc.parallelize([Row(name='Gokul',Class=10,marks=480,grade='A'),Row(name='Usha',Class=12,marks=450,grade='A'),Row(name='Rajesh',Class=12,marks=430,grade='B')]).toDF() 
print("Printing df dataframe below")
df.show()
#In below line we are using multiple condition
df2=df.withColumn("Second_Level",when(col("grade") == 'A','Excellent').when(col("grade") == 'B','Good').otherwise("Average"))
print("Printing df2 dataframe below")
df2.show()

Output

The column Second_Level is created from the above program

Printing df dataframe below
+-----+-----+-----+------+
|Class|grade|marks|  name|
+-----+-----+-----+------+
|   10|    A|  480| Gokul|
|   12|    A|  450|  Usha|
|   12|    B|  430|Rajesh|
+-----+-----+-----+------+

Printing df2 dataframe below
+-----+-----+-----+------+------------+
|Class|grade|marks|  name|Second_Level|
+-----+-----+-----+------+------------+
|   10|    A|  480| Gokul|   Excellent|
|   12|    A|  450|  Usha|   Excellent|
|   12|    B|  430|Rajesh|        Good|
+-----+-----+-----+------+------------+

Sample program with logical operators & and |

Logical operators & (AND) , |(OR) is used in when otherwise as like below .

import findspark 
findspark.init() 
from pyspark import SparkContext,SparkConf 
from pyspark.sql import Row 
from pyspark.sql.functions import * 
sc=SparkContext.getOrCreate() 
#creating dataframe with three records
df=sc.parallelize([Row(name='Gokul',Class=10,marks=480,grade='A'),Row(name='Usha',Class=12,marks=450,grade='A'),Row(name='Rajesh',Class=12,marks=430,grade='B'),Row(name='Mahi',Class=5,marks=350,grade='C')]).toDF() 
print("Printing df dataframe")
df.show()
# In below line we are using logical operators
df3=df.withColumn("Third_Level",when((col("grade") =='A') | (col("Marks") > 450) ,"Excellent").when((col("grade") =='B') | ((col("Marks") > 400) & (col("Marks") < 450)),"Good").otherwise("Average") )
print("Printing  df3 dataframe ")
df3.show()

Output

Printing df dataframe
+-----+-----+-----+------+
|Class|grade|marks|  name|
+-----+-----+-----+------+
|   10|    A|  480| Gokul|
|   12|    A|  450|  Usha|
|   12|    B|  430|Rajesh|
|    5|    C|  350|  Mahi|
+-----+-----+-----+------+

Printing  df3 dataframe 
+-----+-----+-----+------+-----------+
|Class|grade|marks|  name|Third_Level|
+-----+-----+-----+------+-----------+
|   10|    A|  480| Gokul|  Excellent|
|   12|    A|  450|  Usha|  Excellent|
|   12|    B|  430|Rajesh|       Good|
|    5|    C|  350|  Mahi|    Average|
+-----+-----+-----+------+-----------+

Reference

https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.functions.when

case when statement in pyspark with example

Tags pyspark, whenotherwise

collections java

iterate stream with index in Java8

Post author By Rajesh Kumar
Post date February 25, 2020
No Comments on iterate stream with index in Java8

In this post, we will learn to iterate stream with index in Java8

We will come across multiple streams tutorials, but in all the tutorials we never iterate with index numbers

In order to find the current index or other numeric operations index number will help us to get the position

In this tutorial, we are going to use IntStream to get index of the arraylist

Because there is no straight way to get index number in streams

Syntax

IntStream.range(start, end)

Example


import java.util.ArrayList;
import java.util.List;
import java.util.stream.IntStream;

public class IterateStreamWithIndex {

	public static void main(String[] args) {
		try {
			List<String> list = new ArrayList<String>();
			list.add("India");
			list.add("USA");
			list.add("Germany");

			IntStream.range(0, list.size())
					.forEach(x -> System.out.println("The index is " + x + " Value is " + list.get(x)));

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

}

Output

The index is 0 Value is India
The index is 1 Value is USA
The index is 2 Value is Germany

Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/IterateStreamWithIndex.java

foreach in java8 using streams

sorting operations in java using streams

Tags iterate, java8, streams

java

convert int to String in java with example

Post author By Rajesh Kumar
Post date February 22, 2020
No Comments on convert int to String in java with example

In this post, we will learn to convert int to String in java with example

Even this is a simple post, But it is difficult to remember all syntax. We can easily convert int to String using valueOf() method

Syntax

String.valueOf(i);

Example

public class ConvertInttoString {

	public static void main(String[] args) {

		int i = 99;
		String stringValue = String.valueOf(i);
		System.out.println("Converted String " + stringValue);
	}

}

Output

Converted String 99

Convert String to lowercase using java

convert String to date in java with example

Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/ConvertInttoString.java

Tags int to string, java, string

collections java

foreach in java8 using streams

Post author By Rajesh Kumar
Post date February 22, 2020
No Comments on foreach in java8 using streams

In this tutorial, we will learn about foreach in java8 using streams

In java8, We can easily iterate a list using streams.

Streams API provided a lot of features. In this post, we are going to see about foreach

Syntax

stream().forEach({operations});

Example


import java.util.ArrayList;
import java.util.List;

public class ForEachExample {

	public static void main(String[] args) {
		try {

			List studentList = new ArrayList();
			Student student = null;

			student = new Student();
			student.setId(1);
			student.setName("Rajesh");
			student.setAddress("Mumbai");
			studentList.add(student);

			student = new Student();
			student.setId(1);
			student.setName("Kumar");
			student.setAddress("Chennai");
			studentList.add(student);

			studentList.stream()
					.forEach(x -> System.out.println("Name is " + x.getName() + " Address is " + x.getAddress()));

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

}

Student.java


public class Student {

	private int id;
	private String name;
	private String address;

	public int getId() {
		return id;
	}

	public void setId(int id) {
		this.id = id;
	}

	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}

	public String getAddress() {
		return address;
	}

	public void setAddress(String address) {
		this.address = address;
	}

}

Output

Name is Rajesh Address is Mumbai
Name is Kumar Address is Chennai

sorting operations in java using streams

Iterate list using streams in java

Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/ForEachExampleUsingStreams.java

Tags arraylist, foreach, java, java8, streams

ansible devops

Ignore Error in Ansible playbook

Post author By Rajesh Kumar
Post date February 21, 2020
No Comments on Ignore Error in Ansible playbook

In this tutorial, we will learn to ignore error in ansible playbook.

While Executing play book if any error occurred in a task. Remaining tasks will get ignored

Some Cases we want to ignore the error. Ansible provide option to ignore error on Exception handling

below is the Syntax to ignore error in ansible playbook

Syntax

ignore_errors: yes

Sample playbook

---

- name: Network Getting Started First Playbook  
  gather_facts: false
  hosts: all
  tasks:

    - name: Make a directory    
      shell: "mkdir sample"       
	- name: Make another directory with same name  
      shell: "mkdir sample" 
      ignore_errors: yes
	- name: Make another directory with another name  
      shell: "mkdir demo" 
      ignore_errors: yes

Output

fatal: [127.0.01]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "mkdir sample", "delta": "0:00:00.008711", "end": "2020-02-21 10:23:37.305132", "msg": "non-zero return code", "rc": 1, "start": "2020-02-21 10:23:37.296421", "stderr": "mkdir: cannot create directory ‘sample’: File exists", "stderr_lines": ["mkdir: cannot create directory ‘sample’: File exists"], "stdout": "", "stdout_lines": []}
...ignoring

In the above example. we are trying to make a duplicate directory which will occur an error. So we used ignore_errors command to ignore the error and to proceed further

we can see the <span class="token punctuation">..</span>.ignoring

Reference

https://docs.ansible.com/ansible/latest/user_guide/playbooks_error_handling.html

ignore warning message in ansible

Tags anisble, ignoreerror, playbook

date java

compare two dates in java example

Post author By Rajesh Kumar
Post date February 20, 2020
No Comments on compare two dates in java example

In this post, we will learn to compare two dates in java example

There are multiple ways to compare two dates in java.

In the First example we will learn to compare dates using compareTo() method

While using compareTo() method we are having 3 chances of output. Below are the possible output

0 – When both the dates are equal

1 – When date 1 is greater than date 2

-1 – When date 1 is lesser than date 2

Syntax

date1.compareTo(date2)

Example


import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

public class DateCompareExample {
	public static void main(String[] args) {
		try {

			String date1 = "22/02/2020";
			String date2 = "21/02/2020";
			DateFormat format = new SimpleDateFormat("DD/MM/yyyy");
			Date dateObj1 = format.parse(date1);
			Date dateObj2 = format.parse(date2);

			// If Both the date are same comparTo Method will return 0
			if (dateObj1.compareTo(dateObj2) == 0) {
				System.out.println("Both the dates are Equal !!");
			} // If date1 greater than date 2 it will return 1
			else if (dateObj1.compareTo(dateObj2) > 0) {

				System.out.println("Date 1 is greater than date 2");
			} // If date1 lesser than date 2 it will return -1
			else {
				System.out.println("Date 1 is lesser than date 2");
			}

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

}

Date 1 is greater than date 2

Using equals(),after(),before()

In the above example, we learned using compareTo() method. But we can compare dates using predefined functions also.

Below example is another way to compare two dates in java

equals() – return true if both the dates are equal

after() – return true if date 1 greater than date 2

before() – return true if date 1 lesser than date 2

Example


import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

public class DateCompareExample {
	public static void main(String[] args) {
		try {

			String date1 = "20/02/2020";
			String date2 = "21/02/2020";
			DateFormat format = new SimpleDateFormat("DD/MM/yyyy");
			Date dateObj1 = format.parse(date1);
			Date dateObj2 = format.parse(date2);

			// If Both the date are same comparTo Method will return 0
			if (dateObj1.equals(dateObj2)) {
				System.out.println("Both the dates are Equal !!");
			} // If date1 greater than date 2 it will return 1
			else if (dateObj1.after(dateObj2)) {

				System.out.println("Date 1 is after than date 2");
			} // If date1 lesser than date 2 it will return -1
			else if (dateObj1.before(dateObj2)) {
				System.out.println("Date 1 is before than date 2");
			}

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

}

Output

Date 1 is before than date 2

Reference

https://docs.oracle.com/javase/8/docs/api/java/util/Date.html#compareTo-java.util.Date-

convert String to date in java with example

Tags compare, date, java

date java

convert String to date in java with example

Post author By Rajesh Kumar
Post date February 20, 2020
No Comments on convert String to date in java with example

In this tutorial, we will learn to convert String to date in java with example.While we working with java this is one of the most scenario

We can easily convert String to Data using SimpleDateFormat.java. But before that we should know the date patterns which mentioned below

Letter	Component	Presentation	Examples
y	Year	Year	1996; 96
M	Month in year	Month	July; Jul; 07
d	Day in month	Number	10
a	Am/pm marker	Text	PM
H	Hour in day (0-23)	Number	0
k	Hour in day (1-24)	Number	24
K	Hour in am/pm (0-11)	Number	0
h	Hour in am/pm (1-12)	Number	12
m	Minute in hour	Number	30
s	Second in minute	Number	55
S	Millisecond	Number	978

In this example we will convert 20/02/2020 to Date class

Syntax

DateFormat simpleDateFormat = new SimpleDateFormat("dd/MM/YYYY");
 simpleDateFormat.parse(dateString);

Example


import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

public class SimpleDateFormatExample {

	public static void main(String[] args) {
		try {
			String dateString = "20/02/2020";
			DateFormat simpleDateFormat = new SimpleDateFormat("dd/MM/YYYY");
			// Below line will convert from string to date
			Date parse = simpleDateFormat.parse(dateString);
			// The output will be like Sun Dec 29 00:00:00 IST 2019
			System.out.println(" Output :: "+parse);
		} catch (ParseException e) {
			e.printStackTrace();
		}

	}

}

Output

 Output :: Sun Dec 29 00:00:00 IST 2019

Reference

https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html

Download

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/SimpleDateFormatExample.java

print current date in java with example

compare two dates in java example

Tags date, java

pyspark

Creating dataframes in pyspark using parallelize

Post author By Rajesh Kumar
Post date February 19, 2020
No Comments on Creating dataframes in pyspark using parallelize

In this Post, We will learn about Creating dataframes in pyspark using parallelize method .

Dataframes are nothing but the tabular structure with rows and columns as similar to the relational database .

Libraries required

Following classes need to be called before executing the program , .

import findspark
findspark.init()
from pyspark import SparkContext,SparkConf
sc=SparkContext.getOrCreate()
from pyspark.sql import Row

Sample program – Creating dataframes using parallelize

Row() – used for creating records

parallelize – used for creating the collection of elements .

toDF() – used for converting the parallelized collection in to a dataframe as seen below .

show() helps us to view the dataframes with the default of 20 rows . We can increase it by specifying the numbers needed like show(40) .

df=sc.parallelize([Row(name='Gokul',Class=10,marks=480,grade='A'),Row(name='Usha',Class=12,marks=450,grade='A'),Row(name='Rajesh',Class=12,marks=430,grade='B')]).toDF()
df.show()

Output

We created the dataframe in our sample program with the name df as seen below.

The dataframes will have column names in the first row and the actual data in all other rows .

+-----+-----+-----+------+ 
|Class|grade|marks| name| 
+-----+-----+-----+------+ 
| 10| A| 480| Gokul| 
| 12| A| 450| Usha|
| 12| B| 430|Rajesh| 
+-----+-----+-----+------+

Reference

https://spark.apache.org/docs/2.1.1/programming-guide.html#parallelized-collections

https://beginnersbug.com/case-when-statement-in-pyspark-with-example/

ansible devops

ignore warning message in ansible

Post author By Rajesh Kumar
Post date February 18, 2020
No Comments on ignore warning message in ansible

In this tutorial, we will learn to ignore the warning message in ansible logs

While working with ansible playbooks, we will get some annoying warning message which makes us uncomfortable.

Below is the useful code snippet to get rid of ansible warning logs

 args:
    warn: false

Example

---

- name: Network Getting Started First Playbook  
  gather_facts: false
  hosts: all
  tasks:

    - name: Download java file from github using curl      
      shell:  curl https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/AddDaysToDate.java --output sample.java  
      args:
        warn: false

Output

PLAY [Network Getting Started First Playbook] ********************************************************************************************************

TASK [Download java file from github using curl] *****************************************************************************************************

PLAY RECAP *******************************************************************************************************************************************
127.0.01                   : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Previous Output

PLAY [Network Getting Started First Playbook] ********************************************************************************************************

TASK [Download java file from github using curl] *****************************************************************************************************
[WARNING]: Consider using the get_url or uri module rather than running 'curl'.  If you need to use command because get_url or uri is insufficient
you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.

PLAY RECAP *******************************************************************************************************************************************
127.0.01                   : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Command

ansible-playbook  first_playbook.yml

Ignore Error in Ansible playbook

Tags ansible, devops

date java

add 3 days to the current date in java

Post author By Rajesh Kumar
Post date February 18, 2020
1 Comment on add 3 days to the current date in java

In this post we will learn to add 3 days to the current date in java
While we working with dates we will come up with adding or subtracting from the current date

In Java there are lot of method for Date. So it is little bit difficult to remember all the things.

In this example we used Calendar.java and Date.java from java.util package
In below example we added 3 days

Syntax

object.add(Calendar.DAY_OF_YEAR, 1)

Example

import java.util.Calendar;
import java.util.Date;

public class AddDaysToDate {

	public static void main(String[] args) {
		try {
			Calendar calender = Calendar.getInstance();
			Date beforeAdding = calender.getTime();
			System.out.println("Printing the value before adding date ::: " + beforeAdding);

			// Adding the 3 days below
			calender.add(Calendar.DAY_OF_YEAR, 3);
			Date afterAdding = calender.getTime();
			System.out.println("Printing the value after adding date  ::: " + afterAdding);

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}

Output

Printing the value before adding date ::: Thu Feb 13 17:00:21 IST 2020
Printing the value after adding date  ::: Sun Feb 16 17:00:21 IST 2020

Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/AddDaysToDate.java

convert String to date in java with example

compare two dates in java example

Tags date, java

Sample program – Single condition check

Output

Sample program – Multiple checks

Output

Sample program with logical operators & and |

Output

Reference

Related Articles

Syntax

Example

Output

Github

Related Articles

Syntax

Example

Output

Related Articles

Github

Syntax

Example

Student.java

Output

Related Article

Github

Syntax

Sample playbook

Output

Reference

Related Articles

Syntax

Example

Using equals(),after(),before()

Example

Output

Reference

Related Articles

Syntax

Example

Output

Reference

Download

Related Articles

Libraries required

Sample program – Creating dataframes using parallelize

Output

Reference

Related Articles

Example

Output

Previous Output

Command

Related Articles

Syntax

Example

Output

Github

Related Articles