Categories
pyspark

How to use filter condition in pyspark

Filter condition in pyspark

In this post, we will learn how to use filter condition in pyspark with example.

Sample program using filter condition

We will create a dataframe using the following sample program.

Then we filter the dataframe based on marks and store the result in another dataframe.

The following classes imported at the beginning of the code.

import findspark 
findspark.init() 
from pyspark import SparkContext,SparkConf 
from pyspark.sql import Row 
from pyspark.sql.functions import *
sc=SparkContext.getOrCreate()
#creating dataframe with three records
df=sc.parallelize([Row(name='Gokul',Class=10,marks=480,grade='A'),Row(name='Usha',Class=12,marks=450,grade='A'),Row(name='Rajesh',Class=12,marks=430,grade='B')]).toDF() 
print("Printing df dataframe below ")
df.show() 
#Filtering based on the marks
df1=df.filter(col("marks")==480)
print("Printing df1 dataframe below")
df1.show()
Output

The following dataframes created as the result of the above sample program.

Here this filter condition helps us to filter the records having marks as 480 from the dataframe.

Printing df dataframe below 
+-----+-----+-----+------+ 
|Class|grade|marks| name| 
+-----+-----+-----+------+ 
| 10| A| 480| Gokul| 
| 12| A| 450| Usha| 
| 12| B| 430|Rajesh| 
+-----+-----+-----+------+
 Printing df1 dataframe below
 +-----+-----+-----+-----+ 
|Class|grade|marks| name| 
+-----+-----+-----+-----+ 
| 10| A| 480|Gokul| 
+-----+-----+-----+-----+

Must use double equal to inside the filter condition.

I hope that everyone got an idea about how to use filter condition in pyspark now.

Reference

http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=filter#pyspark.sql.DataFrame

https://beginnersbug.com/where-condition-in-pyspark-with-example/

Categories
Interview java matrix

multiply two matrix using java with example

multiply two matrix using java with example

In this post, we will learn about multiply two matrix in java with example

We are declaring two matrix named as a,b and one more variable mul to store the multiplication of a&b matrix

Example
public class MatrixMultiplication {

	public static void main(String[] args) {
		try {

			int a[][] = { { 50, 60 }, { 20, 15 } };
			int b[][] = { { 25, 20 }, { 10, 5 } };

			// Declaring the diff matrix
			int[][] mul = new int[a.length][b.length];

			// multiply 2 matrices
			for (int i = 0; i < a.length; i++) {
				for (int j = 0; j < b.length; j++) {
					mul[i][j] = 0;
					for (int k = 0; k < 2; k++) {
						mul[i][j] += a[i][k] * b[k][j];
					}
					System.out.print(mul[i][j] + " ");
				}
				System.out.println();
			}

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Output
1850  1300 
650   475
Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/MatrixMultiplication.java

Related Articles

subtracting two matrix in java with example

Categories
Interview java matrix

subtracting two matrix in java with example

subtracting two matrix in java with example

In this post, we will learn about subtracting two matrix in java with example

We are declaring two matrix named as a,b and one more variable diff to store the subtraction of a&b matrix

Example
public class SubtractingMatrix {

	public static void main(String[] args) {
		try {

			int a[][] = { { 50, 60 }, { 20, 15 } };
			int b[][] = { { 25, 20 }, { 10, 5 } };

			// Declaring the diff matrix
			int[][] diff = new int[a.length][b.length];

			// Subtracting two matrix
			for (int i = 0; i < a.length; i++) {

				for (int j = 0; j < b.length; j++) {
					diff[i][j] = a[i][j] - b[i][j];
				}
			}

			// Printing the diff matrix
			for (int i = 0; i < diff.length; i++) {
				for (int j = 0; j < diff[i].length; j++) {
					System.out.print(diff[i][j] + " ");
				}
				System.out.println();
			}

		} catch (Exception e) {
			e.printStackTrace();
		}

	}
}
Output
25 40 
10 10
Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/SubtractingMatrix.java

Related Articles

adding two matrix in java with example

Categories
Interview java matrix

adding two matrix in java with example

adding two matrix in java with example

In this post, we will learn about adding two matrix in java with example

In below example,We are using two dimensional array for this operations.

Here we are declaring two matrix named as a,b and one more variable sum to store the addition of a&b matrix

The first for loop is used to add two matrix and the second for loop used to print the addition value

Example
public class AddingMatrix {

	public static void main(String[] args) {
		try {

			int a[][] = { { 54, 67 }, { 45, 56 } };
			int b[][] = { { 25, 56 }, { 85, 96 } };

			// Declaring the sum matrix
			int[][] sum = new int[a.length][b.length];

			// Adding two matrix
			for (int i = 0; i < a.length; i++) {

				for (int j = 0; j < b.length; j++) {
					sum[i][j] = a[i][j] + b[i][j];
				}
			}

			// Printing the sum matrix
			for (int i = 0; i < sum.length; i++) {
				for (int j = 0; j < sum[i].length; j++) {
					System.out.print(sum[i][j] + " ");
				}
				System.out.println();
			}

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

}
Output
79 123 
130 152
Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/AddingMatrix.java

Related Articles

create 2×2 matrix using java

Categories
java matrix

create 2×2 matrix using java

create 2x2 matrix using java

In this post, we will learn to create 2×2 matrix using java

We all know that In mathematics Matrix is a two dimensional array with rows and column as like above image. Here we are going to create that matrix using java

Java have a feature to create two dimensional array as like below

Two dimensional array Syntax
			int a[][] = { { 1, 2 }, { 2, 4 } };

The above array will convert as like below image

Example
public class CreateMatrixExample {
	public static void main(String[] args) {
		try {

			// Creating 2x2 matrix
			int a[][] = { { 1, 2 }, { 2, 4 } };

			// printing above 2d array in matrix format
			for (int i = 0; i < a.length; i++) {
				for (int j = 0; j < a[i].length; j++) {
					System.out.print(a[i][j] + " ");
				}
				System.out.println();
			}

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Output
1 2 
2 4
Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/CreateMatrixExample.java

Categories
ansible devops

when condition in ansible playbook

ansible playbook example

In this tutorial, we will learn about when condition in ansible playbook

We don’t have If condition here. Instead of that we can use when condition

Syntax
when: {condition}
when: inventory_hostname == "local"

In below example we are using when condition based on host

Example
---
- hosts: 127.0.0.1
  tasks:
  - name:  Make a directory  
    shell: "mkdir sample"
    when: inventory_hostname == "127.0.0.1"
    
  - name: make directory
    shell: "mkdir sample"
    when: inventory_hostname == "10.118.226.25" or inventory_hostname == "10.118.225.65"
Execute Command
ansible-playbook whenexample.yml
Output
PLAY [127.0.0.1] *************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************
ok: [127.0.0.1]

TASK [Make a directory] ******************************************************************************************************************************
[WARNING]: Consider using the file module with state=directory rather than running 'mkdir'.  If you need to use command because file is insufficient
you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.

changed: [127.0.0.1]

TASK [make directory] ********************************************************************************************************************************
skipping: [127.0.0.1]

PLAY RECAP *******************************************************************************************************************************************
127.0.0.1                  : ok=2    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

when condition matches you can see the changed: [127.0.0.1]in result.
Else it will skip the task skipping: [127.0.0.1] in result

Reference

Below is Ansible documentation for conditions https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html

Related Articles

ignore warning message in ansible

Ignore Error in Ansible playbook

Categories
file java

delete a file from java code

while loop in java

In this post, we will learn to delete a file from java code.

When you are delete a file from java. It will delete the file permanently. It will not moved to Trash or Recycle bin

Before executing this code, please make sure you have delete permission for that file from java.
Else you will get Permission denied Exception

Syntax
boolean isDeleted = file.delete();
Example
import java.io.File;

public class DeleteFileExample {

	public static void main(String[] args) {
		try {
			File file = new File("C:\\sample.txt");
			// Below if condition will check that file exists or not
			if (file.exists()) {
				boolean isDeleted = file.delete();
				if (isDeleted == true) {
					System.out.println("File deleted Successfully");
				} else {
					System.out.println("Not able to delete the file");
				}
			} else {
				System.err.println("File path is not correct");
			}

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Output
File deleted Successfully
File delete using nio pacakage

In Above example, we can customize our exceptions like file not found, file delete staus.

Below example is quick way of deleting file, But we cannot customize out exception

Syntax 
	Files.deleteIfExists(Paths.get("C://temp//sample.txt"));
Example
import java.nio.file.Files;
import java.nio.file.Paths;

public class DeleteFileExample {

	public static void main(String[] args) {
		try {
			// Below line will delete if the file exist
			Files.deleteIfExists(Paths.get("C://temp//sample.txt"));
			System.out.println("File deleted successfully");
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Output
File deleted successfully
Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/files/DeleteFileExample.java

Related Articles

how to list all files inside a folder using java

how to read a file from java with example

Categories
java Linux

ssh to linux box using Kerberos from Java

get the last modified date of a file in Java

In this post, we will learn about ssh to linux box using Kerberos from Java

If you want to login to VM without using password from Java we can use this Kerberos for this.

Prerequisites
  • JSch.jar
  • Kerberos Configuration file
  • Jaas file
  • Remote Host information
  • Keytab file location
  • Principle Name

In the code we need to set the below system property

  • System.setProperty(“java.security.krb5.conf”, “/etc/krb5.config”);
  • System.setProperty(“java.security.auth.login.config”, “jass.config”); — Jass file

java.security.krb5.conf – you can see the krb5.config file on /etc directory
java.security.auth.login.config- jass.config will look like below

com.sun.security.jgss.krb5.accept {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        doNotPrompt=true
        principal="[email protected]"
		keyTab="location of keytab"
		
    };
Example

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Properties;

import com.jcraft.jsch.Channel;
import com.jcraft.jsch.ChannelExec;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;

public class SSHKerberosExample {

	public static void main() {

		// Remote host that we are going to connect
		String host = "10.118.224.56";
		// Remote host user
		String user = "aws-test-user";
		// Command that we are going to execute on remote host
		String command = "mkdir /home/kerberosexample";

		JSch jsch = new JSch();
		// Kerberos config file
		System.setProperty("java.security.krb5.conf", "/etc/krb5.config");
		// Jass file location
		System.setProperty("java.security.auth.login.config", "/home/jaas.config");
		System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
		System.setProperty("sun.security.krb5.debug", "true");

		try {

			Session session = jsch.getSession(user, host, 22);
			Properties config = new java.util.Properties();
			config.put("StrictHostKeyChecking", "no");
			config.put("PreferredAuthentications", "gssapi-with-mic");

			session.setConfig(config);
			session.connect(20000);

			Channel channel = session.openChannel("exec");

			// Creating BufferReader to read input from host
			BufferedReader bufferReader = new BufferedReader(new InputStreamReader(channel.getInputStream()));

			((ChannelExec) channel).setCommand(command);
			channel.connect();

			StringBuilder stringBuilder = new StringBuilder();
			for (String string = bufferReader.readLine(); string != null; string = bufferReader.readLine()) {
				stringBuilder.append(string);
			}

			System.out.println(stringBuilder.toString());

			channel.disconnect();
			session.disconnect();
			System.out.println("created folder using kerberos authentication");

		} catch (JSchException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}
	}

}
Output
created folder using kerberos authentication

In case if you didn’t configured the properties correctly you will face auth exception

Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/SSHKerberosExample.java

Categories
java

Base64 Encoding using Java

get the last modified date of a file in Java

In Below example we will learn how to use Base64 Encoding using Java

Finally Java8 incorporated Base64 Encoding. Base64 is one of the common Encoding mechanism.

Syntax
Base64.getEncoder().encodeToString(textForEncoding.getBytes());
Example

import java.util.Base64;

public class Base64Encodng {

	public static void main(String[] args) {
		try {
			String textForEncoding = "QWERTY TEXT";
			String encodedString = Base64.getEncoder().encodeToString(textForEncoding.getBytes());
			System.out.println(encodedString);

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Output
UVdFUlRZIFRFWFQ=

The length of output encoded String must be a multiple of 3. else the output will be padded with additional pad characters (`=`).
But we have option to encode without padding also

Syntax
Base64.getEncoder().withoutPadding().encodeToString(textForEncoding.getBytes());
Example without Padding

import java.util.Base64;

public class Base64Encodng {

	public static void main(String[] args) {
		try {
			String textForEncoding = "QWERTY TEXT";
			String encodedString = Base64.getEncoder().withoutPadding().encodeToString(textForEncoding.getBytes());
			System.out.println(encodedString);		

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Output
UVdFUlRZIFRFWFQ
Decoding Syntax
Base64.getDecoder().decode(encodedString.getBytes());
Example

import java.util.Base64;

public class Base64Encodng {

	public static void main(String[] args) {
		try {
			String textForEncoding = "QWERTY TEXT";
			String encodedString = Base64.getEncoder().withoutPadding().encodeToString(textForEncoding.getBytes());
			System.out.println(encodedString);
			
			byte[] decodeByte = Base64.getDecoder().decode(encodedString.getBytes());
			String decodedValue=new String(decodeByte);
			System.out.println(decodedValue);

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}
Output
UVdFUlRZIFRFWFQ
QWERTY TEXT
Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/Base64Encodng.java

Categories
file java

how to list all files inside a folder using java

while loop in java

In this post, we will how to list all files inside a folder using java. Here we will use File.java for these operations

In below example we are going to use .list method to list all files and directories without trans versing into sub directories.

The file path I am going to use is F:\Java which having file like below image 

Example
import java.io.File;

public class ListAllFilesExample {

	public static void main(String[] args) {
		try {

			File file = new File("F:\\Java");
			String[] list = file.list();
			for (String fileName : list) {
				System.out.println(fileName);
			}

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

}

Output

access.txt
apache-tomcat-8.5.35
eclipse
Maven
sample.txt
sts-bundle
Transverse into sub folders using Java 8

Using java8 we can easily transverse into all sub folders in side the folder. In below Example we are using Files.walk to get all the files and folders by trans versing inside the folders

import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.stream.Stream;

public class ListAllFilesUsingJava8 {

	public static void main(String[] args) {
		try {
			Path path = Paths.get("F:/Java");
			Stream<Path> walk = Files.walk(path);
			walk.forEach(x -> System.out.println(x.getFileName()));

		} catch (Exception e) {
			e.printStackTrace();
		}

	}

}
Output
org.eclipse.datatools.enablement.hsqldb.ui_1.2.1.201712071719.jar
org.eclipse.datatools.enablement.hsqldb_1.2.1.201712071719.jar
org.eclipse.datatools.enablement.ibm.db2.iseries.dbdefinition_1.2.1.201712071719.jar
Github

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/files/ListAllFilesExample.java

https://github.com/rkumar9090/BeginnersBug/blob/master/BegineersBug/src/com/geeks/example/files/ListAllFilesUsingJava8.java

Related Articles

how to read a file from java with example

create text file from java with example