Free Interactive Python Tutorial

When you join two DFs with similar column names: df = df1.join(df2, df1['id'] == df2['i… Read more Removing Duplicate Columns After A Df Join In Spark

I have the below code. I am using pyspark 1.2.1 with python 2.7 (cpython) for colname in shuffle_co… Read more Improve Parallelism In Spark Sql

Given a DataFrame with the following rows: rows = [ Row(col1='abc', col2=[8], col3=[18]… Read more Selecting Empty Array Values From A Spark Dataframe

I applied an algorithm from the question below(in NOTE) to transpose and explode nested spark dataf… Read more Spark: How To Transpose And Explode Columns With Nested Arrays

Let's say I have a PySpark data frame like so: 1 0 1 0 0 0 1 1 0 1 0 1 How can I append the la… Read more Pyspark - Append Previous And Next Row To Current Row

I have a spark dataframe (prof_student_df) that lists student/professor pair for a timestamp. There… Read more Implementing A Recursive Algorithm In Pyspark To Find Pairings Within A Dataframe

Suppose I had a function to generate a (py)spark data frame, caching the data frame into memory as … Read more If I Cache A Spark Dataframe And Then Overwrite The Reference, Will The Original Data Frame Still Be Cached?

I have a DataFrame: +-----+--------+---------+ | usn|log_type|item_code| +-----+--------+--------… Read more How To Make An Integer Index Row?

I am trying to measure the performance impact on having to copy a dataframe from scala to python an… Read more Pyspark. Transformer That Generates A Random Number Generates Always The Same Number

I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column. I've tried t… Read more How Do I Add A New Column To A Spark Dataframe (using Pyspark)?

I'm stuck in a process where I need to perform some action for each column value in my Datafram… Read more Pyspark Best Alternative For Using Spark Sql/df Withing A Udf?

I am writing a function which takes a RDD as input splits the comma separated values then convert… Read more Error In Labelled Point Object Pyspark

I have a script with the below setup. I am using: 1) Spark dataframes to pull data in 2) Converting… Read more Convert A Pandas Dataframe To A Pyspark Dataframe

I am having 100,000+ of records in dataframe. I want to create a file dynamically and push 1000 rec… Read more How To Get 1000 Records From Dataframe And Write Into A File Using Pyspark?

I have three columns in a pyspark data frame ( sample data given below ) I wanted to get the remov… Read more Get 20th To 80th Percentile Of Each Group - Pyspark

I have a query about loading the schema onto cdsw using pyspark. I have a dataframe which is create… Read more Uploading Custom Schema From A Csv File Using Pyspark

Let's say I have a numpy array a that contains the numbers 1-10: [1 2 3 4 5 6 7 8 9 10] I also … Read more Passing Array To Spark Lit Function

I want to be able to use a Scala function as a UDF in PySpark package com.test object ScalaPySpark… Read more How To Use Scala Udf In Pyspark?

I applied an algorithm from the question Spark: How to transpose and explode columns with nested ar… Read more Spark: How To Transpose And Explode Columns With Dynamic Nested Arrays

How can I read the following JSON structure to spark dataframe using PySpark? My JSON structure {&#… Read more Read Json File As Pyspark Dataframe Using Pyspark?