Skip to main content

Posts

RDD commands

  Category Command Description / What It Does Example Creation sc.parallelize(data) Create RDD from a Python list rdd = sc.parallelize([1,2,3]) sc.textFile(path) Read text file (each line = 1 element) rdd = sc.textFile("data.txt") sc.wholeTextFiles(path) Read files as (filename, content) pairs rdd = sc.wholeTextFiles("folder/") Basic Transformations map(func) Apply function to each element rdd.map(lambda x: x*2) flatMap(func) Apply function & flatten result rdd.flatMap(lambda x: x.split(" ")) filter(func) Keep elements matching condition rdd.filter(lambda x: x>10) distinct() Remove duplicate elements rdd.distinct() union(rdd2) Combine two RDDs rdd1.union(rdd2) intersection(rdd2) Keep common elements rdd1.intersection(rdd2) subtract(rdd2) Elements in first RDD but not second rdd1.subtract(rdd2) cartesian(rdd2) All pairs between two RDDs rdd1.cartesian(rdd2) sample(withRepla...
Post a Comment
Read more
Recent posts

Map Reduce

 /*******************************************************************************************  * BIG DATA LAB – 9  * Name  : 21MIS1029  * Topic : MapReduce Programs – Join Operations & Aggregations  *******************************************************************************************/ /*******************************************************************************************  * (i) MAP SIDE JOIN  * ------------------------------------------------------------------------------------------  * Q1. Develop a Map Reduce Program to display the details of employees using two text files  *     (sample code is given) using Map Side Join.  *  * Input:  *   empdetails.txt → (emp_id, emp_name)  *   empdata.txt    → (emp_id, salary, dept_id)  *  * Output:  *   emp_id, salary, dept_id, emp_name  ****************************************************...
Post a Comment
Read more

Map Side Join

package my.org; import java.io.*; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.*; import org.apache.hadoop.filecache.DistributedCache;   // Deprecated in newer versions, but fine for Hadoop 1.x import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.Mapper.Context; import org.apache.hadoop.mapreduce.lib.input.*; import org.apache.hadoop.mapreduce.lib.output.*; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; public class mapsid1036 {     // ---------------- Mapper Class ----------------     public static class Map extends Mapper<LongWritable, Text, Text, Text> {         private ArrayList<String> employees = new ArrayList<String>();         // Setup method loads data from Distributed Cache         @Override         public void setup(Con...
Post a Comment
Read more

Reducer Side Join

package my.org; import java.io.IOException; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.MultipleInputs; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class redjoin1036 {     // -------------------- CUSTOMER MAPPER --------------------     public static class CustMapper extends Mapper<LongWritable, Text, Text, Text> {         public void map(LongWritable key, Text value, Context context)                 throws IOException, InterruptedException {             String[] line = value.toString().split(",");   ...
Post a Comment
Read more

The Final: gameplay detail, system requirements, release date, early access.

The Final is the upcoming AAA free to play game, and here is all you need to know about the game.  The final is a new competitive FPS shooter game with destructive environment. Here is everything about the release date, gameplay details, system requirements and how you can get closed beta access.  The game is made and published by Embark studio which is there first game as a studio by the developers are ex dice studio employees and have worked in battlefield . So you can definitely expect a high refined game .  On the first look it looks like a mixture of gta online and csgo with a lot of caos and destruction. The game graphics looks absolutely stunning like upgraded gta 5. The game is not a battle Royal or a arena shooter. In the game you most probably have to complete objective by stealing coin. The game main theme is still not fully revealed.  The Final game Release date.    The game will most probably be out in September or Novem...
Post a Comment
Read more

Valorant new agent gekko ability explained

So there are few images of gekko has been leaked from facebook which shows us all his ability and you won't believe how amazing and over powered they are and specially his ultimate.  By the way if you don’t know gekko is a initiator. And can control animal like creature. In his first ability we can clearly see him controlling a creature while closing his eyes with his hand.   This ability would be used to scout a area like sova drone or syeke’s dog and will also stun enemies near to it. And it will also be faster than the Skye’s dog basically its Syke’s dog not much difference. This second ability would be like a slimy grenade which would slow down enemies like sage slow orb. There is also a probability that this will also reveal enemies because why not if its a initiator. So now this ability looks interesting . It looks like this creature could defuse and plant t...
Post a Comment
Read more

XDefiant the next free to play game like COD

XDefiant is a first-person shooter game that is currently in development by Ubisoft.  The game is set to be a free-to-play, multiplayer shooter that features various factions, each with their unique abilities and playstyles. The game is set in a world where factions of rebels, outcasts, and other misfits come together to battle it out in various game modes. The game is being developed by Ubisoft San Francisco and is expected to be released on PC, consoles, and cloud gaming platforms. XDefiant will feature a range of different game modes, including Domination, Escort, and Arena. It will also include a robust character customization system, allowing players to create their characters and personalize their playstyles. Ubisoft has not announced an official release date for XDefiant yet, but they have stated that the game will be available soon. Interested players can sign up for the game's beta on the official XDefiant website.    Even though currently no release date is ...
Post a Comment
Read more