MapReduce编程.pptx

MapReduce编程目标12 掌握Mapreduce编程技巧目录 Hello WorldAPI介绍和编程详解 案例与高级编程123MapReduce剖析图Hello WorldWordCount v2.0? Hadoop原生实现“单词统计”实现类org.apache.hadoop.examples.WordCount? 执行命令hadoop jar hadoop-examples-1.1.2.jarwordcount in out? 构造数据文件 file1：Hello World Bye World文件 file2：Hello Hadoop Goodbye HadoopWordCount v2.0? 加载数据hadoop fs -put file1 /input/wordcount/hadoop fs -put file2 /input/wordcount/? 执行hadoop jar hadoop-examples-1.1.2.jar wordcount /input/wordcount/ /output? 结果查看hadoop fs -ls /outputhadoop fs -cat /output/part-r-00000WordCount v2.0? 内部如何实现？? 如何编写MR程序？? 代码什么流程？? 下面的wordcount v2代码讲解内容只做参考，详细参见课堂实际代码示例讲解 WordCount v2.0public class WordCount { public static void main(String[] args) throws Exception{ Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length != 2) { System.err.println(Usage: wordcount in out); System.exit(2);}Job job = new Job(conf, word count);job.setJarByClass(WordCount.class);job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);}} Mapper函数类class TokenizerMapper extends MapperObject, Text, Text, IntWritable{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()){ word.set(itr.nextToken()); context.write(word, one);}}} Reducer函数类class IntSumReducer extends ReducerText,IntWritable,Text,IntWritable{ private IntWritable result = new IntWritable(); public void reduce(Text key, IterableIntWritable values, Context context) throws IOException, Interr