- The master branch is the prototype. For more details, please reference the azure_hdinsight branch.
- Use the following command to search the frequently occurring sentences(the generic options like -files and -D show be placed before the command options):
hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.8.0.jar \
-files /home/rav009/PycharmProjects/untitled/PhraseExtract/sentence_mapper.py,/home/rav009/PycharmProjects/untitled/PhraseExtract/sentence_reducer.py \
-D mapred.map.tasks=7 \
-D mapred.reduce.tasks=3 \
-input /input/text.txt \
-output /sentences/above100/ \
-mapper "python sentence_mapper.py" \
-reducer "python sentence_reducer.py -t 100"
python sentence_reducer.py -t 100 stands for output all the sentence appears for more than 100 times.
- Use the following command to search the frequently occurring phrases which contains 2 or 3 words:
hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.8.0.jar \
-files /home/rav009/PycharmProjects/untitled/PhraseExtract/phrase_mapper.py,/home/rav009/PycharmProjects/untitled/PhraseExtract/phrase_reducer.py,hdfs://127.0.0.1:9000/sentences/above100/part-00000 \
-D mapred.map.tasks=4 \
-D mapred.reduce.tasks=4 \
-D mapred.text.key.partitioner.options=-k1 \
-input hdfs://namenode/input.txt \
-output /phrase/above2000 \
-mapper "python phrase_mapper.py -l 3" \
-reducer "python phrase_reducer.py -t 2000 -c" \
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner
python phrase_mapper.py -l 3 stands for generate the phrases contain less than or equal to 3 words.
python phrase_reducer.py -t 2000 -c stands the threshold of frequency of phrase is 2000 and also output the ID number of each passage(assume the ID and the content is split by '|').
- The zip file is the Kettle ETL project and the SSAS project.