DEV Community: Thomas Jaskiewicz The latest articles on DEV Community by Thomas Jaskiewicz (@tjay_dev). https://dev.to/tjay_dev https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F161888%2Fec9c27ab-4d6f-4683-9e6b-f8250ae0c605.png DEV Community: Thomas Jaskiewicz https://dev.to/tjay_dev en Question - Should I write about it? Thomas Jaskiewicz Wed, 27 Jan 2021 21:21:33 +0000 https://dev.to/tjay_dev/question-should-i-write-about-it-253k https://dev.to/tjay_dev/question-should-i-write-about-it-253k <p>Hi!</p> <p>I’ve recently decided to quit google and replace google’s services with different services to secure own my data + avoid privacy issues...</p> <p>If you wonder what’s “degoogle” about, see this article: <a href="proxy.php?url=https://www.reddit.com/r/degoogle/comments/huk4rp/why_you_should_degoogle_intro_degoogling/">here</a></p> <p>I was thinking about writing a series of articles about deploying some self-hosted solution to your private servers/NAS/VPS. All in Docker, I could use Terraform to spin up some VPS. Some reverse-proxy, securing the setup, using SSL, etc.</p> <p>Some initial ideas for articles:</p> <ol> <li>Building a simple Infrastructure as a Code using Terraform to spin up our server.</li> <li>Deploying Ghost as a blog engine.</li> <li>Adding Comments engine and plugging it to the blog.</li> <li>Adding Analytics solution and plugging it to the blog.</li> <li>Deploying some cloud storage solution for storing files.</li> <li>Separate article about reverse proxy, securing the setup, using SSL certs, hardening the server setup.</li> <li>Other services that will be needed to replace google with something.</li> <li>If you have more ideas, just let me know.</li> </ol> <p>Would you find such articles useful? <br> Maybe it would be a great opportunity for you to learn something new, like #terraform, #docker, #reverse-proxy, #iaac? :)</p> <p>Let me know your thoughts!</p> <p>Cheers,<br> Thomas</p> discuss docker selfhosted degoogle Looking for a new terminal setup? I have something for you! ZSH + Starship + Hyper! Thomas Jaskiewicz Tue, 07 Jul 2020 18:57:30 +0000 https://dev.to/tjay_dev/looking-for-a-new-terminal-setup-i-have-something-for-you-zsh-starship-hyper-20gb https://dev.to/tjay_dev/looking-for-a-new-terminal-setup-i-have-something-for-you-zsh-starship-hyper-20gb <p>Hi, Devs!</p> <p>Many of you asked me to <em>revise &amp; update</em> my article about setting up the terminal with <strong>Hyper.js, ZSH and Starship</strong>. </p> <p>I'm happy to say that it has finally arrived. Please, use it, share it and comment! I'm also curious about your tips when it comes to the terminal environment.</p> <h3> Revised and updated article can be found here: </h3> <div class="ltag__link"> <a href="proxy.php?url=/netguru" class="ltag__link__link"> <div class="ltag__link__org__pic"> <img src="proxy.php?url=https://res.cloudinary.com/practicaldev/image/fetch/s--D-2qHueA--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://res.cloudinary.com/practicaldev/image/fetch/s--Ogu8qq9m--/c_fill%2Cf_auto%2Cfl_progressive%2Ch_150%2Cq_auto%2Cw_150/https://dev-to-uploads.s3.amazonaws.com/uploads/organization/profile_image/1008/f941ae53-1c67-45e1-8c6b-a1536380f8af.png" alt="Netguru" width="150" height="150"> <div class="ltag__link__user__pic"> <img src="proxy.php?url=https://res.cloudinary.com/practicaldev/image/fetch/s--qwKKkoO_--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://res.cloudinary.com/practicaldev/image/fetch/s--JTl8gumj--/c_fill%2Cf_auto%2Cfl_progressive%2Ch_150%2Cq_auto%2Cw_150/https://dev-to-uploads.s3.amazonaws.com/uploads/user/profile_image/161888/ec9c27ab-4d6f-4683-9e6b-f8250ae0c605.png" alt="" width="150" height="150"> </div> </div> </a> <a href="proxy.php?url=/netguru/howto-my-terminal-shell-setup-hyper-js-zsh-starship-2j2k" class="ltag__link__link"> <div class="ltag__link__content"> <h2>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥</h2> <h3>Thomas Jaskiewicz for Netguru ・ Sep 7 '19 ・ 20 min read</h3> <div class="ltag__link__taglist"> <span class="ltag__link__tag">#terminal</span> <span class="ltag__link__tag">#shell</span> <span class="ltag__link__tag">#productivity</span> <span class="ltag__link__tag">#tutorial</span> </div> </div> </a> </div> <p>Cheers! <br> Have a productive day :)</p> <p><span>Photo by <a href="proxy.php?url=https://unsplash.com/@tracycodes?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tracy Adams</a> on <a href="proxy.php?url=https://unsplash.com/s/photos/computer-terminal?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></span></p> tutorial zsh starship terminal HowTo: My Terminal & Shell setup - Hyper.js + ZSH + starship ☄🔥 Thomas Jaskiewicz Sat, 07 Sep 2019 22:10:00 +0000 https://dev.to/netguru/howto-my-terminal-shell-setup-hyper-js-zsh-starship-2j2k https://dev.to/netguru/howto-my-terminal-shell-setup-hyper-js-zsh-starship-2j2k <p><strong>[Revised at 2020-07-07]</strong></p> <p>Probably, the first thing that developers do after getting a new computer is… terminal setup. I wanted to share with you my current development setup - here, terminal setup.</p> <p>Talk is cheap, let’s show some code.</p> <h2> 1. Terminal - Hyper.js </h2> <h3> 1.1 Download and install Hyper. </h3> <p>Let’s go to <a href="proxy.php?url=https://hyper.is/#installation" rel="noopener noreferrer">Hyper™</a> page and download a version for your OS.<br><br> I downloaded a Mac OS version. When we extract zip file <code>Hyper-3.0.2-mac.zip</code> we get a <code>Hyper.app</code> file. To install it just drag it to <code>Applications</code> folder. Now, by using the <em>Spotlight Search</em> (<code>CMD</code> + <code>SPACE</code>) type <code>hyper</code> and there should be a Hyper application on the list.</p> <p>The Hyper’s window looks like this in my case:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-5.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-5.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <p>Don’t worry if your prompt looks differently. We are going to take care of it later. You might be using a different shell, different config at the beginning.</p> <h3> 1.2 Basic configuration </h3> <p>Now, the magic happens. Hyper’s config is defined as <code>~/.hyper.js</code> file. File is structured like this:</p> <div class="highlight js-code-highlight"> <pre class="highlight javascript"><code> <span class="c1">// Future versions of Hyper may add additional config options,</span> <span class="c1">// which will not automatically be merged into this file.</span> <span class="c1">// See https://hyper.is#cfg for all currently supported options.</span> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="na">config</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// choose either `'stable'` for receiving highly polished,</span> <span class="c1">// or `'canary'` for less polished but more frequent updates</span> <span class="na">updateChannel</span><span class="p">:</span> <span class="dl">'</span><span class="s1">stable</span><span class="dl">'</span><span class="p">,</span> <span class="c1">// default font size in pixels for all tabs</span> <span class="na">fontSize</span><span class="p">:</span> <span class="mi">12</span><span class="p">,</span> <span class="c1">// font family with optional fallbacks</span> <span class="na">fontFamily</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Menlo, "DejaVu Sans Mono", Consolas, "Lucida Console", monospace</span><span class="dl">'</span><span class="p">,</span> <span class="c1">// default font weight: 'normal' or 'bold'</span> <span class="na">fontWeight</span><span class="p">:</span> <span class="dl">'</span><span class="s1">normal</span><span class="dl">'</span><span class="p">,</span> <span class="c1">// rest of the config</span> <span class="p">}</span> <span class="c1">// rest of the file</span> <span class="p">}</span> </code></pre> </div> <p>This is awesome information! We have complete control over our settings.<br><br> Let’ start with basic config.</p> <h4> 1.2.1 Font </h4> <ul> <li> <code>fontSize</code> - personally, I prefer a little bigger font, let’s use 13. Later on you can change the font size in the terminal by using <code>CMD</code> + <code>+</code> and <code>CMD</code> + <code>-</code> keyboard shortcuts.</li> <li> <code>fontFamily</code> - let’s use a really nice font with ligatures - <a href="proxy.php?url=https://github.com/tonsky/FiraCode" rel="noopener noreferrer">FiraCode</a> . I attach an overview below:</li> </ul> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-6.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-6.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <p>Installing instructions are described <a href="proxy.php?url=https://github.com/tonsky/FiraCode/wiki#installing-font" rel="noopener noreferrer">here</a>. I prefer the manual way (for Mac OS):</p> <ul> <li>Download the latest version of the font from Github <code>releases</code> tab: <a href="proxy.php?url=https://github.com/tonsky/FiraCode/releases" rel="noopener noreferrer">Releases · tonsky/FiraCode · GitHub</a> - As for now, it’s <a href="proxy.php?url=https://github.com/tonsky/FiraCode/releases/download/5.2/Fira_Code_v5.2.zip" rel="noopener noreferrer">Fira_Code_v5.2.zip</a> .</li> <li>Extract the archive and go to <code>ttf</code> directory.</li> <li>Select all fonts <code>.ttf</code> files , click right , then <code>Open</code> and select <code>Install Font</code> </li> </ul> <p>Now, when our font is installed let’s set it in our Hyper terminal. The config file should be placed in the home directory:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> nano ~/.hyper.js </code></pre> </div> <div class="highlight js-code-highlight"> <pre class="highlight javascript"><code> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="na">config</span><span class="p">:</span> <span class="p">{</span> <span class="na">fontSize</span><span class="p">:</span> <span class="mi">13</span><span class="p">,</span> <span class="na">fontFamily</span><span class="p">:</span> <span class="dl">'</span><span class="s1">"Fira Code", Menlo, "DejaVu Sans Mono", Consolas, "Lucida Console", monospace</span><span class="dl">'</span><span class="p">,</span> <span class="c1">// rest of the config</span> <span class="p">}</span> <span class="c1">// rest of the file</span> <span class="p">}</span> </code></pre> </div> <p>At this point we enabled <code>Fira Code</code> font in the terminal but ligatures are still not enabled. Let’s turn them on.<br><br> We need to install a dedicated plugin: <code>hyper-font-ligatures</code>:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hyper-font-ligatures </code></pre> </div> <p>Plugin should immediately appear in our config file:</p> <div class="highlight js-code-highlight"> <pre class="highlight javascript"><code> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="c1">// rest of the file</span> <span class="na">plugins</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">hyper-font-ligatures</span><span class="dl">"</span><span class="p">],</span> </code></pre> </div> <p>To enable ligatures we need to add one more line to the config (See <a href="proxy.php?url=https://github.com/tolbertam/hyper-font-ligatures/issues/8" rel="noopener noreferrer">issue #8</a> and <a href="proxy.php?url=https://github.com/zeit/hyper/issues/3607" rel="noopener noreferrer">zeit/hyper#3607</a> for more details):</p> <div class="highlight js-code-highlight"> <pre class="highlight javascript"><code> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="na">config</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// rest of the config</span> <span class="na">webGLRenderer</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span> <span class="p">}</span> <span class="c1">// rest of the file</span> <span class="p">}</span> </code></pre> </div> <p>Now, everything should work as expected and we should see new font with ligatures in the terminal:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-7.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-7.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <h4> 1.2.2 Theme </h4> <p>With Hyper we can download multiple themes that are available here: <a href="proxy.php?url=https://hyper.is/themes?newest" rel="noopener noreferrer">Hyper Store - Plugins</a>. I’m going to use <a href="proxy.php?url=https://hyper.is/plugins/hyper-one-dark" rel="noopener noreferrer">Hyper Store - hyper-one-dark</a> theme. To install it, just run:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hyper-one-dark </code></pre> </div> <p>With my current shell it looks like this:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-8.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-8.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <h4> 1.2.3 Search Plugin </h4> <p>To be able to search for a keyword in the terminal we need to install a dedicated plugin: <a href="proxy.php?url=https://hyper.is/plugins/hyper-search" rel="noopener noreferrer">Hyper Store - hyper-search</a></p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hyper-search </code></pre> </div> <p>This time we need to restart the Hyper completely. After the restart we can start using our plugin:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-9.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-9.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <h4> 1.2.4 Pane Navigation Plugin </h4> <p>If we want to jump between panes in the terminal we need to add <a href="proxy.php?url=https://hyper.is/plugins/hyper-pane" rel="noopener noreferrer">Hyper Store - hyper-pane</a> plugin:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hyper-pane </code></pre> </div> <p>How it works?</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2F56z06zuoym8pn4l52cz3.gif" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2F56z06zuoym8pn4l52cz3.gif" alt="Hyper-pane"></a></p> <h4> 1.2.5 Open new tabs with the same directory </h4> <p>When opening new tabs, I often want to be in the same directory. To be able to do it, let’s add <a href="proxy.php?url=https://hyper.is/plugins/hypercwd" rel="noopener noreferrer">Hyper Store - hypercwd</a>:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hypercwd </code></pre> </div> <h4> 1.2.6 Mark active tab by adding a symbol to it </h4> <p>Sometimes when we have multiple tabs opened it might be difficult to notice which one is active at the moment. Make our life easier and add <a href="proxy.php?url=https://hyper.is/plugins/hyper-active-tab" rel="noopener noreferrer">Hyper Store - hyper-active-tab</a>:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hyper-active-tab </code></pre> </div> <p>I’m going to use a custom symbol:</p> <div class="highlight js-code-highlight"> <pre class="highlight javascript"><code> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="na">config</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// rest of the config</span> <span class="na">activeTab</span><span class="p">:</span> <span class="dl">'</span><span class="s1">🚀</span><span class="dl">'</span> <span class="p">}</span> <span class="c1">// rest of the file</span> <span class="p">}</span> </code></pre> </div> <p>Now, it looks like this:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-11.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-11.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <h4> 1.2.7 Showing CPU, RAM, battery stats </h4> <p>If we want to track our CPU, Memory, Battery resources we can add: <a href="proxy.php?url=https://github.com/hyperline/hyperline" rel="noopener noreferrer">GitHub - Hyperline/hyperline: ✨ Status line plugin for Hyper ✨</a>:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hyperline </code></pre> </div> <p>After full reload new status line will appear:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-12.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-12.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <p>There is even a Spotify plugin showing the currently played song.</p> <h4> 1.2.8 Bonus: feel this power! </h4> <p>Javascript enables users to create almost anything. Let’s see what <a href="proxy.php?url=https://hyper.is/plugins/hyperpower" rel="noopener noreferrer">Hyper Store - hyperpower</a> can do:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> hyper i hyperpower </code></pre> </div> <p>This one is quite showy:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fcloud.githubusercontent.com%2Fassets%2F13041%2F16820268%2F13c9bfe6-4905-11e6-8fe4-baf8fc8d9293.gif" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fcloud.githubusercontent.com%2Fassets%2F13041%2F16820268%2F13c9bfe6-4905-11e6-8fe4-baf8fc8d9293.gif" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <p>There are a lot of other plugins and the list is rapidly growing. For more plugins check especially those two sources:</p> <ul> <li>Official plugins store: <a href="proxy.php?url=https://hyper.is/plugins" rel="noopener noreferrer">Hyper Store - Plugins</a> </li> <li>A repository with the list of Hyper plugins: <a href="proxy.php?url=https://github.com/bnb/awesome-hyper" rel="noopener noreferrer">https://github.com/bnb/awesome-hyper</a> </li> </ul> <h2> 2. Shell - ZSH + starship prompt </h2> <p>There are a few most popular shells for linux: </p> <ul> <li>Bash Shell - <a href="proxy.php?url=https://en.wikipedia.org/wiki/Bash_(Unix_shell)" rel="noopener noreferrer">Bash (Unix shell) - Wikipedia</a> </li> <li>Tcsh/Csh Shell - <a href="proxy.php?url=https://pl.wikipedia.org/wiki/Tcsh" rel="noopener noreferrer">https://pl.wikipedia.org/wiki/Tcsh</a> </li> <li>Ksh Shell - <a href="proxy.php?url=https://pl.wikipedia.org/wiki/Korn_shell" rel="noopener noreferrer">https://pl.wikipedia.org/wiki/Korn_shell</a> </li> <li>Zsh Shell - <a href="proxy.php?url=https://pl.wikipedia.org/wiki/Zsh" rel="noopener noreferrer">https://pl.wikipedia.org/wiki/Zsh</a> </li> <li>Fish - <a href="proxy.php?url=https://pl.wikipedia.org/wiki/Friendly_interactive_shell" rel="noopener noreferrer">https://pl.wikipedia.org/wiki/Friendly_interactive_shell</a> </li> </ul> <p>After experimenting with some shells I decided to use ZSH.</p> <h3> 2.1 Installing ZSH </h3> <p>For Mac OS:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> brew <span class="nb">install </span>zsh </code></pre> </div> <p>If you don’t have a <code>homebrew</code> installed you can add it (official installation docs <a href="proxy.php?url=https://docs.brew.sh/Installation" rel="noopener noreferrer">here</a>):</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> /usr/bin/ruby <span class="nt">-e</span> <span class="s2">"</span><span class="si">$(</span>curl <span class="nt">-fsSL</span> https://raw.githubusercontent.com/Homebrew/install/master/install<span class="si">)</span><span class="s2">"</span> </code></pre> </div> <p>Homebrew is a package manager for macOS.</p> <p>After installing ZSH check its version:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> ❯ zsh <span class="nt">--version</span> zsh 5.7.1 <span class="o">(</span>x86_64-apple-darwin19.0<span class="o">)</span> ❯ which zsh /usr/local/bin/zsh </code></pre> </div> <p>To use ZSH with Hyper we need to specify it in the config (<code>~/.hyper.js</code>):</p> <div class="highlight js-code-highlight"> <pre class="highlight javascript"><code> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="na">config</span><span class="p">:</span> <span class="p">{</span> <span class="c1">// rest of the config</span> <span class="na">shell</span><span class="p">:</span> <span class="dl">'</span><span class="s1">/usr/local/bin/zsh</span><span class="dl">'</span><span class="p">,</span> <span class="p">}</span> <span class="c1">// rest of the file</span> <span class="p">}</span> </code></pre> </div> <p>At this point Hyper should start with freshly installed ZSH as default shell.<br><br> We can move on to the next point.</p> <h3> 2.2 Installing starship prompt </h3> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2F33aexbnaatk0bflj3vz1.gif" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2F33aexbnaatk0bflj3vz1.gif" alt="Starship"></a></p> <p>There are many different prompts that can be used. After testing some of them I decided to stay with <a href="proxy.php?url=https://github.com/starship/starship" rel="noopener noreferrer">GitHub - starship/starship: ☄️🌌️ The cross-shell prompt for astronauts.</a></p> <h4> 2.2.1 Install </h4> <p>There are a few different ways to install <em>starship</em> described in the documentation: <a href="proxy.php?url=https://github.com/starship/starship#-installation" rel="noopener noreferrer">GitHub - starship/starship: ☄️🌌️ The cross-shell prompt for astronauts.</a></p> <p>For Mac OS:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> brew <span class="nb">install </span>starship </code></pre> </div> <p>Check whether it’s working:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> ❯ starship <span class="nt">--version</span> starship 0.44.0 </code></pre> </div> <p>To enable <em>startship</em> prompt we need to add it to <code>~/.zshrc</code>. Add following code to the end of <code>~/.zshrc</code> file:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> <span class="nb">echo</span> <span class="s1">'eval "$(starship init zsh)"'</span> <span class="o">&gt;&gt;</span> ~/.zshrc </code></pre> </div> <p>After restarting the Hyper our terminal should look like this:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-15.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-15.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <p>Unfortunately, we don’t have any syntax highlighting, commands completion, history enabled, aliases and a few other things. We need to adjust it.</p> <h4> 2.2.2 Syntax Highlighting </h4> <p>There is another great repo with ZSH plugins that you should check out: <a href="proxy.php?url=https://github.com/unixorn/awesome-zsh-plugins" rel="noopener noreferrer">GitHub - unixorn/awesome-zsh-plugins: A collection of ZSH frameworks, plugins &amp; themes inspired by the various awesome list collections out there.</a>. I searched it for a syntax highlighting plugin.<br><br> I’m going to use <a href="proxy.php?url=https://github.com/zdharma/fast-syntax-highlighting" rel="noopener noreferrer">GitHub - zdharma/fast-syntax-highlighting: (Short name F-Sy-H). Syntax-highlighting for Zshell – fine granularity, number of features, 40 work hours themes</a>.</p> <p>I will install it by cloning the repo (it will allow later to easily keep it up-to-date by pulling the master):</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> <span class="c"># Create a `.zsh` directory to store our plugins in one place</span> <span class="nb">mkdir</span> ~/.zsh <span class="c"># Clone repo to `~/.zsh/` directory</span> <span class="nb">cd</span> ~/.zsh <span class="o">&amp;&amp;</span> git clone [email protected]:zdharma/fast-syntax-highlighting.git <span class="c"># Enable 'fast-syntax-highlighting' plugin in ZSH</span> <span class="nb">echo</span> <span class="s2">"source </span><span class="nv">$HOME</span><span class="s2">/.zsh/fast-syntax-highlighting/fast-syntax-highlighting.plugin.zsh"</span> <span class="o">&gt;&gt;</span> ~/.zshrc <span class="c"># Reload ZSH</span> <span class="nb">source</span> ~/.zshrc </code></pre> </div> <p>If you couldn't clone the repo and got error like:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> [email protected]: Permission denied <span class="o">(</span>publickey<span class="o">)</span> </code></pre> </div> <p>Then try using "Clone with HTTPS", it would look like this:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> git clone https://github.com/zdharma/fast-syntax-highlighting.git </code></pre> </div> <p>After those few steps we should have already syntax highlighting enabled:</p> <p><a href="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-16.png" class="article-body-image-wrapper"><img src="proxy.php?url=https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Ftjay.dev%2Fcontent%2Fimages%2F2019%2F09%2Fimage-16.png" alt="HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥"></a></p> <h4> 2.2.3 Setting completion plugin </h4> <p>We are going to use already defined <code>completion.zsh</code> from <a href="proxy.php?url=https://github.com/robbyrussell/oh-my-zsh" rel="noopener noreferrer">GitHub - robbyrussell/oh-my-zsh</a>:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> <span class="c"># Download completion config</span> <span class="nb">cd</span> ~/.zsh <span class="o">&amp;&amp;</span> wget https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/lib/completion.zsh <span class="c"># Enable 'completion' plugin in ZSH</span> <span class="nb">echo</span> <span class="s2">"source </span><span class="nv">$HOME</span><span class="s2">/.zsh/completion.zsh"</span> <span class="o">&gt;&gt;</span> ~/.zshrc </code></pre> </div> <p>Additionally, this time we need to add a few more lines to <code>~/.zshrc</code> file:</p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code> <span class="c"># rest of the `~/.zshrc` file</span> <span class="c"># Load completion config</span> <span class="nb">source</span> <span class="nv">$HOME</span>/.zsh/completion.zsh <span class="c"># Initialize the completion system</span> autoload <span class="nt">-Uz</span> compinit <span class="c"># Cache completion if nothing changed - faster startup time</span> <span class="nb">typeset</span> <span class="nt">-i</span> <span class="nv">updated_at</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +<span class="s1">'%j'</span> <span class="nt">-r</span> ~/.zcompdump 2&gt;/dev/null <span class="o">||</span> <span class="nb">stat</span> <span class="nt">-f</span> <span class="s1">'%Sm'</span> <span class="nt">-t</span> <span class="s1">'%j'</span> ~/.zcompdump 2&gt;/dev/null<span class="si">)</span> <span class="k">if</span> <span class="o">[</span><span class="si">$(</span><span class="nb">date</span> +<span class="s1">'%j'</span><span class="si">)</span> <span class="o">!=</span> <span class="nv">$updated_at</span><span class="o">]</span><span class="p">;</span> <span class="k">then </span>compinit <span class="nt">-i</span> <span class="k">else </span>compinit <span class="nt">-C</span> <span class="nt">-i</span> <span class="k">fi</span> <span class="c"># Enhanced form of menu completion called `menu selection'</span> zmodload <span class="nt">-i</span> zsh/complist <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; Restart the terminal and completion should work now: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: image--&gt; <span class="o">![</span>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥]<span class="o">(</span>https://tjay.dev/content/images/2019/09/image-17.png<span class="o">)</span> <span class="o">![</span>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥]<span class="o">(</span>https://tjay.dev/content/images/2019/09/image-18.png<span class="o">)</span> <span class="o">![</span>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥]<span class="o">(</span>https://tjay.dev/content/images/2019/09/image-19.png<span class="o">)</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: image--&gt; You might ask, why I don’t use Oh My Zsh? It feels overloaded <span class="k">for </span>me. I like to have control over the tools I use. I prefer to build things starting from small parts and build on that something bigger, step by step. <span class="c">#### 2.2.4 Setting autosuggestions plugin</span> Except completion feature it’s <span class="nb">nice </span>to have also suggestions. When you start typing there is a suggestion displayed <span class="k">in </span>the <span class="nb">command </span>line. Let’s use <span class="o">[</span>GitHub - zsh-users/zsh-autosuggestions: Fish-like autosuggestions <span class="k">for </span>zsh]<span class="o">(</span>https://github.com/zsh-users/zsh-autosuggestions<span class="o">)</span> plugin. Install it: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="c"># Download 'zsh-autosuggestions' plugin</span> <span class="nb">cd</span> ~/.zsh <span class="o">&amp;&amp;</span> git clone [email protected]:zsh-users/zsh-autosuggestions.git <span class="c"># Enable 'zsh-autosuggestions' plugin in ZSH</span> <span class="nb">echo</span> <span class="s2">"source </span><span class="nv">$HOME</span><span class="s2">/.zsh/zsh-autosuggestions/zsh-autosuggestions.zsh"</span> <span class="o">&gt;&gt;</span> ~/.zshrc <span class="c"># Reload ZSH</span> <span class="nb">source</span> ~/.zshrc <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; It works really nicely by suggesting the rest of the <span class="nb">command</span>, you can use arrows to accept suggested <span class="nb">command</span>: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: image--&gt; <span class="o">![</span>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥]<span class="o">(</span>https://tjay.dev/content/images/2019/09/image-20.png<span class="o">)</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: image--&gt; <span class="c">#### 2.2.5 Commands history config</span> Here again, we are going to use a ready config from oh-my-zsh: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="c"># Download history config</span> <span class="nb">cd</span> ~/.zsh <span class="o">&amp;&amp;</span> wget https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/lib/history.zsh <span class="c"># Enable 'history' config in ZSH</span> <span class="nb">echo</span> <span class="s2">"source </span><span class="nv">$HOME</span><span class="s2">/.zsh/history.zsh"</span> <span class="o">&gt;&gt;</span> ~/.zshrc <span class="c"># Reload ZSH</span> <span class="nb">source</span> ~/.zshrc <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; Now, by using the Up Arrow we can go back to our previous commands. We can also use backwards search by clicking <span class="sb">`</span>CTRL<span class="sb">`</span> + <span class="sb">`</span>R<span class="sb">`</span>: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: image--&gt; <span class="o">![</span>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥]<span class="o">(</span>https://tjay.dev/content/images/2019/09/image-21.png<span class="o">)</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: image--&gt; <span class="c">#### 2.2.6 Colorized `ls` output</span> Right now when we want to list a directory’s content we have a raw gray list: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: image--&gt; <span class="o">![</span>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥]<span class="o">(</span>https://tjay.dev/content/images/2019/09/image-22.png<span class="o">)</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: image--&gt; It doesn’t look good. Let’s use some colours to make it more readable. When you go to <span class="sb">`</span><span class="nb">ls</span><span class="sb">`</span> <span class="nb">command</span>’s manual there is a <span class="sb">`</span><span class="nt">-G</span><span class="sb">`</span> option: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="nt">-G</span> Enable colorized output. This option is equivalent to defining CLICOLOR <span class="k">in </span>the environment. <span class="o">(</span>See below.<span class="o">)</span> <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; But, we are not going to add <span class="sb">`</span><span class="nt">-G</span><span class="sb">`</span> each <span class="nb">time </span>when we want to use <span class="sb">`</span><span class="nb">ls</span><span class="sb">`</span>command. We can create an <span class="nb">alias </span><span class="k">for </span>it: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="c"># Enable colorized output for `ls` command.</span> <span class="nb">echo</span> <span class="s2">"alias ls='ls -G'"</span> <span class="o">&gt;&gt;</span> ~/.zshrc <span class="c"># Reload ZSH</span> <span class="nb">source</span> ~/.zshrc <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; It looks better now: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: image--&gt; <span class="o">![</span>HowTo: My Terminal &amp; Shell setup - Hyper.js + ZSH + starship ☄🔥]<span class="o">(</span>https://tjay.dev/content/images/2019/09/image-23.png<span class="o">)</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: image--&gt; <span class="c">#### 2.2.7 Key bindings</span> When working <span class="k">in </span>the terminal on daily basis it’s good to have shortcuts enabled. Going back to the beginning of the line <span class="o">(</span><span class="sb">`</span>CMD<span class="sb">`</span> + <span class="sb">`</span>LEFT ARROW<span class="sb">`</span><span class="o">)</span>, or to the end <span class="o">(</span><span class="sb">`</span>CMD<span class="sb">`</span> + <span class="sb">`</span>RIGHT ARROW<span class="sb">`</span><span class="o">)</span>? Let’s <span class="nb">install </span>key bindings: &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="c"># Download key bindings config</span> <span class="nb">cd</span> ~/.zsh <span class="o">&amp;&amp;</span> wget https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/lib/key-bindings.zsh <span class="c"># Enable 'key-bindings' config in ZSH</span> <span class="nb">echo</span> <span class="s2">"source </span><span class="nv">$HOME</span><span class="s2">/.zsh/key-bindings.zsh"</span> <span class="o">&gt;&gt;</span> ~/.zshrc <span class="c"># Reload ZSH</span> <span class="nb">source</span> ~/.zshrc <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; <span class="c">#### 2.2.8 Aliases</span> Shell aliases play a huge part <span class="k">in </span>productivity when using a <span class="nb">command </span>line. If you have well defined shortcuts it allows you to be really efficient and quick when developing new apps, scripts, etc. &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="nb">touch </span>aliases.zsh <span class="nb">echo</span> <span class="s2">"source </span><span class="nv">$HOME</span><span class="s2">/.zsh/aliases.zsh"</span> <span class="o">&gt;&gt;</span> ~/.zshrc <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; A few useful aliases that I often use are listed below: <span class="sb">`</span>~/.zsh/aliases.zsh<span class="sb">`</span> &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="nb">alias ls</span><span class="o">=</span><span class="s1">'ls -G'</span> <span class="c"># colorize `ls` output</span> <span class="nb">alias </span><span class="nv">zshreload</span><span class="o">=</span><span class="s1">'source ~/.zshrc'</span> <span class="c"># reload ZSH</span> <span class="nb">alias </span><span class="nv">shtop</span><span class="o">=</span><span class="s1">'sudo htop'</span> <span class="c"># run `htop` with root rights</span> <span class="nb">alias grep</span><span class="o">=</span><span class="s1">'grep --color=auto'</span> <span class="c"># colorize `grep` output</span> <span class="nb">alias</span> ..<span class="o">=</span><span class="s1">'cd ..'</span> <span class="nb">alias</span> ...<span class="o">=</span><span class="s1">'cd ../..'</span> <span class="nb">alias</span> ....<span class="o">=</span><span class="s1">'cd ../../..'</span> <span class="nb">alias </span><span class="nv">less</span><span class="o">=</span><span class="s1">'less -R'</span> <span class="nb">alias </span><span class="nv">g</span><span class="o">=</span><span class="s1">'git'</span> <span class="nb">alias rm</span><span class="o">=</span><span class="s1">'rm -i'</span> <span class="c"># confirm removal</span> <span class="nb">alias cp</span><span class="o">=</span><span class="s1">'cp -i'</span> <span class="c"># confirm copy</span> <span class="nb">alias mv</span><span class="o">=</span><span class="s1">'mv -i'</span> <span class="c"># confirm move</span> <span class="nb">alias </span><span class="nv">cal</span><span class="o">=</span><span class="s1">'gcal --starting-day=1'</span> <span class="c"># print simple calendar for current month</span> <span class="nb">alias </span><span class="nv">weather</span><span class="o">=</span><span class="s1">'curl v2.wttr.in'</span> <span class="c"># print weather for current location (https://github.com/chubin/wttr.in)</span> <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; As a bonus, I thought that sharing git aliases might also benefit you. The list below might seem to be quite broad but I added comments explaining what each of them is meant to be doing. <span class="sb">`</span>~/.gitconfig<span class="sb">`</span> &lt;<span class="o">!</span><span class="nt">--kg-card-begin</span>: code--&gt; <span class="sb">```</span>shell <span class="o">[</span><span class="nb">alias</span><span class="o">]</span> a <span class="o">=</span> add <span class="c"># Add file contents to the index</span> ai <span class="o">=</span> add <span class="nt">--interactive</span> <span class="c"># Add modified contents in the working tree interactively to the index.</span> <span class="c">##############</span> b <span class="o">=</span> branch ba <span class="o">=</span> branch <span class="nt">--all</span> <span class="c"># List both remote-tracking branches and local branches.</span> bav <span class="o">=</span> branch <span class="nt">--all</span> <span class="nt">--verbose</span> <span class="c"># When in list mode, show sha1 and commit subject line for each head, along with relationship to upstream branch (if any)</span> bd <span class="o">=</span> branch <span class="nt">--delete</span> <span class="c"># Delete a branch. The branch must be fully merged in its upstream branch, or in HEAD if no upstream was set with --track or --set-upstream-to.</span> bdd <span class="o">=</span> branch <span class="nt">-D</span> <span class="c"># Shortcut for --delete --force.</span> bm <span class="o">=</span> branch <span class="nt">--move</span> <span class="c"># Move/rename a branch and the corresponding reflog.</span> bmm <span class="o">=</span> branch <span class="nt">-M</span> <span class="c"># Shortcut for --move --force.</span> br <span class="o">=</span> branch <span class="nt">--remotes</span> <span class="c"># List or delete (if used with -d) the remote-tracking branches.</span> <span class="c">##############</span> c <span class="o">=</span> commit <span class="c"># Record changes to the repository</span> ca <span class="o">=</span> commit <span class="nt">--all</span> <span class="c"># Tell the command to automatically stage files that have been modified and deleted, but new files you have not told Git about are not affected.</span> cm <span class="o">=</span> commit <span class="nt">-m</span> <span class="c"># Use the given &lt;msg&gt; as the commit message.</span> cam <span class="o">=</span> commit <span class="nt">-am</span> <span class="c"># Shortcut for --all and -m</span> cem <span class="o">=</span> commit <span class="nt">--allow-empty</span> <span class="nt">-m</span> <span class="c"># Allows to create a commit without any files modified</span> <span class="nb">cd</span> <span class="o">=</span> commit <span class="nt">--amend</span> <span class="c"># Replace the tip of the current branch by creating a new commit.</span> cad <span class="o">=</span> commit <span class="nt">--all</span> <span class="nt">--amend</span> <span class="c"># Shortcut for --amend and --all</span> cadne <span class="o">=</span> commit <span class="nt">--all</span> <span class="nt">--amend</span> <span class="nt">--no-edit</span> <span class="c"># Amends a commit without changing its commit message.</span> <span class="c">##############</span> cl <span class="o">=</span> clone <span class="c"># Clone a repository into a new directory</span> cld <span class="o">=</span> clone <span class="nt">--depth</span> 1 <span class="c"># Create a shallow clone with a history truncated to the specified number of commits.</span> <span class="c">##############</span> <span class="nb">cp</span> <span class="o">=</span> cherry-pick <span class="c"># Apply the changes introduced by some existing commits</span> cpa <span class="o">=</span> cherry-pick <span class="nt">--abort</span> <span class="c"># Cancel the operation and return to the pre-sequence state.</span> cpc <span class="o">=</span> cherry-pick <span class="nt">--continue</span> <span class="c"># Continue the operation in progress using the information in .git/sequencer. Can be used to continue after resolving conflicts in a failed cherry-pick or revert.</span> cps <span class="o">=</span> cherry-pick <span class="nt">--skip</span> <span class="c"># Skip the current commit and continue with the rest of the sequence.</span> <span class="c">##############</span> d <span class="o">=</span> diff <span class="c"># Show changes between commits, commit and working tree, etc</span> di <span class="o">=</span> <span class="o">!</span><span class="s2">"d() { git diff --patch-with-stat HEAD~</span><span class="nv">$1</span><span class="s2">; }; git diff-index --quiet HEAD -- || clear; d"</span> <span class="c"># `git di $number` shows the diff between the state `$number` revisions ago and the current state</span> dt <span class="o">=</span> difftool <span class="c"># Show changes using common diff tools</span> <span class="c">##############</span> f <span class="o">=</span> fetch <span class="c"># Download objects and refs from another repository</span> fo <span class="o">=</span> fetch origin <span class="c"># Update the remote-tracking branches</span> fu <span class="o">=</span> fetch upstream <span class="c"># Fetch the branches and their respective commits from the upstream repository.</span> <span class="c">##############</span> fk <span class="o">=</span> fsck <span class="c"># Verifies the connectivity and validity of the objects in the database</span> <span class="c">##############</span> g <span class="o">=</span> <span class="nb">grep</span> <span class="nt">-p</span> <span class="c"># Print lines matching a pattern</span> <span class="c">##############</span> l <span class="o">=</span> log <span class="nt">--oneline</span> <span class="c"># Show commit logs, the commit message is prefixed with this information on the same line.</span> lg <span class="o">=</span> log <span class="nt">--oneline</span> <span class="nt">--graph</span> <span class="nt">--decorate</span> <span class="c"># Draw a text-based graphical representation of the commit history on the left hand side of the output.</span> lgs <span class="o">=</span> <span class="o">!</span><span class="s2">"git log --pretty=format:"</span>%C<span class="o">(</span>yellow<span class="o">)</span>%h<span class="se">\\</span> %ad%Cred%d<span class="se">\\</span> %Creset%s%Cblue<span class="se">\\</span> <span class="o">[</span>%cn]<span class="s2">" --decorate --date=short"</span> <span class="c"># SHA + date + Commit message + author</span> lgc <span class="o">=</span> <span class="o">!</span><span class="s2">"git log --pretty=format:"</span>%C<span class="o">(</span>yellow<span class="o">)</span>%h%Cred%d<span class="se">\\</span> %Creset%s%Cblue<span class="se">\\</span> <span class="o">[</span>%cn]<span class="s2">" --decorate --numstat"</span> <span class="c"># SHA + Commit message + author + changed files</span> lgt <span class="o">=</span> <span class="o">!</span><span class="s2">"git log --graph --pretty='%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)&lt;%an&gt;%Creset' --abbrev-commit --all"</span> <span class="c"># As tree: SHA + Commit message + Time ago + author</span> <span class="c">##############</span> <span class="nb">ls</span> <span class="o">=</span> ls-files <span class="c"># Show information about files in the index and the working tree</span> lsm <span class="o">=</span> ls-files <span class="nt">--modified</span> <span class="c"># Show modified files in the output</span> lss <span class="o">=</span> ls-files <span class="nt">--stage</span> <span class="c"># Show staged contents' mode bits, object name and stage number in the output.</span> <span class="c">##############</span> m <span class="o">=</span> merge <span class="c"># Join two or more development histories together</span> ma <span class="o">=</span> merge <span class="nt">--abort</span> <span class="c"># Abort the current conflict resolution process, and try to reconstruct the pre-merge state.</span> mc <span class="o">=</span> merge <span class="nt">--continue</span> <span class="c"># After a git merge stops due to conflicts you can conclude the merge by running git merge --continue</span> mq <span class="o">=</span> merge <span class="nt">--quit</span> <span class="c"># Forget about the current merge in progress. Leave the index and the working tree as-is.</span> mm <span class="o">=</span> merge master <span class="c"># Merge 'master' branch to the current branch.</span> <span class="c">##############</span> o <span class="o">=</span> checkout <span class="c"># Switch branches or restore working tree files.</span> om <span class="o">=</span> checkout master <span class="c"># Switch branch to master.</span> ob <span class="o">=</span> checkout <span class="nt">-b</span> <span class="c"># Create and switch to a new branch</span> <span class="c">##############</span> <span class="nb">pr</span> <span class="o">=</span> prune <span class="nt">--verbose</span> <span class="nt">--progress</span> <span class="c"># Prune all unreachable objects from the object database. Report all removed objects. Show progress.</span> prn <span class="o">=</span> prune <span class="nt">--dry-run</span> <span class="c"># Do not remove anything; just report what it would remove.</span> <span class="c">##############</span> ps <span class="o">=</span> push <span class="c"># Update remote refs along with associated objects</span> psa <span class="o">=</span> push <span class="nt">--all</span> <span class="c"># Push all branches (i.e. refs under refs/heads/); cannot be used with other &lt;refspec&gt;.</span> psf <span class="o">=</span> push <span class="nt">--force</span> <span class="c"># Usually, the command refuses to update a remote ref that is not an ancestor of the local ref used to overwrite it. This flag disables these checks, and can cause the remote repository to lose commits; use it with care.</span> psu <span class="o">=</span> push <span class="nt">--set-upstream</span> <span class="c"># For every branch that is up to date or successfully pushed, add upstream (tracking) reference.</span> <span class="c">##############</span> pso <span class="o">=</span> push origin <span class="c"># `origin` is an alias in the system for a particular remote repository. Can be checked by running `git remote -v`.</span> psao <span class="o">=</span> push <span class="nt">--all</span> origin <span class="c"># Same as `push --all` but for origin.</span> psfo <span class="o">=</span> push <span class="nt">--force</span> origin <span class="c"># Same as `push --force` but for origin.</span> psuo <span class="o">=</span> push <span class="nt">--set-upstream</span> origin <span class="c"># Same as `push --set-upstream` but for origin.</span> <span class="c">#############</span> psom <span class="o">=</span> push origin master <span class="c"># Same as `push origin` but for master branch.</span> psaom <span class="o">=</span> push <span class="nt">--all</span> origin master <span class="c"># Same as `push --all origin` but for master branch.</span> psfom <span class="o">=</span> push <span class="nt">--force</span> origin master <span class="c"># Same as `push --force origin` but for master branch.</span> psuom <span class="o">=</span> push <span class="nt">--set-upstream</span> origin master <span class="c"># Same as `push --set-upstream origin` but for master branch.</span> <span class="c">#############</span> pl <span class="o">=</span> pull <span class="c"># Fetch from and integrate with another repository or a local branch.</span> plr <span class="o">=</span> pull <span class="nt">--rebase</span> <span class="c"># When true, rebase the current branch on top of the upstream branch after fetching.</span> plv <span class="o">=</span> pull <span class="nt">--verbose</span> <span class="c"># Pass --verbose to git-fetch and git-merge.</span> <span class="c">#############</span> plo <span class="o">=</span> pull origin <span class="c"># Same as `pull` but for origin.</span> plro <span class="o">=</span> pull <span class="nt">--rebase</span> origin <span class="c"># Same as `pull --rebase` but for origin.</span> plom <span class="o">=</span> pull origin master <span class="c"># Same as `pull origin` but for master branch.</span> <span class="c">#############</span> plu <span class="o">=</span> pull upstream <span class="c"># Same as `pull` but for upstream.</span> plum <span class="o">=</span> pull upstream master <span class="c"># Same as `pull upstream` but for master branch.</span> plrum <span class="o">=</span> pull <span class="nt">--rebase</span> upstream master <span class="c"># Same as `pull --rebase` but for upstream and master branch.</span> <span class="c">#############</span> rb <span class="o">=</span> rebase <span class="c"># Reapply commits on top of another base tip.</span> rba <span class="o">=</span> rebase <span class="nt">--abort</span> <span class="c"># Abort the rebase operation and reset HEAD to the original branch.</span> rbc <span class="o">=</span> rebase <span class="nt">--continue</span> <span class="c"># Restart the rebasing process after having resolved a merge conflict.</span> rbi <span class="o">=</span> rebase <span class="nt">--interactive</span> <span class="c"># Make a list of the commits which are about to be rebased. Let the user edit that list before rebasing. This mode can also be used to split commits.</span> rbs <span class="o">=</span> rebase <span class="nt">--skip</span> <span class="c"># Restart the rebasing process by skipping the current patch.</span> rbin <span class="o">=</span> <span class="s2">"!r() { git rebase -i HEAD~</span><span class="nv">$1</span><span class="s2">; }; r"</span> <span class="c"># Interactive rebase with the given number of latest commits.</span> <span class="c">#############</span> re <span class="o">=</span> reset <span class="c"># Reset current HEAD to the specified state</span> rh <span class="o">=</span> reset HEAD <span class="c"># HEAD is defined explicitly</span> reh <span class="o">=</span> reset <span class="nt">--hard</span> <span class="c"># Resets the index and working tree. Any changes to tracked files in the working tree since &lt;commit&gt; are discarded.</span> rem <span class="o">=</span> reset <span class="nt">--mixed</span> <span class="c"># Resets the index but not the working tree (i.e., the changed files are preserved but not marked for commit) and reports what has not been updated. This is the default action.</span> res <span class="o">=</span> reset <span class="nt">--soft</span> <span class="c"># Does not touch the index file or the working tree at all (but resets the head to &lt;commit&gt;, just like all modes do). This leaves all your changed files "Changes to be committed".</span> rehh <span class="o">=</span> reset <span class="nt">--hard</span> HEAD <span class="c"># HEAD is defined explicitly</span> remh <span class="o">=</span> reset <span class="nt">--mixed</span> HEAD <span class="c"># HEAD is defined explicitly</span> resh <span class="o">=</span> reset <span class="nt">--soft</span> HEAD <span class="c"># HEAD is defined explicitly</span> rehom <span class="o">=</span> reset <span class="nt">--hard</span> origin/master <span class="c"># Throw away all my staged and unstaged changes, forget everything on my current local branch and make it exactly the same as origin/master.</span> <span class="c">#############</span> r <span class="o">=</span> remote <span class="c"># Manage set of tracked repositories</span> ra <span class="o">=</span> remote add <span class="c"># Adds a remote named &lt;name&gt; for the repository at &lt;url&gt;.</span> rr <span class="o">=</span> remote remove <span class="c"># Remove the remote named &lt;name&gt;. All remote-tracking branches and configuration settings for the remote are removed.</span> rv <span class="o">=</span> remote <span class="nt">--verbose</span> <span class="c"># Be a little more verbose and show remote url after name.</span> rn <span class="o">=</span> remote rename <span class="c"># Rename the remote named &lt;old&gt; to &lt;new&gt;. All remote-tracking branches and configuration settings for the remote are updated.</span> rp <span class="o">=</span> remote prune <span class="c"># Deletes stale references associated with &lt;name&gt;. By default, stale remote-tracking branches under &lt;name&gt; are deleted, but depending on global configuration and the configuration of the remote we might even prune local tags that haven't been pushed there.</span> rs <span class="o">=</span> remote show <span class="c"># Gives some information about the remote &lt;name&gt;.</span> rao <span class="o">=</span> remote add origin <span class="c"># Add new origin.</span> rau <span class="o">=</span> remote add upstream <span class="c"># Add new upstream.</span> rro <span class="o">=</span> remote remove origin <span class="c"># Remove origin.</span> rru <span class="o">=</span> remote remove upstream <span class="c"># Remove upstream.</span> rso <span class="o">=</span> remote show origin <span class="c"># Show current origin.</span> rsu <span class="o">=</span> remote show upstream <span class="c"># Show current upstream.</span> rpo <span class="o">=</span> remote prune origin <span class="c"># Prune current origin.</span> rpu <span class="o">=</span> remote prune upstream <span class="c"># Prune current upstream.</span> <span class="c">#############</span> rmf <span class="o">=</span> <span class="nb">rm</span> <span class="nt">-f</span> <span class="c"># Remove files from the working tree and from the index. Override the up-to-date check.</span> rmrf <span class="o">=</span> <span class="nb">rm</span> <span class="nt">-r</span> <span class="nt">-f</span> <span class="c"># Same as above + Allow recursive removal when a leading directory name is given.</span> <span class="c">#############</span> s <span class="o">=</span> status <span class="c"># Show the working tree status</span> sb <span class="o">=</span> status <span class="nt">-s</span> <span class="nt">-b</span> <span class="c"># Same as above + Give the output in the short-format. Show the branch and tracking info even in short-format.</span> <span class="c">#############</span> sa <span class="o">=</span> stash apply <span class="c"># Like pop, but do not remove the state from the stash list.</span> sc <span class="o">=</span> stash clear <span class="c"># Remove all the stash entries. Note that those entries will then be subject to pruning, and may be impossible to recover.</span> sd <span class="o">=</span> stash drop <span class="c"># Remove a single stash entry from the list of stash entries. When no &lt;stash&gt; is given, it removes the latest one.</span> sl <span class="o">=</span> stash list <span class="c"># List the stash entries that you currently have.</span> sp <span class="o">=</span> stash pop <span class="c"># Remove a single stashed state from the stash list and apply it on top of the current working tree state, i.e., do the inverse operation of git stash push.</span> sps <span class="o">=</span> stash push <span class="c"># Save your local modifications to a new stash entry and roll them back to HEAD (in the working tree and in the index). The &lt;message&gt; part is optional and gives the description along with the stashed state.</span> spsk <span class="o">=</span> stash push <span class="nt">-k</span> <span class="c"># All changes already added to the index are left intact.</span> sw <span class="o">=</span> stash show <span class="c"># Show the changes recorded in the stash entry as a diff between the stashed contents and the commit back when the stash entry was first created. When no &lt;stash&gt; is given, it shows the latest one.</span> st <span class="o">=</span> <span class="o">!</span>git stash list | <span class="nb">wc</span> <span class="nt">-l</span> 2&gt;/dev/null | <span class="nb">grep</span> <span class="nt">-oEi</span> <span class="s1">'[0-9][0-9]*'</span> <span class="c">#############</span> t <span class="o">=</span> tag <span class="c"># Create, list, delete or verify a tag object signed with GPG.</span> td <span class="o">=</span> tag <span class="nt">--delete</span> <span class="c"># Delete existing tags with the given names.</span> tl <span class="o">=</span> tag <span class="nt">--list</span> <span class="c"># Show verbose output about tags.</span> <span class="c">#############</span> w <span class="o">=</span> show <span class="c"># Show various types of objects.</span> wo <span class="o">=</span> show <span class="nt">--oneline</span> <span class="c"># This is a shorthand for "--pretty=oneline --abbrev-commit" used together.</span> wf <span class="o">=</span> show <span class="nt">--format</span><span class="o">=</span>fuller <span class="c"># Print more extensive info.</span> <span class="c">#############</span> aliases <span class="o">=</span> <span class="o">!</span>git config <span class="nt">-l</span> | <span class="nb">grep alias</span> | <span class="nb">cut</span> <span class="nt">-c</span> 7- <span class="c"># List git aliases</span> branches <span class="o">=</span> branch <span class="nt">--all</span> <span class="c"># List both remote-tracking branches and local branches.</span> remotes <span class="o">=</span> remote <span class="nt">--verbose</span> <span class="c"># Be a little more verbose and show remote url after name.</span> contributors <span class="o">=</span> shortlog <span class="nt">--summary</span> <span class="nt">--numbered</span> <span class="c"># List contributors with number of commits</span> amend <span class="o">=</span> commit <span class="nt">--amend</span> <span class="nt">--no-edit</span> <span class="c"># Amend the currently staged files to the latest commit.</span> go <span class="o">=</span> <span class="s2">"!f() { git checkout -b </span><span class="se">\"</span><span class="nv">$1</span><span class="se">\"</span><span class="s2"> 2&gt; /dev/null || git checkout </span><span class="se">\"</span><span class="nv">$1</span><span class="se">\"</span><span class="s2">; }; f"</span> <span class="c"># Switch to a branch, creating it if necessary</span> fb <span class="o">=</span> <span class="s2">"!f() { git branch -a --contains </span><span class="nv">$1</span><span class="s2">; }; f"</span> <span class="c"># Find branches containing commit</span> ft <span class="o">=</span> <span class="s2">"!f() { git describe --always --contains </span><span class="nv">$1</span><span class="s2">; }; f"</span> <span class="c"># Find tags containing commit</span> <span class="nb">fc</span> <span class="o">=</span> <span class="s2">"!f() { git log --pretty=format:'%C(yellow)%h %Cblue%ad %Creset%s%Cgreen [%cn] %Cred%d' --decorate --date=short -S</span><span class="nv">$1</span><span class="s2">; }; f"</span> <span class="c"># Find commits by source code</span> fm <span class="o">=</span> <span class="s2">"!f() { git log --pretty=format:'%C(yellow)%h %Cblue%ad %Creset%s%Cgreen [%cn] %Cred%d' --decorate --date=short --grep=</span><span class="nv">$1</span><span class="s2">; }; f"</span> <span class="c"># Find commits by commit message</span> dm <span class="o">=</span> <span class="s2">"!git branch --merged | grep -v '</span><span class="se">\\</span><span class="s2">*' | xargs -n 1 git branch -d"</span> <span class="c"># Remove branches that have already been merged with master (a.k.a. ‘delete merged’)</span> <span class="sb">```</span> &lt;<span class="o">!</span><span class="nt">--kg-card-end</span>: code--&gt; <span class="c">## Summary</span> I hope you will find this article useful and it will <span class="nb">help </span>you boost your productivity <span class="k">in </span>the <span class="nb">command </span>line. It’s not an ultimate setup, rather treat it as a beginning or enhancement <span class="k">for </span>your current setup. The amount of available tools, plugins, configs is simply overwhelming. Each user has different use cases, different technologies that he uses. I tried to focus on the common ground which could be used by anyone. If you know and recommend some of the tools you use and can’t live without please <span class="nb">let </span>me know <span class="k">in </span>the comments. <span class="k">**</span>This article was originally posted on my personal dev blog: <span class="o">[</span>https://tjay.dev/]<span class="o">(</span>https://tjay.dev/howto-my-terminal-shell-setup-hyper-js-zsh-starship/?utm_source<span class="o">=</span>dev.to&amp;utm_medium<span class="o">=</span>footer-link&amp;utm_campaign<span class="o">=</span>reblogging<span class="o">)</span><span class="k">**</span> _Photo by <span class="o">[</span>Анна В]<span class="o">(</span>https://unsplash.com/@niakris?utm_source<span class="o">=</span>unsplash&amp;utm_medium<span class="o">=</span>referral&amp;utm_content<span class="o">=</span>creditCopyText<span class="o">)</span> on <span class="o">[</span>Unsplash]<span class="o">(</span>https://unsplash.com/search/photos/shell?utm_source<span class="o">=</span>unsplash&amp;utm_medium<span class="o">=</span>referral&amp;utm_content<span class="o">=</span>creditCopyText<span class="o">)</span>_ </code></pre> </div> terminal shell productivity tutorial HowTo: Working with large files in Ruby efficiently. Thomas Jaskiewicz Tue, 03 Sep 2019 13:07:41 +0000 https://dev.to/netguru/howto-working-with-large-files-in-ruby-efficiently-189e https://dev.to/netguru/howto-working-with-large-files-in-ruby-efficiently-189e <h2> How can we read files in Ruby? </h2> <h3> * Testing file generated by running a following command: </h3> <div class="highlight"><pre class="highlight shell"><code>❯ openssl req <span class="nt">-newkey</span> rsa:2048 <span class="nt">-new</span> <span class="nt">-nodes</span> <span class="nt">-x509</span> <span class="nt">-days</span> 3650 <span class="nt">-keyout</span> key.pem <span class="nt">-out</span> cert.pem </code></pre></div> <p>It has a clearly defined the beginning and the end of the file which fill be useful while reading the files.</p> <h3> 1. <code>File.read()</code> which is actually <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-c-read"><code>IO.read()</code></a>: </h3> <div class="highlight"><pre class="highlight ruby"><code><span class="o">&gt;</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="s2">"cert.pem"</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="s2">"-----BEGIN CERTIFICATE-----</span><span class="se">\n</span><span class="s2">MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ</span><span class="se">\n</span><span class="s2">TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM</span><span class="se">\n</span><span class="s2">MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ</span><span class="se">\n</span><span class="s2">zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr</span><span class="se">\n</span><span class="s2">1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z</span><span class="se">\n</span><span class="s2">TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H</span><span class="se">\n</span><span class="s2">7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi</span><span class="se">\n</span><span class="s2">aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/</span><span class="se">\n</span><span class="s2">4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE</span><span class="se">\n</span><span class="s2">raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR</span><span class="se">\n</span><span class="s2">9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp</span><span class="se">\n</span><span class="s2">GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd</span><span class="se">\n</span><span class="s2">aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6</span><span class="se">\n</span><span class="s2">lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV</span><span class="se">\n</span><span class="s2">-----END CERTIFICATE-----</span><span class="se">\n</span><span class="s2">"</span> <span class="o">&gt;</span> <span class="n">file</span><span class="p">.</span><span class="nf">bytesize</span> <span class="o">=&gt;</span> <span class="mi">956</span> <span class="o">&gt;</span> <span class="n">file</span><span class="p">.</span><span class="nf">class</span> <span class="o">=&gt;</span> <span class="no">String</span> </code></pre></div> <p><code>read</code> method reads the entire file's content and assigns it to the variable as single String.</p> <h3> 2. <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/File.html#method-c-new"><code>File.new()</code></a> and its synonym <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/File.html#method-c-open"><code>File.open()</code></a>: </h3> <div class="highlight"><pre class="highlight ruby"><code><span class="o">&gt;</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"cert.pem"</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="c1">#&lt;File:cert.pem&gt;</span> <span class="o">&gt;</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">file</span><span class="p">.</span><span class="nf">readlines</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="s2">"-----BEGIN CERTIFICATE-----</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"-----END CERTIFICATE-----</span><span class="se">\n</span><span class="s2">"</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">lines</span><span class="p">.</span><span class="nf">class</span> <span class="o">=&gt;</span> <span class="no">Array</span> </code></pre></div> <p><code>new</code> or <code>open</code> methods returns an instance of the <code>File</code> class on which we can call <code>readlines</code> method which reads the entire file's content, splits it line by line and returns an Array of Strings where one element is one line from the file.</p> <h3> 3. <code>File.readlines()</code> which is actually <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-c-readlines"><code>IO.readlines()</code></a>: </h3> <div class="highlight"><pre class="highlight ruby"><code><span class="o">&gt;</span> <span class="n">lines</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">readlines</span><span class="p">(</span><span class="s2">"cert.pem"</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="s2">"-----BEGIN CERTIFICATE-----</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"-----END CERTIFICATE-----</span><span class="se">\n</span><span class="s2">"</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">lines</span><span class="p">.</span><span class="nf">class</span> <span class="o">=&gt;</span> <span class="no">Array</span> </code></pre></div> <p>Here, we have the same output as in the previous example by calling just class method <code>readlines</code> on <code>File</code> class.</p> <h3> 4. <code>File.foreach()</code> which is actually <code>IO.foreach()</code>: </h3> <div class="highlight"><pre class="highlight ruby"><code><span class="o">&gt;</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">foreach</span><span class="p">(</span><span class="s2">"./cert.pem"</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="c1">#&lt;Enumerator: ...&gt;</span> <span class="o">&gt;</span> <span class="n">file</span><span class="p">.</span><span class="nf">entries</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="s2">"-----BEGIN CERTIFICATE-----</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"-----END CERTIFICATE-----</span><span class="se">\n</span><span class="s2">"</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">lines</span><span class="p">.</span><span class="nf">class</span> <span class="o">=&gt;</span> <span class="no">Array</span> </code></pre></div> <p><code>foreach</code> method returns an Enumerator instance on which we call <code>entries</code> which returns an Array of String, again each element is a line from the file.</p> <p>As we can see above there are many methods that allow us to read the file. However which one should we use and why? Let's create a large file and check those methods again!</p> <h2> Which methods should we use to read large files? </h2> <h3> Generating our test file </h3> <p>At first, let's generate a large file with randomized data inside:<br> </p> <div class="highlight"><pre class="highlight ruby"><code><span class="nb">require</span> <span class="s1">'securerandom'</span> <span class="n">one_megabyte</span> <span class="o">=</span> <span class="mi">1024</span> <span class="o">*</span> <span class="mi">1024</span> <span class="nb">name</span> <span class="o">=</span> <span class="s2">"large_1G"</span> <span class="n">size</span> <span class="o">=</span> <span class="mi">1000</span> <span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"./</span><span class="si">#{</span><span class="nb">name</span><span class="si">}</span><span class="s2">.txt"</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span> <span class="n">size</span><span class="p">.</span><span class="nf">times</span> <span class="k">do</span> <span class="n">file</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="no">SecureRandom</span><span class="p">.</span><span class="nf">random_bytes</span><span class="p">(</span><span class="n">one_megabyte</span><span class="p">))</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div> <ul> <li> <code>w</code> - Write-only, truncates existing file to zero length or creates a new file for writing.</li> <li> <code>b</code> - Binary file mode. Suppresses EOL &lt;-&gt; CRLF conversion on Windows. And sets external encoding to ASCII-BIT unless explicitly specified.</li> </ul> <p>As the result we generated 1GB file:<br> </p> <div class="highlight"><pre class="highlight shell"><code>❯ <span class="nb">ls</span> <span class="nt">-lah</span> ... <span class="nt">-rw-r--r--</span> 1 user user 1.0G Aug 31 22:10 large_1G.txt </code></pre></div> <h3> Defining our metrics and profilers </h3> <p>There are probably 2 the most important metrics that we would like to track in our experiment:</p> <ul> <li> <strong>Time</strong> - How long does it take to open and read the file?</li> <li> <strong>Memory</strong> - How much memory does it take to open and read the file?</li> </ul> <p>Also there will be one additional metric describing how many objects were freed by Garbage Collector.</p> <p>We can prepare simple profiling methods:<br> </p> <div class="highlight"><pre class="highlight ruby"><code><span class="c1"># ./helpers.rb</span> <span class="nb">require</span> <span class="s1">'benchmark'</span> <span class="k">def</span> <span class="nf">profile_memory</span> <span class="n">memory_usage_before</span> <span class="o">=</span> <span class="sb">`ps -o rss= -p </span><span class="si">#{</span><span class="no">Process</span><span class="p">.</span><span class="nf">pid</span><span class="si">}</span><span class="sb">`</span><span class="p">.</span><span class="nf">to_i</span> <span class="k">yield</span> <span class="n">memory_usage_after</span> <span class="o">=</span> <span class="sb">`ps -o rss= -p </span><span class="si">#{</span><span class="no">Process</span><span class="p">.</span><span class="nf">pid</span><span class="si">}</span><span class="sb">`</span><span class="p">.</span><span class="nf">to_i</span> <span class="n">used_memory</span> <span class="o">=</span> <span class="p">((</span><span class="n">memory_usage_after</span> <span class="o">-</span> <span class="n">memory_usage_before</span><span class="p">)</span> <span class="o">/</span> <span class="mf">1024.0</span><span class="p">).</span><span class="nf">round</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="nb">puts</span> <span class="s2">"Memory usage: </span><span class="si">#{</span><span class="n">used_memory</span><span class="si">}</span><span class="s2"> MB"</span> <span class="k">end</span> <span class="k">def</span> <span class="nf">profile_time</span> <span class="n">time_elapsed</span> <span class="o">=</span> <span class="no">Benchmark</span><span class="p">.</span><span class="nf">realtime</span> <span class="k">do</span> <span class="k">yield</span> <span class="k">end</span> <span class="nb">puts</span> <span class="s2">"Time: </span><span class="si">#{</span><span class="n">time_elapsed</span><span class="p">.</span><span class="nf">round</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="si">}</span><span class="s2"> seconds"</span> <span class="k">end</span> <span class="k">def</span> <span class="nf">profile_gc</span> <span class="no">GC</span><span class="p">.</span><span class="nf">start</span> <span class="n">before</span> <span class="o">=</span> <span class="no">GC</span><span class="p">.</span><span class="nf">stat</span><span class="p">(</span><span class="ss">:total_freed_objects</span><span class="p">)</span> <span class="k">yield</span> <span class="no">GC</span><span class="p">.</span><span class="nf">start</span> <span class="n">after</span> <span class="o">=</span> <span class="no">GC</span><span class="p">.</span><span class="nf">stat</span><span class="p">(</span><span class="ss">:total_freed_objects</span><span class="p">)</span> <span class="nb">puts</span> <span class="s2">"Objects Freed: </span><span class="si">#{</span><span class="n">after</span> <span class="o">-</span> <span class="n">before</span><span class="si">}</span><span class="s2">"</span> <span class="k">end</span> <span class="k">def</span> <span class="nf">profile</span> <span class="n">profile_memory</span> <span class="k">do</span> <span class="n">profile_time</span> <span class="k">do</span> <span class="n">profile_gc</span> <span class="k">do</span> <span class="k">yield</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> </code></pre></div> <h3> Testing our methods for reading files </h3> <ul> <li> <code>.read</code> </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">)</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">39</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">0.52</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">1000.05</span> <span class="no">MB</span> </code></pre></div> <ul> <li> <code>.new</code> + <code>#readlines</code> </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">).</span><span class="nf">readlines</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">39</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">4.19</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">1298.4</span> <span class="no">MB</span> </code></pre></div> <ul> <li> <code>.readlines</code> </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">readlines</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">)</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">39</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">4.24</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">1284.61</span> <span class="no">MB</span> </code></pre></div> <ul> <li> <code>.foreach</code> </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">foreach</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">).</span><span class="nf">to_a</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">40</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">4.42</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">1284.31</span> <span class="no">MB</span> </code></pre></div> <p>The examples we can see above allowed us to read the whole file and store it in local memory as one String or as an Array of Strings (each line from the file as one element in the Array).</p> <p>As we can see, it requires <strong>at least as much memory as the size of the file</strong>:</p> <ul> <li>one <strong>String</strong> - 1GB file requires 1GB of memory. </li> <li>an <strong>Array of Strings</strong> - 1GB memory for file's content + additional memory for an Array (+- 300MB here). This approach has one advantage, we can access whichever line of the file we want as long as we know which line is it.</li> </ul> <p>At this point we can see that the methods that we tested are not really efficient. The bigger the file, the more memory we need. In longer term this approach might lead to some serious consequences, even killing the application.</p> <p>Now, we need to us ourselves a question. <strong>Can we process our files line by line?</strong> If so, then we can read our files in a different way:</p> <ul> <li> <code>.new</code> + <code>#each</code> </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">)</span> <span class="n">file</span><span class="p">.</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">line</span><span class="o">|</span> <span class="n">line</span> <span class="p">}</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">4100808</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">2.08</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">57.68</span> <span class="no">MB</span> </code></pre></div> <ul> <li> <code>.new</code> + <code>#advise</code> + <code>#each</code> </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">)</span> <span class="n">file</span><span class="p">.</span><span class="nf">advise</span><span class="p">(</span><span class="ss">:sequential</span><span class="p">)</span> <span class="n">file</span><span class="p">.</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">line</span><span class="o">|</span> <span class="n">line</span> <span class="p">}</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">4100808</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">2.22</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">55.71</span> <span class="no">MB</span> </code></pre></div> <p>Calling <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-i-advise"><code>#advise</code></a> method announces an intention to access data from the current file in a specific pattern. No major improvement here with using <code>#advise</code> method.</p> <ul> <li> <code>.new</code> + <code>#read</code> - reading chunk by chunk </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">chunk_size</span> <span class="o">=</span> <span class="mi">4096</span> <span class="n">buf</span> <span class="o">=</span> <span class="s2">""</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">)</span> <span class="k">while</span> <span class="n">buf</span> <span class="o">=</span> <span class="n">file</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">chunk_size</span><span class="p">)</span> <span class="n">buf</span><span class="p">.</span><span class="nf">tap</span> <span class="p">{</span> <span class="o">|</span><span class="n">buf</span><span class="o">|</span> <span class="n">buf</span> <span class="p">}</span> <span class="k">end</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">256037</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">1.27</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">131.64</span> <span class="no">MB</span> </code></pre></div> <p>We defined the <code>chunk</code> as 4096 bytes and we read our file chunk by chunk. Depending on the structure of your file this approach might be useful.</p> <ul> <li> <code>.foreach</code> + <code>#each_entry</code> </li> </ul> <div class="highlight"><pre class="highlight ruby"><code><span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">foreach</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">)</span> <span class="n">file</span><span class="p">.</span><span class="nf">each_entry</span> <span class="p">{</span> <span class="o">|</span><span class="n">line</span><span class="o">|</span> <span class="n">line</span> <span class="p">}</span> <span class="k">end</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">4100809</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">2.22</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">53.02</span> <span class="no">MB</span> </code></pre></div> <p>Creating an Enumerator instance as <code>file</code> and reading file line by line using <code>each_entry</code> method.</p> <p>First thing we can notice is that memory usage is way lower. Main reason for that is that we read the file line by line and when the line is processed then it's garbage collected. We can see that by the size of the <em>Objects Freed</em>, it's quite high.</p> <p>We also tried to use here an <code>#advise</code> method which we can tell how we want to process our file. More about <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-i-advise"><code>IO#advise</code></a> can be found in the documentation. Unfortunately, it didn't help us out here.</p> <p>Except <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-i-each"><code>IO#each</code></a> method we have also similar methods like <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-i-each_byte"><code>IO#each_byte</code></a> (reading byte by byte),<a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-i-each_char"><code>IO#each_char</code></a> (reading char by char) and <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-i-each_codepoint"><code>IO#each_codepoint</code></a>.</p> <p>In the example with reading by chunks (<a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-i-read"><code>IO#read</code></a>) the memory usage will vary depending on the chunk size. If you find this way useful you can experiment with the chunk size.</p> <p>When using <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/IO.html#method-c-foreach"><code>IO.foreach</code></a> we operate on Enumerator which gives us a few more methods like: <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-each_entry"><code>IO#each_entry</code></a>, <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-each_slice"><code>IO#each_slice</code></a>, <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-each_cons"><code>IO#each_cons</code></a>. There is also <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-lazy"><code>lazy</code></a> method which returns a <a href="proxy.php?url=https://ruby-doc.org/core-2.6.4/Enumerator/Lazy.html">Enumerator::Lazy</a>. Lazy Enumerator has a few additional methods which enumerate values only on an as-needed basis. If you don't need to read the entire file but, for example, looking for a particular line containing given expression then it might be worth to check it out.</p> <p>I could finish the article at this point, but what if before we even start reading the file we need to decrypt it? Let's move further to the example.</p> <h3> Decrypting large file and processing it line by line </h3> <h4> Prerequisites </h4> <p>Before we decrypt the file we need to encrypt our generated large file. We are going to use AES with 256 bits key length with Cipher Block Chaining (CBC) as mode.<br> </p> <div class="highlight"><pre class="highlight ruby"><code><span class="n">cipher</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">Cipher</span><span class="o">::</span><span class="no">AES256</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:CBC</span><span class="p">)</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">encrypt</span> <span class="no">KEY</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">random_key</span> <span class="no">IV</span> <span class="o">=</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">random_iv</span> </code></pre></div> <p>Now, let's encrypt out file:<br> </p> <div class="highlight"><pre class="highlight ruby"><code><span class="n">cipher</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">Cipher</span><span class="o">::</span><span class="no">AES256</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:CBC</span><span class="p">)</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">encrypt</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">key</span> <span class="o">=</span> <span class="no">KEY</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">iv</span> <span class="o">=</span> <span class="no">IV</span> <span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">enc_file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">)</span> <span class="n">enc_file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"large_1G.txt.enc"</span><span class="p">,</span> <span class="s2">"wb"</span><span class="p">)</span> <span class="n">enc_file</span> <span class="o">&lt;&lt;</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">update</span><span class="p">(</span><span class="n">file</span><span class="p">)</span> <span class="n">enc_file</span> <span class="o">&lt;&lt;</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">final</span> <span class="k">end</span> <span class="n">file</span><span class="p">.</span><span class="nf">close</span> <span class="n">enc_file</span><span class="p">.</span><span class="nf">close</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">12</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">3.6</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">1000.02</span> <span class="no">MB</span> </code></pre></div> <p>Seems like encrypting is also a quite memory consuming task. Let's adjust the algorithm a little bit:<br> </p> <div class="highlight"><pre class="highlight ruby"><code><span class="n">cipher</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">Cipher</span><span class="o">::</span><span class="no">AES256</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:CBC</span><span class="p">)</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">encrypt</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">key</span> <span class="o">=</span> <span class="no">KEY</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">iv</span> <span class="o">=</span> <span class="no">IV</span> <span class="n">file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">enc_file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">buf</span> <span class="o">=</span> <span class="s2">""</span> <span class="n">file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"large_1G.txt"</span><span class="p">,</span> <span class="s2">"rb"</span><span class="p">)</span> <span class="n">enc_file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"large_1G.txt.enc"</span><span class="p">,</span> <span class="s2">"wb"</span><span class="p">)</span> <span class="k">while</span> <span class="n">buf</span> <span class="o">=</span> <span class="n">file</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="mi">4096</span><span class="p">)</span> <span class="n">enc_file</span> <span class="o">&lt;&lt;</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">update</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span> <span class="k">end</span> <span class="n">enc_file</span> <span class="o">&lt;&lt;</span> <span class="n">cipher</span><span class="p">.</span><span class="nf">final</span> <span class="k">end</span> <span class="n">file</span><span class="p">.</span><span class="nf">close</span> <span class="n">enc_file</span><span class="p">.</span><span class="nf">close</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">768048</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">5.05</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">145.93</span> <span class="no">MB</span> </code></pre></div> <p>By changing the algorithm to read and cipher the file by chunks made the task much less memory consuming.</p> <h4> Decrypt </h4> <p>All right, let's try to decrypt it now:<br> </p> <div class="highlight"><pre class="highlight ruby"><code><span class="n">decipher</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">Cipher</span><span class="o">::</span><span class="no">AES256</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:CBC</span><span class="p">)</span> <span class="n">decipher</span><span class="p">.</span><span class="nf">decrypt</span> <span class="n">decipher</span><span class="p">.</span><span class="nf">key</span> <span class="o">=</span> <span class="no">KEY</span> <span class="n">decipher</span><span class="p">.</span><span class="nf">iv</span> <span class="o">=</span> <span class="no">IV</span> <span class="n">dec_file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">enc_file</span> <span class="o">=</span> <span class="kp">nil</span> <span class="n">profile</span> <span class="k">do</span> <span class="n">buf</span> <span class="o">=</span> <span class="s2">""</span> <span class="n">enc_file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"large_1G.txt.enc"</span><span class="p">,</span> <span class="s2">"rb"</span><span class="p">)</span> <span class="n">dec_file</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"large_1G.txt.dec"</span><span class="p">,</span> <span class="s2">"wb"</span><span class="p">)</span> <span class="k">while</span> <span class="n">buf</span> <span class="o">=</span> <span class="n">enc_file</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="mi">4096</span><span class="p">)</span> <span class="n">dec_file</span> <span class="o">&lt;&lt;</span> <span class="n">decipher</span><span class="p">.</span><span class="nf">update</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span> <span class="k">end</span> <span class="n">dec_file</span> <span class="o">&lt;&lt;</span> <span class="n">decipher</span><span class="p">.</span><span class="nf">final</span> <span class="k">end</span> <span class="n">dec_file</span><span class="p">.</span><span class="nf">close</span> <span class="n">enc_file</span><span class="p">.</span><span class="nf">close</span> <span class="no">Objects</span> <span class="no">Freed</span><span class="p">:</span> <span class="mi">768050</span> <span class="no">Time</span><span class="p">:</span> <span class="mf">3.5</span> <span class="n">seconds</span> <span class="no">Memory</span> <span class="ss">usage: </span><span class="mf">152.12</span> <span class="no">MB</span> </code></pre></div> <p>Now, let's compare our files whether we properly encrypted and decrypted it:<br> </p> <div class="highlight"><pre class="highlight shell"><code>❯ diff large_1G.txt large_1G.txt.dec </code></pre></div> <p>No differences were found. We are good here!</p> <p>We managed to lower the memory usage quite significantly. That's great! </p> <p>Treat this article as a toolset that you can use in your specific case.</p> <p><strong>This article was originally posted on my personal dev blog: <a href="proxy.php?url=https://tjay.dev/howto-working-efficiently-with-large-files-in-ruby/?utm_source=dev.to&amp;utm_medium=footer-link&amp;utm_campaign=reblogging">https://tjay.dev/</a></strong></p> <p><em>Photo by <a href="proxy.php?url=https://unsplash.com/@erwanhesry">Erwan Hesry</a> on <a href="proxy.php?url=https://unsplash.com/search/photos/luggage">Unsplash</a></em></p> ruby programming filesprocessing