Fancy

Bpf

2020-07-20T00:00:00+00:00

BPF

bpf

image from http://www.brendangregg.com/ebpf.html

kprobes
- kprobes can trace arbitrary functions, but may easily break from one Linux version to the next
tracepoints
- tracepoints are more stable, they remain mostly the same
- and documented in /sys/kernel/debug/tracing
- can list with sudo perf list
perf
- http://www.brendangregg.com/perf.html

refer:

https://stackoverflow.com/questions/45618274/are-tracepoints-redundant-in-linux-kernel-after-kprobes-support-for-ftrace

Projects/Tools/Examples

bcc
bpftrace
libbpf
libbpf-tools
bpftool
sampes/bpf/ in kernel tree
perf, ftrace

Usage

cgroup
systemd service
tc-bpf
iptables bpf
socket bpf
ip-route bpf
sockmap

Build kernel samples

# kernel config
gunzip -c /proc/config.gz > .config
make oldconfig && make prepare
# install headers
make headers_install -j8 
# build bpf
make M=samples/bpf -j8

# run
ulimit -l unlimited
sudo ./sockex1

Must Read

bpf.h in kernel tree
sudo bpftool feature probe output
Linux Socket Filtering aka Berkeley Packet Filter (BPF)
Notes on bpf series
Run a BPF program on socket lookup
- presentation
- inet_lookup
- BPF_PROG_TYPE_SK_LOOKUP
SOCKMAP - TCP splicing of the future
- two bpf prog attached to one sockmap
- BPF_MAP_TYPE_SOCKMAP
- BPF_SK_SKB_STREAM_PARSER, BPF_SK_SKB_STREAM_VERDICT

TYPE（not complete）

socket: for iptables –bpf, setsockopt
- BPF_PROG_TYPE_SOCKET_FILTER
cgroup/sock:
- BPF_PROG_TYPE_CGROUP_SOCK
- BPF_CGROUP_INET_SOCK_CREATE
- parameters: see bpf_ctx_convert in vmlinux.h
  - bpf_sock or __sk_buff
cgroup_skb/egress: for systemd cgroup
- BPF_PROG_TYPE_CGROUP_SKB
- BPF_CGROUP_INET_EGRESS
see bpf_sec_def in libbpf.c
see samples in linux/samples/bpf/

Transparent_proxy

2020-04-17T00:00:00+00:00

透明代理技术总结–Linux

1. iptables REDIRECT/TPROXY

在linux网络堆栈上实现透明代理，功能上最强大，最完美。具体可以有以下三种方式：

global，即全局透明代理
network namespace，在网络命名空间中透明代理，主机网络相当于中间路由器，具体实现参考 ~/Scripts/ns-proxy.sh
net_cls, only in cgroup v1，对特定cgroup进行透明代理，参考 example
iptables cgroup2 path match

监听REDIRECT/TPROXY端口的技术实现：

redsocks link
v2ray dokodemo-door link
gost -L red://:12345 link

未来会出现基于bpf技术，以最小的footprint，实现透明代理

linux内核的bpf技术正在快速发展

2. proxychains-ng

原理是利用LD_PRELOAD，挂钩(hook)网络相关的libc函数，因此只对动态链接的程序有用，例如不支持go程序（静态编译），不支持脚本等。只支持tcp。

配置和使用相当的简单。

3. cgproxy

基于cgroup2和TPROXY的透明代理，支持cgroup级别、进程级别的控制。

支持tcp和udp。

可以代理任何程序，可全局代理、网关代理。

4. graftcp

可以通过ptrace跟踪或修改任何给定程序的网络连接，因此它可用于任何程序。

只支持tcp，暂不支持ipv6，issue

some tests

see Dokodemo-door and v2ray tproxy config,

note v2ray redirect only work in ipv4

experiments result following:

v2ray tproxy + no root
- tcp REDIRECT ✓
- tcp TPROXY ✗
- udp TPROXY ✗
v2ray redirect + no root
- tcp REDIRECT ✓
- tcp TPROXY ✗
- udp TPROXY ✗
v2ray tproxy + root ✓
v2ray redirect + root ✓

结论：

REDIRECT总是可用, 而TPROXY需要root才能使用
在not root下只能代理透明代理tcp

一些代理软件

gost
clash
v2ray
trojan

Mpv_shooter_subtitles_plugin

2020-01-01T00:00:00+00:00

MPV-射手影音字幕插件

put SPlayerSub.py and download_subtitles_using_SPlayer_API.js to ~/.config/mpv/scripts/
put mpv.conf to ~/.config/mpv/
打开视频后，按a 开始下载字幕，字幕会下载到视频~/.cache/mpv/目录，并自动加载

refer:

Openvpn

2019-10-07T00:00:00+00:00

openvpn

OpenVPN installer for Debian, Ubuntu, Fedora, CentOS and Arch Linux.

遇到的问题

主机有多个IP的情况下，使用UDP会存在地址绑定不一致的情况，即：

解决方案有二：

选择使用TCP，我的选择
-multihome编译，扩展sock API，具体参见OPENVPN文档

–multihome

Configure a multi-homed UDP server. This option needs to be used when a server has more than one IP address (e.g. multiple interfaces, or secondary IP addresses), and is not using –local

Note 2: if you do an IPv6+IPv4 dual-stack bind on a Linux machine with multiple IPv4 address, connections to IPv4 addresses will not work right on kernels before 3.15, due to missing kernel support for the IPv4-mapped case (some distributions have ported this to earlier kernel versions, though).

手动安装配置

OpenVPN installer 的脚本已经一团麻的，处理的方式我不是很满意，所以最终选择自己配置。

主要参考https://wiki.archlinux.org/index.php/OpenVPN

Archwiki建议证书在另一个主机上生成CA（证书颁发机构），这里将会在同一个主机上进行。

# CA
cd /etc/easy-rsa
export EASYRSA=$(pwd)
easyrsa init-pki
easyrsa build-ca
cp /etc/easy-rsa/pki/ca.crt /etc/openvpn/server/

# server
export servername="server"
openssl dhparam -out /etc/openvpn/server/dh2048.pem 2048
openvpn --genkey --secret /etc/openvpn/server/ta.key
cd /etc/easy-rsa
easyrsa gen-req $servername nopass
cp /etc/easy-rsa/pki/private/$servername.key /etc/openvpn/server/
easyrsa sign-req server $servername
cp /etc/easy-rsa/pki/issued/$servername.crt /etc/openvpn/server/

# config server to use tcp,port,dns,gateway,etc
cp /usr/share/openvpn/examples/server.conf /etc/openvpn/server/server.conf
vi /etc/openvpn/server/server.conf

# for P20 client
mkdir -p /etc/easy-rsa/pki/signed
export client_name="P20"
cd /etc/easy-rsa
easyrsa gen-req $client_name nopass
easyrsa sign-req client $client_name
mv /etc/easy-rsa/pki/issued/$client_name.crt /etc/easy-rsa/pki/signed

# generate client ovpn
export client_name="P20"
ovpngen 172.24.68.231 /etc/openvpn/server/ca.crt /etc/easy-rsa/pki/signed/$client_name.crt /etc/easy-rsa/pki/private/$client_name.key /etc/openvpn/server/ta.key 1194 tcp > $client_name.ovpn
# change cipher to match server config

启动

sudo systemctl enable openvpn-server@server
sudo systemctl start openvpn-server@server

配置Iptables

sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -A INPUT -p tcp -i enp4s0 --dport 1194 -j ACCEPT
sudo iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j MASQUERADE
sudo iptables -A INPUT -i tun+ -j ACCEPT
sudo iptables -A FORWARD -i tun+ -j ACCEPT
sudo iptables -A FORWARD -o tun+ -j ACCEPT

systemd的service文件是可以友好的overide的，参考systemd-drop in files，因此我们通过 sudo systemctl edit openvpn-server@server 的方式来自动处理

/etc/systemd/system/openvpn-server@server.service.d/override.conf
-------------------------------------------------
[Service]
ExecStartPost=/etc/iptables/add-openvpn-rules.sh
ExecStopPost=/etc/iptables/rm-openvpn-rules.sh

/etc/iptables/add-openvpn-rules.sh
-------------------------------------------------
!/bin/sh
iptables -A INPUT -p tcp -i enp4s0 --dport 1194 -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j MASQUERADE
iptables -A INPUT -i tun+ -j ACCEPT
iptables -A FORWARD -i tun+ -j ACCEPT
iptables -A FORWARD -o tun+ -j ACCEPT

/etc/iptables/rm-openvpn-rules.sh
-------------------------------------------------
!/bin/sh
iptables -D INPUT -p tcp -i enp4s0 --dport 1194 -j ACCEPT
iptables -t nat -D POSTROUTING -s 10.8.0.0/24 -j MASQUERADE
iptables -D INPUT -i tun+ -j ACCEPT
iptables -D FORWARD -i tun+ -j ACCEPT
iptables -D FORWARD -o tun+ -j ACCEPT

配置nftables

/etc/nftables.conf
-------------------------------------------------
#!/usr/bin/nft -f
# ipv4/ipv6 Simple & Safe Firewall
# you can find examples in /usr/share/nftables/

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;

    # allow established/related connections
    ct state {established, related} accept

    # early drop of invalid connections
    ct state invalid drop

    # allow from loopback
    iifname lo accept

    # allow icmp
    ip protocol icmp accept
    ip6 nexthdr icmpv6 accept

    # allow ssh
    tcp dport ssh accept

    # allow transmission
    tcp dport 51413 accept

    # dhcp
    #ip protocol udp udp dport 68 accept
    #ip6 nexthdr udp udp dport 546 accept
    udp dport 68 accept
    udp dport 546 accept
		
    # vpn
    jump vpn_input

    # everything else
    reject with icmpx type port-unreachable
  }
  chain forward {
    type filter hook forward priority 0; policy drop;
		jump vpn_forward;
  }
  chain output {
    type filter hook output priority 0;
  }
	chain nat {
		type nat hook postrouting priority 100;
		jump vpn_nat
	}
	
	chain vpn_input {
		tcp dport 1194 accept
	}
	chain vpn_forward {
		meta iifname tun0 accept
		meta oifname tun0 accept
	}
	chain vpn_nat {
		ip saddr 10.8.0.0/24 masquerade
	}
}


# vim:set ts=2 sw=2 et:

Application_i_use_in_linux

2019-09-11T00:00:00+00:00

Fancy Applications

Packages default included in KDE Plasma won’t list here.

‘🗸’ means I am using for now in my computer.

神器

Proxy or VPN

Video downloader

You-Get
youtube-dl
anni

System

Internet

Transmission BitTorrent Client
FileZilla FTP client
Samba
Syncthing continuous file synchronization program

Write

Code

IntelliJ IDEA
PyCharm
Sublime Text
Visual Studio Code

Git

delta A syntax-highlighter for git and diff output
GitKraken Git Client
SmartGit Git Client

Multi media

Command line tool

Monitor

vnstat console-based network traffic monitor
nethogs Net top tool grouping bandwidth per process

Management

Korganizer
KCharSelect

Archlinux

pamac Pamac is a Package Manager based on libalpm with AUR and Appstream support
yay Pacman wrapper and AUR helper written in go.

Web Server

apache
caddy 2 open source web server with automatic HTTPS written in Go

Others

crow-translator
chezmoi dotfile manage and sync

Projects Impressed

webhook is a lightweight incoming webhook server to run shell commands
FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.
jumpserver 开源堡垒机

Misc_records

2019-09-11T00:00:00+00:00

Misc Records

connect to another tty

# Print the file name of the terminal connected  to  standard input.
tty
# connect to it
script -f /dev/tty1

tar

# sudo to keep permission
# pack
sudo tar -cvf mysql.tar mysql
# unpack
sudo tar -xvf www.tar -C .

manually update zsh completion cache
```
rehash
```
zsh completion
```
rm  ~/.zcompdump; compinit
```

Generate random password

# use package pwgen
pwgen 8 # generate with length 8

find file in a certain day

# more to see option `-newerXY` in `man find`
sudo find /var -type d -newermt 2019-08-30 ! -newermt 2019-08-31 -print 2>/dev/null

find dhcp server

sudo nmap --script broadcast-dhcp-discover -e eth0
# for dhcpv6
sudo nmap -6 --script broadcast-dhcp6-discover -e eth0

find out the MAC of a given IP

  sudo arping -I enp4s0 -c1 172.24.68.179

find hostname by IP

# netbios names
nmblookup -A <ip>
# from dns server
nslookup <ip>

Using the KDE Wallet to store ssh key passphrases ssh-add
zip文件名乱码
```
unzip -O cp936 filename
```

run as root in Gui

# in .desktop
# see https://wiki.archlinux.org/index.php/Sudo#kdesu
X-KDE-SubstituteUID=true

visudo导致sudo无法使用时

# 前提是你已经安装相应的 policykit-1-gnome 或者 polkit-kde-agent
/usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1 &
pkexec visudo
# 否则就只能重启进入recovery或者livecd修复

Clean journal
```
journalctl --rotate --vacuum-time=1d
```

rsync with root

rsync -avPpr -e "ssh" --rsync-path="sudo rsync"  fancy@172.24.71.53:/opt/cudnn/cudnn-10.1-linux-x64-v7.5.0.56   /opt/cudnn/

runtime path

# refer https://en.wikipedia.org/wiki/Rpath
# see RPATH or RUNPATH
readelf -d binary-or-library
# see `man chrpath`
chrpath -l binary-or-library

iptables LOG in netns

sysctl -w net.netfilter.nf_log_all_netns=1

Server

List human users

cut -d: -f1,3 /etc/passwd | egrep ':[0-9]{4}$' | cut -d: -f1

disable or enable users

# disable
chage -E 0 <user>
# enable
chage -E -1 <user>

You-Get

# bilibili, 下载  		
you-get <url>
# bilibili, 批量下载  
you-get --playlist <url>

video info
```
ffprobe -hide_banner <video>
```

show videos codec

ffprobe -v error  -select_streams v:0   -show_entries stream=codec_name <url>
  
find . -type f -print  -exec sh -c "ffprobe -v error  -select_streams v:0   -show_entries stream=codec_name '{}' |grep codec|sed -e 's/codec_name=//'" \;

flv to mp4 with loss

# single
ffmpeg -i input.flv -codec copy output.mp4
# loop way
for f in *.flv
do
ffmpeg -i "$f" -codec copy "${f/.flv/.mp4}"
done

NeoVIM

显示行号，显示空白字符

:set nu # show line number
:set list # show space
:noh # turn off highlighting until the next search

Vim < 8.1.1365， Neovim < 0.3.6 中存在漏洞，modeline 在root情况下默认关闭

/etc/xdg/nvim/sysinit.vim
-------------------------------
set modeline

ZSH

^left ^right Backward or forward one word

^u Clear whole line

^k Clear to end of line

^w Delete one word backward

^l Clear screen

^x u Undo last change

^y Yank

pacman

Remove orphan package, repeat the cmd multi times
```
sudo pacman -Ru $(pacman -Qdtq)
```

cgroup

systemd-cgls systemd-cgtop

journal

journalctl -f -u <service>

journalctl -b <boot_number>

Exit stuck sessions

ssh includes the escape character ~ by default. The command ~. closes an open connection and brings you back to the terminal.

Kde_desktop_in_docker

2019-09-10T00:00:00+00:00

kde desktop in docker container

install docker

see this

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/debian \
   $(lsb_release -cs) \
   stable"
sudo apt-get install docker-ce

install nvidia-docker