Introduction

公司发了macbook，用zscaler，用家里网很慢，当然在公司也没快到哪里去。macbook被公司远程管理的，很多限制，这篇文章就是关于我怎么在限制下提高macbook使用体验。

Debug

zscaler会显示一个IP,查IP是香港的，直接Ping，至少25%的掉包。200ms以上的延迟。间断性的甚至100%掉包。
挂了梯子测试，但是ping无法用proxy。这时就考虑是否挂梯子试试。

遇到的问题

梯子有三种

1. 电脑直接装应用，例如clash，这个会出现unverified app的警告，由于电脑被远程控制了，所以没办法安装上去
2. 用路由，但是这种路由的价格都比较贵，我手上的并不能翻，需要买新的
3. 用网关，macbook和我自己的电脑连接同一个wifi,我自己电脑上clash打开”allow Lan”然后用macbook连IP+clash port就行

目前看最合适的是第三种，但是第三种遇到了一点问题：macbook里面固定使用pac文件了，无法改成web或者socket的系统代理。
退一步想看看macbook的input output端口做一个端口转发，例如macbook是走496端口访问外网，那么我用脚本做一个端口转发让macbook访问496的时候，其实访问的是123端口
查詢目前使用中的 port 及 process id
lsof -n -i | grep LISTEN
然后用ssh隧道转发，没有走通
退而求其次，准备改Chrome本身的代理，Chrome本身是跟着系统proxy走的，但是可以添加一个Extension叫做policyswitchyOmega，然后在里面设置socket5代理就行了

效果

效果非常好，比在公司的速度都要快上很多，也就是证明了zscaler是因为服务器设在香港被墙了导致了网速慢，但是用网关梯子这个前提是在一个局域网里面，这时就考虑是否能够在其他地方也加速

设计

macbook -> 腾讯云服务器 -> nginx -> (clash on tencent)

• 为什么要用nginx代理一次
  因为clash只接局域网的数据，所以用nginx代理一次

• 参考文档 & 具体方法

用docker运行clash on linux
https://blog.zzsqwq.cn/posts/how-to-use-clash-on-linux/

1
2
3
4
5
6
7
8

config.yaml
secret: '12345678'
mixed-port: 7890
allow-lan: true
bind-address: '*'
mode: rule
log-level: info
external-controller: 0.0.0.0:9090

其中wget -O config.yaml $your-proxy-url，这里是订阅链接，catnet有问题，返回的是一串String, 用其他的梯子

运行nginx

1
2
3
4
5
6
7
8
9
10
11
12

docker-compose logs | tail -n 20
docker run -it -d -p 1234:1234 -v /data:/etc/nginx/conf.d nginx

stream {
    server {
       listen 80;
       proxy_connect_timeout 1s;
       proxy_timeout 3s;
       proxy_pass 172.17.0.2:7890;
    }
}

在macbook设置好代理后，整条通路没有通，查Log发现是访问到了Nginx但是没有访问到clash docker,怀疑Nginx访问时依旧带了真实地址，或者地址不清晰，被clash拒绝了
在nginx的配置中加一个set_header_IP x_read_IP 类似的东西，并设置为192.168.1.1，依然不可以。

换条路，用其他的转发工具

ncat –sh-exec “ncat 172.17.0.2 7890” -l 80 –keep-open
将80端口收到的流量转发到clash docker上，然后设置macbook访问IP:80端口，这时可以访问到了，连YouTube都能打开
但是有个问题，很慢，一查，发现公司将腾讯云IP地址也做了过滤，导致很慢很慢，也就是搞了半天，路通了，但是很烂

问题不大， 在此记下，方便以后.

Lecture 1 Shell

$PATH

1. all of the path in the machine the shell will look for program
2. which echo: will tell me which echo is gonna run

Permission

• Read Permission on directory
  To list the content of directory

• Write Permission on directory
  rename create or remove files within this directory
  if you have write permission on file but not on it’s directory
  you can empty the file, but you can’t delete it

• Execute permission on directory
  it is known as search permission, are you allow to enter this directory
  If you want to open/read/write a file, you must have execute permisison on all it’s parent directories

• Example one
  if shell didn’t have write permission on file abc
  then even if you run the “sudo echo 500 > abc” you will still get permission denied
  and you can run it as: “echo 500 | sudo tee abc”

single quote & double quote

1
2
3

foo=bar
echo "Value is $foo" //foo will be replaced
echo 'Value is $foo' //foo won't be replaced'

Lecture 2 Scripting

parameter of shell

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

mcd(){
  mkdir -p "$1"
  cd "$1"
}
// $0 is the name of file
// $? will get the error code from previous command, and zero means ok
// $_ will get the last argument of the previous command
// $# the number of parameters
// $$ the process id of this command
// $@ is the whole parameter stored
for file in "$@"; do
  grep foobar "$file" > /dev/null
  if [["$?" -ne 0 ]]; then
    echo "File $file does not have any foobar, adding one"
    echo "# foobar" >> "$file"
  fi 
done

get output of command into variable

1

foo=$(pwd)

file evaluation

1
2
3
4
5
6
7

-b file True if file exists and is a block special file
-c file True if file exists and is a character special file
-d file True if file exists and is a drectory
-e file True if file exists and regardless of type 
-f file True if file exists and is a regular file
-g file True if file exists and its set group ID flag is set 
-h file True if file exists and is a symbolic link

globbing

project* will match project1 project12 project 123 and so on
,and project? will only expand one char

first line of shell

1

#!/usr/bin/env python

to tell the shell what env to trigger

operator

1
2
3
4
5

false || echo "this will be printed"
true || echo "this won't be printed"
// the second one will be excuted only if the first one is false
// and && is the opposite
// use ; it will always print 

Find

1
2
3

find . -path '**/test/*.py' -type f
find . -mtime -1 // been modified in the last day
find . -name "*.tmp" -exec rm {} \

reverse searching

click Ctrl+R and type the command you want to search for,in the same time you can click Ctrl+R too
and prints all the command that matches

string operation

1. Get Length

   1 2 3

   string="abcd" echo ${#string} #print 4 length=${array_name[@]} # print length of array_name

2. Extraction

   1 2	string="runoob is a site" echo ${string:1:4} # print unoo

equation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

-eq检测两个数是否相等，相等则返回true
-ne检测两个数是否不等，不等则返回true
-gt检测左边的是否大于右边的，大于则返回true
-lt检测左边的是否小于右边的，小于则返回true
-ge检测左边的是否大于等于右边的，大于等于则返回true
-le检测左边的是否小于等于右边的，小于等于则返回true
!非运算
-o或运算
-a与运算

# 数值测试
num1=100
num2=200
if test $[num1] -eq $[num2]
then
echo '两个数相等'
else
echo '两个数不相等'
fi
  
# 文件测试
cd /bin
if test -e ./bash
then 
echo '文件存在'
else
echo '文件不存在'
fi

flow control

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

for var in item1 item2 ... itemN
do
command1
command2
commandN
done

int=1
while [ $int -lt 5 ]
do
echo $int
let "int++"  # let是bash中用于计算的工具，变量计算中不需要加上$表示变量
done
# 运行结果
# 1
# 2
# 3
# 4
# 5

echo '输入1到4之间的数字'
echo '你输入的数字为'
read aNum
case $aNum in
1) echo '你选择了1'
;;
2) echo '你选择了2'
;;
3) echo '你选择了3'
;;
4) echo '你选择了4'
;;
*) echo '你没有输入1到4之间的任何数字'
;;
esac

while true
do
echo -n "输入1到5之间的数字"
read aNum
case $aNum in
1|2|3|4|5) echo "你输入的数字为$aNum!"
;;
*) echo "你输入的数字不是1到5之间的！结束"
break
        ;;
    esac
done

This page is about how to build a router, a network interface, and
the TCP protocol

structure

4 Layer
Application / Transport / Network / Link

1. Link(Ethernet / WIFI)

2. Network Layer(IP)
   and it is the only Thin waist
   it is the logical smallest packets which internet break up data into
   and it is format is very easy Three parts: “To” “From” “Data”
   Use Internet Protocol, it makes a best effor to deliver datagrams,
   they can be delivered out of order and corrupted

3. Transport Layer(TCP / UDP)
   transmission control protocol
   user datagram protocol

4. Application
   HTTP
   ascii text / human readable

Lab1 Warm Up

Fetch a Web Page

1
2
3
4
5

telnet cs144.keithw.org http
GET /hello HTTP/1.1
Host: cs144.keithw.org
Connection: close
one more enter

Send yourself an email

pass, it requires standford network

Listening and connecting

1
2

netcat -v -l -p 9090
telnet localhost 9090

Lecture 03

This is a page about Microservice and monitoring

Prometheus

A tool for monitoring,install monitor application,it will send out the log metrics,
and in the UI it can search anything based on rate / frequence and so on
take node for example

1. it monitor the machine side
2. it using job config in yml file to add target

Microservices

1. break down application into microservices based on the business functionalities
2. keep one service doing one specifical jobs
3. self-contained and independent from each other , can be developed deployed scaled separately
4. it is loosely coupled
5. communication use api call / message broker / service mesh

downside of MicroSerivces

1. communication complexity
2. how to monitor so many microservice when it is down

repo

when use microservice you might choose below two Repo type

• monorepo
• polyreop

Near Zero DownTime

1. Blue/Green deployment strategy
   running two server and use load balance

2. Canary release strategy
   slowly roll out the new version to a small group of users

k8s AutoScaling

K8s has three types of autoscaling

HPA(Horizontal Pod Autoscaler)

Pod number autoscaling, depends on metrics-server,adjust to CPU/memory/QPS

VPA(Vertical Pod Autoscaler)

Inside a Pod

CA(cluster Autoscaler)

Node number autoscaling

Source code link

folder structure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

- core #the core part of CA
  - autoscaler.go #interface of the autoscaling
  - scale_down.go            
  - scale_up.go              

- cloudprovider #interface with all cloud provider
  - alicloud
  - aws
  - azure
  - baidu

- expander # different strategy of expand the node
  - factory # design pattern of factory
  - mostpods # use the most pods
  - price # the lowest price
  - priority # according to NodeGroup priority
  - random # random pick
  - waste # the most usage percentage

- simulator #

How k8s manage to increase Node(AWS)?

use awsmanager to increase ASG, the same we increase ASG inside aws console:
IncreaseSize .231

which strategy k8s is using?

scale up 475

quick trying

1. apt install default-jdk
2. write First.java with System(Class).out(field).println(function) Hello World
3. javac First.java,it will generate First.class, it is actually java Byte Code, will been translated by JVM to native code
4. java First to invoke JVM execute the Class file

different Editions

1. Standard Edition(SE) : it is core java platform
2. Enterprise Edition (EE): for very large scale and distributed systems
3. Micro Edition(ME): subset of SE and for mobile

Most important feature

1. Object-Oriented
2. Core Java APIs
3. String / Threads / Database Program

Grammer

Variables Primitive and Reference Types

1
2
3
4

int x = 1;
int y = x;
x = 2;
// y still equal 1

1
2
3

Point p1 = new Point(1, 1);
Point p2 = p1;

Casting

Numbers Strings and Arrays

Read Input

Knowing the Process, it is the very basic thing of all.

Handy Command

1
2
3
4
5
6
7
8

1. ps
2. kill  
3. pgrep # get process name and return pid
4. top # monitor 
5. jobs # show the table that been stopped by Contrl+Z
6. nice -n -20 cmd # run cmd at it's highest priority 
7. renice -n priority -p pid # reset the nice of process
8. lsof | grep filename # check which process hold this file

/proc/ filesystem Explanation

1. fd
   how many file descriptor has been opened by a process

2. maps
   what is the mem map of process

3. status
   check the process state: Sleeping / Running / Stopped / Zombie

4. cwd
   The link the process currently is operating from

5. cmd
   the current cmd process is executing

6. cmdline
   how the process is called

7. environ
   environment variable

一个process本质上就是一段代码加一段数据。

对于process最重要的三个是 page table, kernel stack, run state，
其他两个都好理解，这个kernel stack是process enters kernel（system call or interrupt)的时候，kernel code就在这个stack里面运行，
这个stack user process是无法访问的。

pagetable

地址映射

系统将地址分成了一个一个page大小，一个page大小是2的12次方，也就是4096 bytes
page table是一个柜子，每个柜子512个抽屉，每个抽屉最大空间64bit 也就是8个byte.
也就是一个page table其实也是一个page的大小

在这个系统中，一个process拥有一个page table.也就是它可以拥有512*4096 bytes（2G）大小的虚拟空间

虚拟地址转物理地址

做个比喻

每一个pagetable就像一个有着512个抽屉的柜子，第一个柜子就是root pagetable

每个抽屉的编号就是PTE，盒子里有两张纸条，一张写的是下一个柜子地址编号(ppn)，一张写的是当前这个抽屉的权限，例如这个柜子里面的纸条是否有效，哪些应用能打开看等等(flag)。
有个叫做”satp”的寄存器存了第一个root柜子的地址

整个地址转换流程

你会拿着四张纸条，第一张是一个二进制9bit数，首先根据satp的寄存器地址找到到底是哪个柜子
然后呢根据第一个9bit数算一下是第几个抽屉，例如0x03是第三个抽屉，打开第三个抽屉，看看flag纸条这个抽屉是否
有效，有效的话呢，拿出ppn纸条，找下一个柜子在哪。一直找到第三个柜子。

当我们找到第三个柜子的时候，拿到ppn纸条，和最开始拿到的四张纸条的最后一张拼一起
就是一个真正的物理地址

walk

这是pagetable里面最重要的函数。

它的核心本质上就是走了一遍上面的地址转换，先给你四张纸，你拿四张纸，一个一个去找。
如果没找到就创建一个新的page

mappages

这是pagetable里第二重要的函数

它调用walk将va的地址存进walk返回的PTE里面，也就是你手上有一个物理地址，有一个虚拟地址。
简单比喻就是：你首先用walk函数将va转换，最后返回第三个柜子的一个抽屉，然后你将物理地址放进去

A kernel page table per process(HARD)

背景

Kernel的虚拟地址和物理地址一一对应，也就是Kernel是整块映射的。但是每一个process只有page table，
用于映射user space，如果process调用了System Call，传递了一个user address过去，问题是这个address只有
user page table可以翻译，所以要user page table先翻译成物理地址才可以

做法

1. 每一个user process多保存一个kernel_pagetable
2. allocproc会给每一个process分配空间，在这里面帮kernel_pagetable分配空间
3. kvminit调用了mappages将kernel的虚拟和物理一一对应
4. procinit专门用来给一个一个的kernel stack初始化，这里初始化就是个地址
5. 这样做和之前的区别是什么？不用进入supervisor mode?
   原因可能因为:”Each process has two stacks: a user stack and a kernel stack (p->kstack). When the process is
   executing user instructions, only its user stack is in use, and its kernel stack is empty,When the
   process enters the kernel (for a system call or interrupt), the kernel code executes on the process’s
   kernel stack; while a process is in the kernel, its user stack still contains saved data, but isn’t actively used.”
   也就是说kernel stack和user stack不能同时使用
6. 整个核心流程就是每个proc保存多一个kernel_pagetable.但是这个page table用于存什么东西呢？
7. 每一个process的kernel_pagetable要和global的一样，那么是怎么copy的呢？好像只需要copy root table就行了。用什么copy呢？memmove?
   在哪copy呢？scheduler()?
8. process的kernel_pagetable要留一分map给p->kstack，这个kstack在procinit已经赋值成VA了，那么如果要再映射一次，用walkaddr拿到pa,提供一个新VA然后用mappages
   建立新映射? 这一步失败了，因为walkaddr只能用于拿user space的。

惊天大BUG!!!!!!!!

我写的时候一直出现 panic(‘remap’),重新梳理了好多次，陷入深深自我怀疑。
拖了半天终于对比他人做法发现我是：

1

mappages(kpt, UART0, UART0, PGSIZE, PTE_R | PTE_W);

而mappages的参数是

1

mappages(pagetable, va, sz, pa, perm)

我参数都摆错顺序了呀！！！！！！！！！！！ 什么低级错误缠了了我这么久！！！！！！

函数使用错误

我尝试用kalloc()为proc->kernel_pagetable开辟空间，uvmcreate其实也是调用这个函数为UserSpace开辟空间，但是它有memset()，这一步漏掉可不行

理解错误

1. each per-process kernel page table should be identical to the existing global kernel page table

我一直想的是，user_kernel_table 和 kernel_table 是不是需要用memcpy拷贝，这是因为我对walk并没有真的理解，虽然我将它用非递归的方式重写了，
当我为user_kernel_table 写了kvminit2并调用，这时候后,user_kernel_table和kernel_table都存了一样的信息，因为我用的是一样的va去存储pa

2. Make sure that each process’s kernel page table has a mapping for that process’s kernel stack

has a mapping，意思就是将kernel stack记在user_kernel_table上，我之前就有一个问题，在procinit里面，每个process的kernel stack
记在kernel_page上，lab要我全部拷贝到allocproc，并且给user_kernel_table记，那么还需不需要给kernel_page记了。实际上不需要了，
因为调用kernel stack时，都是用的process自己的user_kernel_table

3. You’ll need a way to free a page table without also freeing the leaf physical memory pages

leaf physical memory pages, 不是指第三个page table,而是指第三个page指向的physical page!!!
每个user kernel page table本质上指向的是同一个地址，你肯定不能free掉啊！

非常重要：kfree和kalloc都是以一个page为最小单位，而且是对齐的，也就是随便拿一个PTE，里面的地址被释放，这个地址所在的整个page都会被释放
所以根本不需要va的最后offset，walk返回的pte通过pte2pa拿到的就是那一块page的开头地址

在xv6里面，先用uvmunmap,调用walk找到所有的mapping释放掉，然后再调用freewalk释放掉所有的pagetable

复盘

9/1完成speedup开始做2020的per process kernel page，每个星期算两个小时，也做了近十个小时了。
实际操作中，我被很多Debug卡住了，例如我尝试写kvminit，用mappages的时候，没看到和kvmmap的参数的顺序有差别，再比方说，我全部写完后，
一直有kvmap的panic错误，但是我只是去检查我自己的代码，其实这时需要改动kvmap那里，将kernel_pagetable改为user_kernel_pagetable就行了

为什么我会遇到这种坎坷呢，核心还是我没有完全的熟透这个pagetable的每一个细节，具体行动上，我并没有严格按照Lab的instruction 将要求的vm.c全部代码看一遍，
过一遍，其实如果我将每个代码重新写一下，就像写walk()函数一样，后面就会清晰很多，而不是遇到问题又重新去看。

遇到问题再查资料也可以，但问题是，会更痛苦，更曲折，如果已经有instruction，是可以尝试follow一下的。

很多时候，book里面也已经给了所有信息，例如我遇到的问题书上其实已经说了“Functions starting with kvm manipulate
the kernel page table; functions starting with uvm manipulate a user page table”。 但是当时看的时候并没当回事，所以也不用妄自菲薄，
学而时习之，过了一遍再来复习，肯定又有完全不一样的理解。

Simplify (HARD)

背景

上一个task已经创建了一个user kernel page table,但是依旧没有能够做到调用kernel直接dereference user pointers，因为这时user_kernel_page_table
只存了kernel的map，还有user data的map没做

task已经提供了copyin_new，这里去掉了将va转为pa，然后再memmove，直接从src(va)拷贝到dest(pa)，

以下为错误理解！！！
等于说src(va)这里就是真正的物理地址。user_data和kernel一样都是va=pa，map在user_kernel_pgtb上

所有的user_data都存在了user page table里面，那么要”add user mappings to process’s kernel page table” 应该要遍历user_page_table
拿到所有的物理地址，并且映射到user_kernel_page_table(va=pa)
于是函数逻辑就是：先遍历user page table,拿到了物理地址，然后将物理地址和一样数值的va一对一map在user kernel page table

！！！以上为错误理解

解析

参考了其他人的解法，发现是将user_page_table从0到proc->sz全部遍历一边，取出里面的pa，映射到user_kernel_pgtb.
copyin_new()后，kernel拿到的是user_kernel_pgtb可以解析的虚拟地址。之前拿到的是copyin用walk解析后的实际地址。

我最大的疑问是，kernel是什么做dereference呢，walk()？

是paging hardware做的解析，你只要提供pagetable它就能解析，有一句很重要的话没看到：walk mimics the RISC-V paging hardware
walk只是用来模拟page hardware的

但依旧有个疑问,kernel在什么情况下做的

读到一句话：copyout and copyin copy data to and from user virtual addresses provided as
system call arguments; they are in vm.c because they need to explicitly translate those addresses
in order to find the corresponding physical memory
也就是copyin的explicitly translate是不应该的，也只要satp握着的pgtb含有这个va，它就会自动转换

参考下面这一段
Early in the boot sequence, main calls kvminit (kernel/vm.c:22) to create the kernel’s page
table. This call occurs before xv6 has enabled paging on the RISC-V, so addresses refer directly to
physical memory

对于user page table来说，它的虚拟地址是连续的

uvmalloc代码揭示了，每一次growproc，都是从oldsz -> newsz, 地址是连续的

Detecting which pages have been accessed (HARD)

几个问题：

1. 怎么从Manual里选到底用哪个数值作为PTE_A?
   这里在riscv-privileged里面有图表

   

2. 在哪里设置PTE_A?
   Risc-V自己设置的 Each leaf PTE contains an accessed (A) and dirty (D) bit. The A bit indicates the virtual page has
   been read, written, or fetched from since the last time the A bit was cleared. The D bit indicates
   the virtual page has been written since the last time the D bit was cleared

Trap

只有三种情况会触发Trap

1. System call, user executes the ecall
2. Exception
3. Device interrupt

Trap的特点

1. transparent
2. Three distinct case trap: user space / kernel space /timer interrupt

Trap 流程

1. hardware action by cpu, Vector Assembly
2. C Trap handler
3. system call & driver

Kernel will do instead of CPU

1. switch to kernel page table
2. switch to a stack in the kernel
3. save register

寄存器说明

这些寄存器只能在supervisor mode读取和存储

• stvec: Trap handler的地址
• sepc: 存PC指针的地方，trap完后，调用sret重新将spec写入pc
• scause: 存放为什么调用trap
• sscratch: ?
• sstatus: SIE为1才会允许device interrupts, SSP用于指明trap来自哪个Mode
• stval: 存储无法被translate的地址

Trap from user space流程(with register)

1. 如果是device interrupt 且 sstatus的SIE未set，Stop
2. 清掉SIE bit
3. copy pc sepc 存储pc指针
4. 存储当前mode在sstatus
5. set scause number
6. 换到supervisor mode
7. pc = stvec
8. 开始执行

Trap from user space流程（with code)

1. trampoline.S/uservec
   汇编，stvec指向处，这个user trap和kernel trap都会用到，但是RISC-V的机制不会改satp从user_pgtb到kernel_pgtb
   所以uservec需要同时map同一地址的user_pgtb和kernel_pgtb
   开头这段代码会将寄存器存在trampframe这片地址（每一个user_pgtb都存了，在地址TRAPEFRAME）
   然后从tramframe的最前面提取存好的kernel_pgtb等数据进入下一个函数

2. trap.c/usertrap
   首先就将stvec换成kernelvec，也就是这个trap可以继续被打断？
   判断是哪种trap System Call / Devices / execption
   其中exception会杀掉错误process

3. trap.c/usertrapret
   这个就是重新恢复环境，stvec恢复uservec，sepc恢复到pc等等，然后userret回去，这里注意userret会switch page table

4. trampoline.S/userret
   和uservec对比完全反过来，上一层调用的时候传入两个参数分别是Trapframe和pagetable

Trap from kernel流程（with code)

1. kernelvec.S
   存一些寄存器之类的

2. trap.c/kerneltrap
   如果是device就call devintr,如果是exeption就直接panic
   如果是timer interrupt，就call yield让CPU

System Call(with Code)

前面的lab已经添加过system call了，现在来真实了解一下system call怎么调用的

1. kernel/syscall.c
   有一个数组，存储各个syscall地址,从a7寄存器取offset，然后直接调用了对应的sys_function.
   然后将返回值放在p->trapframe->a0

stvec 是什么寄存器，为什么很重要？

The kernel writes the address of its trap handler here; the RISC-V jumps here to
handle a trap。只要进入trap，就会调用这个地址的函数。
如果不这么做，那么Then a trap could switch to supervisor mode while still running user instructions

trapline是什么,trapframe又是什么

trampline是一段汇编代码，里面用于将硬件的Trap信号导到C代码上，trampoline这个page在每个pagetable都有，而且都是同样的virtual address
方便切换时好继续执行
trapframe是一段内存空间，process拥有的，用于在Trap的时候存寄存器等值
这些东西的产生，核心是因为user_page里面没有kernel memory，如果用Simplify (HARD)的方法，就可以去掉这些东西，并且更高效

COW(copy on write)

Fork后child不会复制parent的PA，而是指向它，到真的要用的时候，才通过page fault触发并创建新的page,
为了区分COW和非法访问，PTE里面存了一个bit用来区分。同时要有一个ref count,这个存在内存里，用来说明此page是否没人引用了。

lazy allocation

sbrk()用于开辟内存，而且是eager allocation,但是呢：applications often ask for more memory than they need
所以真实情况是，最开始并不给application真实的空间，等真的开始访问对应地址的时候，会触发page fault trap，这时kernel才帮忙创建并分配
page table地址

zero fill on demand

全局变量都置为0，这些变量其实都指向了同一个pa,然后都只有只读权限，当尝试写的时候，会触发page fault并且开新空间

Demand paging

这个是在Exec的时候，做lazy allocation

Lab Trap : Backtrace(moderate)

问题1：offset(-8)，这个负八单位是多少？1Byte? 1bit? 4Byte?
这里是一个Byte，系统都是一个Byte一个Byte的操作。

问题2：怎么判断什么时候停止往回回溯backtrace?
每个kernel stack都是一页,所以拿到第一个framepointer的地址，然后通过PGROUNDDOWN和UP分别拿到栈头和栈尾，通过判断地址是否是在这个page下就可知道什么时候停下了。

just copy and use it

word counts

1

grep  filename | wc -l

shell if

1
2
3
4
5
6
7

aflag=$(grep filename | wc -l)
if[[$aflag -eq 1]]
then
sed xxxx
else
echo "done"
fi

bash Grep & find & env & str

1
2
3
4
5

for pat in `find . -name "log4j*"` ; do unzip -p $package META-INF/MANIFEST.MF | grep Implementation-Version; done; // dump and grep key value
${str::-n} // delete last n byte
kubectl get pvc -n pt | grep zookeeper | head -n 100 | awk '{print $1}' | xargs kubectl delte pvc -n pt
bash JAVA_HOME=xxx scripts // activate env for certain command
aws --profile=saml cloudtrail lookup-events --lookup-aatributes xxx | grep -o -P '.{1,3}xxx.{0,200}' // grep and print certain length of content

run docker as a root user

1

docker container exec -u 0 -it mycontainer bash

pip install with Cert

1

pip install requests --cert *.pem --trusted-host *.xxx.com --index-url https://username:password@repolink

CPU usage

1
2

println "uptime".execute().text
println "mpstat -P ALL".execute

Get current available physical on windows

systeminfo | find “Available Physical Memory”

k8s command

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

export http_proxy=
unset http_proxy

helm list -n namespace
helm install --dry-run --debug ./foldername --generate-name
helm install ./foldername --create-namespace -n present-test --generate-name 
helm delete xxx -n namespace

kubectl get pods -n namespace
kubectl describe pod pod_name -n namespace
kubectl exec -it xxx -n namespace -- sh
kubectl get pvc
kubectl delete pvc xxx
kubectl create secret generic xxx_secret --from-literal=DB_USER=*** --from-literal=DB_PASSWORD='***' -n namespace
kubectl get sa(service Account)
kubectl describe sa

special char

u”\u00b0” = °

C Pointer

1
2
3
4
5
6
7
8
9
10
11

int a[10];
int *pa;
pa = &a[0];
pa == a;
a[i] == *(a+i)
pa[i] == *(pa+i)
pa++ legal
a++ illegal
The
meaning of ``adding 1 to a pointer,'' and by extension, all pointer arithmetic, is that pa+1 points
to the next object

Sometimes when using the Opening source tools, i will encounter weird bug or feature, most of it because i am using it in a unique way.

1. NPM CONFIG Auth

when npm is using the private registry, like the Nexus, it will require Token for npm to access it

generally we can use in this way

1
2
3
4

export NPM_CONFIG_STRICT_SSL=false
export NPM_CONFIG_ALWAYS_AUTH=true
export NPM_CONFIG__AUTH=xxxx
npm ping --registry https://xxx.xxx.xxx/

you might have noticed that for the AUTH env it has two underscore instead of one
actually it is a feature of NPM, and it is merged into the tool since 2014
refer link: https://github.com/npm/cli/issues/3985

Solution:
Use NPM_CONFIG__AUTH

2. Docker Config

When many users are using docker on Jenkins， we can restrict user by locking the Global Config

1
2
3
4
5
6
7

touch /var/log/gitconfig
ln -s var/log/gitconfig /home/user/.gitconfig

touch /var/log/dockerconfig.json
mkdir /home/user/.docker
chmod 600 /home/jenbld/.docker 
ln -s /var/log/dockerconfig.json /home/user/.docker/config.json

and we can export home=${workspace}, to use the local config under workspace
There comes with a plugin problem:

The “Docker Build and Publish plugin” will fail on “Failed to deserialize Rejected”.

see the changes:
https://github.com/moby/moby/commit/18c9b6c6455f116ae59cde8544413b3d7d294a5e
change from .dockercfg to config.json

Solution:
upgrade your plugin or use the deprecated config .dockercfg

3. Terraform init

when you run terraform init on laptop you might encounter

1
2
3
4
5
6
7
8

╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/google: could not connect to registry.terraform.io: Failed to
│ request discovery document: Get
│ "https://registry.terraform.io/.well-known/terraform.json": proxyconnect
│ tcp: EOF

it might because the proxy issue, set on the vpn and set proxy
export https_proxy=’http://127.0.0.1:7890' then will be fine

when you encounter “Error: storage.NewClient() failed: dialing: google: could not find default credentials”
during terraform init, just add the credentials = “../key.json” to backend.tf to fix it

Editing

为什么选外汇

1. 它超级大盘，EURUSD这种货币对几乎无人能操纵
2. 外汇的信息获取非常充足
3. 极小的入市成本，0.01手，一美金都能做交易

Why Forex

1. It is Huge, No one can manipulate it Such as EURUSD
2. It is all open information for Forex
3. extremely Low investment cost, 0.01lot you can start your journey

平台和工具

• Python 3.8.8
• MetaTrade5
• MetaTrade的Python Module
• 一个外汇佣商的账号

Platform and Tools

• Python 3.8.8
• MetaTrade5
• MetaTrade的Python Module
• A Forex Account Such as FTXM

背景知识

这里以EURUSD为介绍
这个货币对叫做欧元/美元，在交易系统上有两个价格 卖价（bid）买价（ask）
这个都是对于交易者来说的，也就是ask就是市场上你能买到的最低价,bid就是市场上你能卖的最高价
ask >= bid，点差就是ask-bid，点差是可以为0的，例如EURUSD行情比较平缓时，点差为0

Background

Take EURUSD as introduction
It is called EUR/USD pair, on market, it has two price, bid and ask
it is all refer to trader, ask is the lowest price you can ask to buy
bid is the highest price, you can sell
ask >= bid, Spread = ask - bid, it can be zero

怎么赚钱

外汇的波动非常小，需要用杠杆放大，交易所会借钱给你，帮你放大波动用于赚钱
例如 你有10美元，用2000倍杠杆，此时你如果在1.01254这个价位做空0.01手，交易所会借给你1000美元，你需要用1000/2000 = 0.5美元作为保证金
此时如果价格从1.01254波动到1.01200,那么你就用了0.5美金赚到了(1.01254 - 1.01200)*1000 = 0.54美金
(这里没有计算点差费用和手续费，用于做一个简单说明)

这个赚钱吗风险怎样

依旧以EURUSD为例，它的每小时波动标准差为0.000705641339641201,也就是如果你用2000的杠杆，用0.5美元做保证金，如果赌对了，平均能赚到0.7美元
以三个月内的小时线做了分布图

常见交易方法

1. MACD/RSI/Bollinger Bands/EMA
   这一部分我用Python和EA都跑过，模拟盘没有赚到钱

2. 滚仓
   先开一个单，如果盈利了就继续加仓，这个有赚到过一点点

3. 马丁策略
   这个是非常著名的策略，胜率非常高，但是一旦失败，就爆仓，网上有非常多的教程也有现成的
   我贴一下我的模拟盘净值,分别是失败和成功的

参考代码(Demo code)

https://github.com/presentzeng/forex

参考文档(Demo code)

https://www.mql5.com/zh/docs/constants/structures/mqltraderequest

It is a Note about The MIT course 6.S081. arranged by following it’s lab order
https://pdos.csail.mit.edu/6.S081/2021/index.html
https://pdos.csail.mit.edu/6.S081/2021/labs/util.html

Lab1 Utilities

Key Points

1. The kernel uses the hardware protection mechanisms to ensure that each process executing user space can access only its own memory
2. Fork creates a new process, called the child process, with exactly the same memory contents as the calling process

pingpong

• what is the difference of two code block

  1 2 3

  int fd[2]; pid=fork(); pipe(fd);

  1 2 3

  pid=fork(); int fd[2]; pipe(fd);

  all have 4 fd genereated
  but the first one two process’s fd have no connection.
  and the second one has

• remember to exit at every child process, otherwise it will execute the code below

xargs

• what is char* a[10]?

  format a = {char* char* …. char*}

• exec(command, argv)

  the argv[0] should be not be filled with parameter but with command name

• there are two types format: “123\n456” and 123 456, how we deal with it.

  we can devide with space of \n and treate “ as one of the char, delete it at the last step

• we should clear all the uninitial string otherwise we will get random value like “" which will cause a lot of confusion to us
  
• if you wanna pass char par[10][10] as a parameter in function you should write prototype like below

  func(char * par[10][10]), that means you should write out the dimension of it

• what is the difference

  char *par[] = {}; char * par[10] = {}; char *par[10], the former two are empty array, the last one is the right defination

• why we should fork and call exec

  exec system call will replace the the calling process, that is why

• what is the difference between strlen and sizeof
  pls Do use sizeof with memset
• we can compare char with == but we should compare string by using strcmp
• “” is a string ‘’ is a char
• how to use pointer array properly

  1 2 3 4 5

  char *pass_par[10]; char * test = "xargs"; pass_par[0] = test; printf("%s\n", pass_par[0]);

Lab2 System calls

Mode

mode

Machine
mode is mostly intended for configuring a computer

supervisor mode is for executing privileged instructions: for example, enabling and disabling interrupts, reading and writing the register that holds the address of a page
table

CPUs provide a special instruction that switches the CPU from user mode
to supervisor mode. (RISC-V provides
the ecall)

xv6 is monolithic kernel, with all kernel in one mode

Kernel

the hardware only uses the low 39 bits when looking up virtual addresses in page tables; and
xv6 only uses 38 of those 39 bits. Thus, the maximum address is 2
38 − 1 = 0x3fffffffff

Each process has two stacks: a user stack and a kernel stack

In Entry.S it’s _entry set up a stack so that xv6 can run C code start.c

System call tracing

Preparation:

1. read at least twice the instructions
2. add print command to check the execute order of each function
3. you can check how the kill pass all the argument

Note:

1. The most important part is how to pass the argument to the system call
2. To read the document, you will know the argument is stored in which Register, and you can use function argint
   to “Fetch the nth 32-bit system call argument”

Sysinfo

Preparation:

1. the trick part about the compilation is pls use sys_sysinfo instead of sys_info
2. A process’s most important pieces of kernel state are its page table, its kernel
   stack, and its run state.
3. sysinfotest -> kernel/sysproc.c -> kernel/kalloc*proc

Note:

1. The most important part is to understand the function copyout
2. for counting proc, we should caculate all the proc that not in UNUSED mode
3. for counting mem, we should times the page size

Lab3 Page Table

Note:

1. VA and PA is translated by map
2. MMU won’t store any map
3. kernel will write the map value into the satp register
4. page is 4 KB in the Risk V, which required
5. for satp, it is divided into three pieces 25+27+12, 12 is the byte within the page, index is for the map to translate to physical page
6. every process keeps it’s own page table
7. Kernel is using “direct mapping”

The Steps to translate the virtual to physical

1. the virtual address is divided into PTE-selector and offset, for example 9+9+9(PTE-selector) + 12(Offset)
2. for the first page table it is satp + value(first 9 bit), in it we can get the PPN of next page(it’s physical address)

concepts of each key defination

1. pte = page table entries, it is actually an address
2. ppn = physical table number
3. pagetable is an array string like below
   [*pte1, *pte2, *pte3 …. *pte512]
4. Each PTE contains a 44-bit physical page
   number (PPN) and some flags
5. Sv39 Risc-V only uses bottom 27 bit for virtual address, that means 27 power of 2 is the max number of PTES
6. each pte actually has 64 bit(8 byte), but it only uses the bottom 54 bits
7. each page table has 512 ptes, in total it is 4096 byte
8. *pte = ppn(44) + flag(12), pte is an address it is value is ppn+flag
9. physical address is 56 bit
   10 last physical address(56) = ppn(44) + Virtual offset(12)
10. the middle physical(56)= ppn(44)+ empty zero(12)
11. every VA will has a unique three level pagetable chain
12. walk function will create a link from VA to pte based on one certain first level pagetable
13. mappages will store the pa to the last pagetable pte

rewrite Walk in non recursive way

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

#define BIT2OFFSET(level, va)  ((va >> 12) >> level*9) & 0x1ff
#define PTETOPA(pte)  ((pte >> 10) << 12)
#define PATOPTE(pa) (((uint64)pa >> 12) << 10)
pte_t *
walk(pagetable_t pagetable, uint64 va, int alloc)
{
    if(va > MAXVA)
        panic("walk");

    // get pte from first level page
    uint64 *first_pte = &pagetable[BIT2OFFSET(2, va)];
    // if valid first page
    if((*first_pte & 0x1) != 0) //valid
    {
        //get next pagetable address
        pagetable = (uint64 *)PTETOPA(*first_pte);
        uint64 *second_pte = &pagetable[BIT2OFFSET(1, va)];
        if((*second_pte & 0x1) != 0)
        {
            pagetable = (uint64 *)PTETOPA(*second_pte);
            return &pagetable[PX(0, va)];
        }
        else //second lvl page create
        {
            //uint64 *second_pte = (uint64*)pagetable[BIT2OFFSET(1, va)];
            if(!alloc || ((pagetable = kalloc()) == 0))
            {
               return 0;
            }
            memset(pagetable, 0, PGSIZE);
            *second_pte = PA2PTE(pagetable) | PTE_V;
            // once created, go to next lvl
            pagetable = (uint64 *)PTETOPA(*second_pte);
            //uint64 *third_pte = &pagetable[BIT2OFFSET(0, va)];
            return (pte_t*)PTETOPA(*second_pte);
            //return &pagetable[PX(0, va)];
        }
    }
    else //if first is not valid
    {
        if(!alloc || ((pagetable = kalloc()) == 0))
        {
           return 0;
        }
        else
        {
            memset(pagetable, 0, PGSIZE);
            *first_pte = PA2PTE(pagetable) | PTE_V;
        }
        pagetable = (uint64 *)PTETOPA(*first_pte);
        uint64 *second_pte = &pagetable[BIT2OFFSET(1, va)];
        if((*second_pte & 0x1) != 0)
        {
            pagetable = (uint64 *)PTETOPA(*second_pte);
            //uint64 *third_pte = &pagetable[BIT2OFFSET(0, va)];
            return &pagetable[PX(0, va)];
        }
        else //second lvl page create
        {
            uint64 *second_pte = &pagetable[BIT2OFFSET(1, va)];
            if(!alloc || ((pagetable = kalloc()) == 0))
            {
               return 0;
            }
            else
            {
                memset(pagetable, 0, PGSIZE);
                *second_pte = PA2PTE(pagetable) | PTE_V;
            }
            // once created, go to next lvl
            pagetable = (uint64 *)PTETOPA(*second_pte);
            //uint64 *third_pte = &pagetable[BIT2OFFSET(0, va)];
            return &pagetable[PX(0, va)];
        }
    }
}

Free process’s memory

1. it should free it’s page table and the physical memory it refers to
2. Is only the leaf pagetable has the PTE_R PTE_W flag?

Print a page table(Task)

1. pretty easy, we can just copy and modify freewalk to print

Speed up system calls(Task)

preparation

1. check the user side of ugetpid, to understand how the app gonna use the interface

   1 2 3 4 5

   int ugetpid(void) { struct usyscall *u = (struct usyscall *)USYSCALL; return u->pid; }

   the function directly convert the USYSCALL Address to struct usyscall, and return the stored value of pid

2. To do this, we can kalloc a page and set it readable to userspace, Then in Kernel space we can directly store the pid into it

3. USYSCALL is a virtual address

4. mappages’s main purpose is to map va to pa, it use walk() to create a pte for the va, and store the pa into that pte

5. Each process has a separate page table(First Level), and when xv6 switches between processes, it also changes
   page tables

6. we should create a new pagetable specify for this task, that means each process has two pagetable,
   one is the original, other is the store pid one

Plan

1. you know the Physical Address
2. you know the VA address
3. you can use mappages to map va to physical
4. then you can init this value all done

A kernel page table per process(Task)

1. modify struct proc add kernel field
2. every User process will keep an extra identical kernel_pagetable
3. at before, all the process share the same kernel page table

Simplify (HARD)

1. this feature should use the User_kernel_page_table

when did it translated

copyout and copyin copy data to and from user virtual addresses provided as
system call arguments; they are in vm.c because they need to explicitly translate those addresses in order to find the corresponding physical memory
and after the user_kernel_page_table setup, it will translate by hardware

how this Virtual address range?

From 0 to bottom

Detecting which pages have been accessed (HARD)

how to determine PTE_A?

Inside riscv-privileged.pdf

where to set PTE_A

Risc-V done by itself: Each leaf PTE contains an accessed (A) and dirty (D) bit. The A bit indicates the virtual page has
been read, written, or fetched from since the last time the A bit was cleared. The D bit indicates
the virtual page has been written since the last time the D bit was cleared

Trap

Three main way to trigger Trap

1.System call, user executes the ecall

2.Exception

3.Device interrupt

Trap’s feature

1.transparent

2.Three distinct case trap: user space / kernel space /timer interrupt

Trap procedure

1.hardware action, Vector Assembly

2.C Trap handler

3.system call & drver

Kernel will do instead of CPU

1. switch to kernel page table

2.switch to a stack in the kernel

3. save register

What is stvec

The kernel writes the address of its trap handler here; the RISC-V jumps here to
handle a trap
Without stvec Then a trap could switch to supervisor mode while still running user instructions

What is trampoline

as the name it is, it is a trampline code for user process, it has all the important function such as uservec.
and it is stored at every pages

先给结论：网络瓶颈用线程，cpu瓶颈用进程

运行条件

16核 ubuntu python3+

闲置情况

Python单核跑满cpu

1
2
3
4
5
6

def lp(): 
    while 1: 
        a = 1 
        b = 2 
        c = b + a 
lp()

三线程Python

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

import time 

def add_num(): 
    while 1: 
        a = 1 
        b = 2 
        c = b + a 
 
if __name__ == '__main__': 
    t1 = Thread(target=add_num) 
    t2 = Thread(target=add_num) 
    t3 = Thread(target=add_num) 
    t3.start() 
    t1.start() 
    t2.start() 
    time.sleep(100)

试试开一个线程，是可以跑满100的

两线程

相加为100%

三进程

可以看到是07 11 13三个核在负载，全部跑满100%

两线程C代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

#include <stdio.h> 
#include <stdlib.h> 
#include <pthread.h> 
 
void print_message_function (void *ptr); 
 
int main() 
{ 
    int tmp1, tmp2; 
    void *retval; 
    pthread_t thread1, thread2; 
    char *message1 = "thread1"; 
    char *message2 = "thread2"; 
 
    int ret_thrd1, ret_thrd2; 
 
    ret_thrd1 = pthread_create(&thread1, NULL, (void *)&print_message_function, (void *) message1); 
    ret_thrd2 = pthread_create(&thread2, NULL, (void *)&print_message_function, (void *) message2); 
 
    // 线程创建成功，返回0,失败返回失败号 
    if (ret_thrd1 != 0) { 
        printf("线程1创建失败\n"); 
    } else { 
        printf("线程1创建成功\n"); 
    } 
 
    if (ret_thrd2 != 0) { 
        printf("线程2创建失败\n"); 
    } else { 
        printf("线程2创建成功\n"); 
    } 
 
    //同样，pthread_join的返回值成功为0 
    tmp1 = pthread_join(thread1, &retval); 
    printf("thread1 return value(retval) is %d\n", (int)retval); 
    printf("thread1 return value(tmp) is %d\n", tmp1); 
    if (tmp1 != 0) { 
        printf("cannot join with thread1\n"); 
    } 
    printf("thread1 end\n"); 
 
    tmp2 = pthread_join(thread1, &retval); 
    printf("thread2 return value(retval) is %d\n", (int)retval); 
    printf("thread2 return value(tmp) is %d\n", tmp1); 
    if (tmp2 != 0) { 
        printf("cannot join with thread2\n"); 
    } 
    printf("thread2 end\n"); 
 
} 
 
void print_message_function( void *ptr ) { 
    int i = 0; 
    while(1) 
    { 
        i++; 
    } 
} 

C的多线程没有这个问题

​

现象：

python的线程只能用到一个核。

结论

个人查资料后的判断结论是：python用到了GIL锁导致python线程只能用到一个核
Reference :
https://timber.io/blog/multiprocessing-vs-multithreading-in-python-what-you-need-to-know/

Editing

Input validation

If we don’t validate input properly, Hacker can alter control flow, change the code execution, or get the Important file informations

e.g:
CWE-790
CWE-116
CWE-138

Defense

1. Consider all the aspect:Value Type / Length / Format / Expired Time / Effect Scope and so on
2. Assume all input is malicious, Only accept the thing that we know, That means we only set the whiltlist

Initial

“Right at this moment, I was born” – Millennium Bug