ingbyr

Java8 Spring MVC 5 项目迁移 Java21 SpringBoot 3.5 记录

Mon, 15 Dec 2025 16:26:11 +0800

总结了迁移 JDK8 Spring MVC 5 项目到 JDK21 SpringBoot 3.5 的一些处理方式与步骤。

前置检查

检查与HTTP相关的第三方依赖和私有依赖，是否已经有支持Jakarta的版本，没有则需要克隆源码自行修改后发布到私有仓库。
检查项目中是否使用了Oracle相关的代码库，删除或使用其他开源库替代。

升级依赖

修改版本控制

修改pom中的 parent 为 spring-boot-starter-parent，将大部分依赖版本号管理委托至 spring boot，降低不同版本不兼容风险，例如：


<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.5.8</version>
</parent>

添加版本控制后，需要将旧版本依赖的版本号全部删除，执行 maven dependency:resolve 查看哪些依赖版本号缺失，并添加缺失的依赖版本号。

替换 spring-boot-starter 系列

一些依赖组件提供了原生的 spring-boot-starter，最好直接使用starter依赖替换旧的版本管理，降低迁移成本，例如：

spring-boot-starter-web
spring-boot-starter-test
spring-boot-starter-validation
mybatis-spring-boot-starter
其他的 starter 可以在官网查看

排除重复依赖

有一些无法升级或老旧的第三方包，可能兼容Java21，但自身会带来旧版本Spring或其他的依赖，此时使用命令 mvn dependency:tree -Dverbose检查依赖传递情况，找到重复依赖并排除，例如下面需要排除 commons-fileupload，转而使用新版本 fileupload2：


<dependencyManagement>
    <dependency>
        <groupId>com.bstek.ureport</groupId>
        <artifactId>ureport2-console</artifactId>
        <version>${ureport.version}</version>
        <exclusions>
            <exclusion>
                <groupId>commons-fileupload</groupId>
                <artifactId>commons-fileupload</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-fileupload2-jakarta-servlet6</artifactId>
        <version>${commons-fileupload.version}</version>
    </dependency>
</dependencyManagement>

定制化停止更新的依赖

一些老旧的依赖可能在发布 Jakarta 前就已停更，此时需要获取到源码工程，修改源码兼容Java21和Spring Boot 3.5，并发布到私有仓库。例如本次迁移中项目中使用Ureport2已经停更，且不支持Java21 Jakarta，修改源码大致步骤：

Ureport源码中添加 spring-boot parent，删除由spring-boot管理的版本号。
修改或升级Ureport中的三方依赖。
修改源码以兼容新版的 Jakarta命名空间，操作同后续内容中的项目源码修改

升级 maven 插件

一些maven插件在Java21环境下也需要升级，可以一并升级到最新版本，这个一般不会有兼容性问题。查看具体的版本传递可以使用插件


<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>flatten-maven-plugin</artifactId>
    <version>1.7.3</version>
    <configuration>
    </configuration>
    <executions>
        <!-- enable flattening -->
        <execution>
            <id>flatten</id>
            <phase>process-resources</phase>
            <goals>
                <goal>flatten</goal>
            </goals>
        </execution>
        <!-- ensure proper cleanup -->
        <execution>
            <id>flatten.clean</id>
            <goals>
                <goal>clean</goal>
            </goals>
        </execution>
    </executions>
</plugin>

项目代码适配

javax 迁移 Jakarta 命名空间

使用IDEA自带工具批量替换 javax 命名空间为 jakarta，菜单位于 Refactoring -> Migrate Packages and Classes -> Java EE to Jakarta EE，本质上是替换代码文件中的字符串，可以新建自己的规则批量替换其他命名空间。

迁移 spring xml 配置为代码配置方式

旧项目使用了spring xml 配置，需要将配置文件迁移为代码配置方式，并使用 @Configuration 注解声明为配置类，这个需要按项目具体需求进行修改，注意最好不要使用xml与代码混合使用，建议全部迁移为代码配置方式。

旧配置文件适配 spring boot 3

项目中配置文件为多个properties文件，使用springboot3中新引入的配置文件导入，由spring合并配置，例如在application.yaml中引入旧项目配置文件base.properties、biz.properties等：

spring:
  config:
    import:
      - classpath:base.properties
      - classpath:biz.properties

项目中如果使用了自定义的配置读取类，需要将配置文件注入到Spring Environment中，例如:

public class MyPropertySourcePostProcessor implements EnvironmentPostProcessor {
    private static MapPropertySource mergeMyPropertySource() {
        Properties properties = Global.PROPERTIES_LOADER.getProperties();
        Map<String, Object> map = new HashMap<>();
        properties.forEach((k, v) -> map.put(k.toString(), Global.getConfig(k.toString())));
        MapPropertySource source = new MapPropertySource("myProperties", map);
        return source;
    }

    @Override
    public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
        environment.getPropertySources()
                .addFirst(mergeMyPropertySource());
    }
}

编译测试

完成迁移后，需要编译测试项目，确保项目正常启动，并检查是否有兼容性问题。具体而言要完成回归测试、集成测试和性能测试，同时部署后进行渗透漏洞扫描，确保当前依赖下无重大安全隐患。

一次JVM堆外内存问题排查

Thu, 14 Mar 2024 10:24:03 +0800

记录一下JVM堆外内存泄露排查的过程

出现的问题

线上运行的Java应用在持续运行数小时后会因堆外内存不足而终止，查看日志可以看到类似输出：

先尝试了使用手动释放cached memory，发现还有大量内存未释放，基本排除了操作系统内存释放策略配置错误的问题

开启JVM堆外内存追踪

在启动Java应用的参数中添加 -XX:NativeMemoryTracking=detail ，重启应用后运行一段时间，使用 jcmd $pid VM.native_memory summary 查看内存分配情况，可以看到类似输出

与 top 命令输出的 RSS 比对发现远小于其值，此时基本可以确定是 JNI 调用 Native Code 导致的堆外内存泄露

查看未释放的堆外内存内容

先使用 pmp -x $pid 查看内存分布情况，对比前后两次的内容发现一直会申请一些64MB的内存块，并且始终处于未释放状态
查看 /proc/$pid/smaps 或 /proc/$pid/maps 找到对应内存起始和终止地址
使用 gdb -pid $pid attach 到问题进程， dump memory mem.bin 起始地址终止地址 dump 出对应的多块内存块。
由于此次Java应用中存在中文字符，因此使用 strings -eS mem.bin 查看可显示的字符内容

经查看多个内存块后发现出现频率高的字符串没有明显规律，需要进一步排查

监控内存分配

本次使用 Jemalloc 分析应用内存分配情况，首先在应用启动环境添加

export LD_PRELOAD=/usr/local/lib/libjemalloc.so
export MALLOC_CONF=prof:true,lg_prof_interval:31,prof_prefix:/root/jeprof-result/app

重启应用后等待 Jemalloc 输出内存采样文件，查看多个采样文件后发现端倪

可以看到 524336 0.1% 100.0% 7386790 0.8% Java_java_util_zip_Inflater_init JNI调用

定位问题代码

使用 jstack $pid | grep Inflater 查找调用栈（可能需要多次）

发现在 Redis 序列化与反序列时使用了 JDK 中的 ZLIB 压缩算法，查看源码可以看到未调用 stream.close() infalter.end() 和 defalter.end()方法

顺便确认下 native code 调用

修改问题代码

直接看代码

ELK分布式日志收集

Sat, 07 Oct 2023 14:45:11 +0800

ELK 架构

文档

flowchart LR
    subgraph server-1
        log-file-1 --> filebeat-1
    end
    subgraph server-2
        log-file-2 --> filebeat-2
    end
    filebeat-1 -- push log --> kafka
    filebeat-2 -- push log --> kafka
    kafka -- sub log --> logstash-filter
    logstash-filter -- pub log --> kafka
    kafka -- sub log --> logstash
    logstash -- push --> es
    kibana -- query --> es

配置ES

修改文件config/elasticsearch.yml

path.data
path.logs
network.host
http.port

生成相应账户密码

./bin/elasticsearch-reset-password -u elastic
./bin/elasticsearch-reset-password -u kibana-system

配置Kibana

server.port
server.host
elasticsearch.username
elasticsearch.password
i18n.locale

配置Kafka

新建 config/kafka_server_jass.conf

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password"
    user_tom="tom-password";
};
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password"
    user_tom="tom-password";
};
Client {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password";
};
Server {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password";
};

修改 bin/kafka-server-start.sh, 最后一行前添加 export KAFKA_OPTS="-Djava.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf"

新建 config/tom-client.conf

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
    required username="tom" password="tom-password";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

测试执行命令 ./bin/kafka-topics.sh --bootstrap-server ip:port --command-config config/tom-client.conf --list

算法基础-前缀树

Tue, 15 Mar 2022 10:53:07 +0800

字符串前缀相关操作

实现

package main

type TrieNode struct {
	isWord   bool
	children []*TrieNode
}

func NewTrieNode() *TrieNode {
	return &TrieNode{
		isWord:   false,
		children: make([]*TrieNode, 26),
	}
}

type Trie struct {
	root *TrieNode
}

func NewTrie() Trie {
	return Trie{NewTrieNode()}
}

func (t *Trie) Insert(word string) {
	node := t.root
	for i := range word {
		idx := int(word[i] - 'a')
		if node.children[idx] == nil {
			node.children[idx] = NewTrieNode()
		}
		node = node.children[idx]
	}
	node.isWord = true
}

func (t *Trie) Search(word string) bool {
	node := t.GetNode(word)
	if node == nil || !node.isWord {
		return false
	}
	return true
}

func (t *Trie) StartsWith(prefix string) bool {
	return t.GetNode(prefix) != nil
}

func (t *Trie) GetNode(word string) *TrieNode {
	node := t.root
	for i := range word {
		idx := int(word[i] - 'a')
		child := node.children[idx]
		if child == nil {
			return nil
		}
		node = child
	}
	return node
}

func (t *Trie) Scan(prefix string) []string {
	node := t.GetNode(prefix)
	res := make([]string, 0)
	if node == nil {
		return res
	}
	var dfs func(node *TrieNode, path []byte)
	dfs = func(node *TrieNode, path []byte) {
		if node == nil {
			return
		}
		if node.isWord {
			res = append(res, prefix+string(path))
		}
		for i := 0; i < 26; i++ {
			child := node.children[i]
			if child != nil {
				path = append(path, byte(i+'a'))
				dfs(child, path)
				path = path[:len(path)-1]
			}
		}
	}
	dfs(node, []byte{})
	return res
}

算法基础-最小生成树

Thu, 10 Mar 2022 14:42:47 +0800

由图生成权重最小的树

给你一个points 数组，表示 2D 平面上的一些点，其中 points[i] = [xi, yi] 。

连接点 [xi, yi] 和点 [xj, yj] 的费用为它们之间的曼哈顿距离：|xi - xj| + |yi - yj| ，其中 |val| 表示 val 的绝对值。

请你返回将所有点连接的最小总费用。只有任意两点之间有且仅有一条简单路径时，才认为所有点都已连接。

来源：力扣（LeetCode）

链接：https://leetcode-cn.com/problems/min-cost-to-connect-all-points

著作权归领扣网络所有。商业转载请联系官方授权，非商业转载请注明出处。

Kruskal

所有边按权重排序，使用并查集选出不成环的边

func NewUF(n int) *UnionFind {
	parent := make([]int, n)
	for i := range parent {
		parent[i] = i
	}
	return &UnionFind{
		parent: parent,
		count:  n,
	}
}

type UnionFind struct {
	parent []int
	count  int
}

func (uf *UnionFind) Union(a, b int) {
	rootA := uf.Find(a)
	rootB := uf.Find(b)
	if rootA == rootB {
		return
	}
	uf.parent[rootA] = rootB
	uf.count--
}

func (uf *UnionFind) Find(a int) int {
	for uf.parent[a] != a {
		uf.parent[a] = uf.parent[uf.parent[a]]
		a = uf.parent[a]
	}
	return a
}

func (uf *UnionFind) Connected(a, b int) bool {
	rootA := uf.Find(a)
	rootB := uf.Find(b)
	return rootA == rootB
}

func minCostConnectPoints(points [][]int) int {
	n := len(points)
    // graph [from, to, weight]
	graph := make([][]int, 0)
	for i := 0; i < n; i++ {
		for j := i + 1; j < n; j++ {
			edge := []int{i, j, dist(points, i, j)}
			graph = append(graph, edge)
		}
	}
	sort.Slice(graph, func(i, j int) bool {
		return graph[i][2] < graph[j][2]
	})

	uf := NewUF(n)
	res := 0
	for _, edge := range graph {
		a, b, w := edge[0], edge[1], edge[2]
		if uf.Connected(a, b) {
			continue
		}
		uf.Union(a, b)
		res += w
	}
	return res
}

func abs(a int) int {
	if a > 0 {
		return a
	}
	return -a
}

func dist(points [][]int, a, b int) int {
	return abs(points[a][0]-points[b][0]) + abs(points[a][1]-points[b][1])
}

Prim

图中每次切分，至少包含一条最小生成树的边（即权重最小的边必为最小生成树路径）

func abs(a int) int {
	if a > 0 {
		return a
	}
	return -a
}

func dist(points [][]int, a, b int) int {
	return abs(points[a][0]-points[b][0]) + abs(points[a][1]-points[b][1])
}

type PriorityQueue struct {
	data [][]int
}

func (p *PriorityQueue) Len() int {
	return len(p.data)
}

func (p *PriorityQueue) Less(i, j int) bool {
	return p.data[i][1] < p.data[j][1]
}

func (p *PriorityQueue) Swap(i, j int) {
	p.data[i], p.data[j] = p.data[j], p.data[i]
}

func (p *PriorityQueue) Push(x interface{}) {
	p.data = append(p.data, x.([]int))
}

func (p *PriorityQueue) Pop() interface{} {
	v := p.data[len(p.data)-1]
	p.data = p.data[:len(p.data)-1]
	return v
}

func minCostConnectPoints(points [][]int) int {
	n := len(points)
    // graph[from] [to, weight]
	graph := make([][][]int, n)
	for i := 0; i < n; i++ {
        // 邻接表表示的无向图，需要双向赋值
		for j := 0; j < n; j++ {
			if i == j {
				continue
			}
			graph[i] = append(graph[i], []int{j, dist(points, i, j)})
		}
	}

	used := make([]bool, n)
	pq := &PriorityQueue{[][]int{}}
	cut := func(a int) {
		for _, edge := range graph[a] {
			if used[edge[0]] {
				continue
			}
			heap.Push(pq, edge)
		}
	}
	used[0] = true
	cut(0)

	w := 0
	for pq.Len() > 0 {
		edge := heap.Pop(pq).([]int)
		if used[edge[0]] {
			continue
		}
		w += edge[1]
		used[edge[0]] = true
		cut(edge[0])
	}

	return w
}

算法基础-二分图

Tue, 08 Mar 2022 17:42:32 +0800

用于将节点分为两组

判断是否为二分图

BFS 实现

func isBipartite(graph [][]int) bool {
	n := len(graph)
	visited := make([]bool, n)
	color := make([]bool, n)
	valid := true

	var bfs func(int)
	bfs = func(root int) {
		visited[root] = true
		queue := []int{root}
		for len(queue) > 0 {
			node := queue[0]
			queue = queue[1:]
			for _, next := range graph[node] {
				if !visited[next] {
					color[next] = !color[node]
					visited[next] = true
					queue = append(queue, next)
				} else {
					if color[next] == color[node] {
						valid = false
						return
					}
				}
			}
		}
	}

	for i := 0; i < n; i++ {
		if !visited[i] && valid {
			bfs(i)
		}
	}

	return valid
}

DFS 实现

func possibleBipartition(n int, dislikes [][]int) bool {
	graph := make([][]int, n)
	for _, dislike := range dislikes {
		a, b := dislike[0]-1, dislike[1]-1
		graph[a] = append(graph[a], b)
		graph[b] = append(graph[b], a)
	}

	valid := true
	visited := make([]bool, n)
	color := make([]bool, n)
	var dfs func(node int)
	dfs = func(node int) {
		if !valid {
			return
		}
		visited[node] = true
		for _, next := range graph[node] {
			if !visited[next] {
				color[next] = !color[node]
				dfs(next)
			} else {
				if color[next] == color[node] {
					valid = false
					return
				}
			}
		}
	}

	for i := 0; i < n; i++ {
		if !visited[i] && valid {
			dfs(i)
		}
	}
	return valid
}

算法基础-并查集

Tue, 08 Mar 2022 17:38:39 +0800

用于判断连通性、等效等问题

实现

func NewUF(n int) *UnionFind {
	parent := make([]int, n)
	for i := range parent {
		parent[i] = i
	}
	return &UnionFind{
		parent: parent,
		count:  n,
	}
}

type UnionFind struct {
	parent []int
	count  int
}

func (uf *UnionFind) Union(a, b int) {
	rootA := uf.Find(a)
	rootB := uf.Find(b)
	if rootA == rootB {
		return
	}
	uf.parent[rootA] = rootB
	uf.count--
}

func (uf *UnionFind) Find(a int) int {
	for uf.parent[a] != a {
		// compress parent tree
		uf.parent[a] = uf.parent[uf.parent[a]]
		a = uf.parent[a]
	}
	return a
}

func (uf *UnionFind) Connected(a, b int) bool {
	rootA := uf.Find(a)
	rootB := uf.Find(b)
	return rootA == rootB
}

算法基础-排序

Fri, 10 Sep 2021 15:36:40 +0800

经典排序算法

冒泡排序

func sortBubble1(arr []int) []int {
	n := len(arr)
	// 有序区间 [0, i]
	for i := 0; i < n; i++ {
		// 从最右侧两两交换，将无序区间中的最小值放置到i
		for j := n - 1; j > i; j-- {
			if arr[j] < arr[j-1] {
				swap(arr, j, j-1)
			}
		}
	}
    return arr
}

func sortBubble2(arr []int) {
	n := len(arr)
	exchanged := true
	// 有序区间 [0, i]
	for i := 0; i < n && exchanged; i++ {
        // 当 exchanged 为 true 时，表明剩余无序区间已有序，结束排序
		exchanged = false
		// 从最右侧两两交换，将无序区间中的最小值放置到i
		for j := n - 1; j > i; j-- {
			if arr[j] < arr[j-1] {
				exchanged = true
				swap(arr, j, j-1)
			}
		}
	}
}

快速排序

func sortQuick(arr []int) []int {
	return _sortQuick(arr, 0, len(arr) - 1)
}

func _sortQuick(arr []int, left, right int) []int {
	if left < right {
		p := partition(arr, left, right)
		_sortQuick(arr, left, p-1)
		_sortQuick(arr, p+1, right)
	}
	return arr
}

func partition(arr []int, l int, r int) int {
	// l位空出
	v := arr[l]
	for l < r {
		for l < r && arr[r] >= v {
			r--
		}
		// r位保存至l位，r位空出
		arr[l] = arr[r]
		for l < r && arr[l] <= v {
			l++
		}
		// l位保存至r位，l位空出
		arr[r] = arr[l]
	}
	// 空出的l位填入基准值v
	arr[l] = v
	return l
}

   class Solution {

    public int[] sortArray(int[] nums) {
        if (nums == null || nums.length <= 1) {
            return nums;
        }
        quickSort(nums, 0, nums.length - 1);
        return nums;
    }

    private void quickSort(int[] nums, int l, int r) {
        if (l >= r) {
            return;
        }
        int p = partition(nums, l, r);
        quickSort(nums, l, p - 1);
        quickSort(nums, p + 1, r);
    }

    private int partition(int[] nums, int l, int r) {
        int v = randomPiv(nums, l, r);
        while (l < r) {
            while (l < r && nums[r] >= v) {
                r--;
            }
            nums[l] = nums[r];
            while (l < r && nums[l] <= v) {
                l++;
            }
            nums[r] = nums[l];
        }
        nums[l] = v;
        return l;
    }

    private int randomPiv(int[] nums, int l, int r) {
        int p = ThreadLocalRandom.current().nextInt(r - l + 1) + l;
        int temp = nums[l];
        nums[l] = nums[p];
        nums[p] = temp;
        return nums[l];
    }

}

插入排序

func sortInsert(arr []int) []int{
	// 有序区间 [0, i]，无序区间[i+1, +00]
	for i := 1; i < len(arr); i++ {
		for j := i; j > 0 && arr[j] < arr[j-1]; j-- {
			// 小数前移
			swap(arr, j, j - 1)
		}
	}
	return arr
}

希尔排序（插入排序改进版）

希尔为该算法发明人

func sortShell(arr []int) []int {
	for step := len(arr) / 2; step > 0; step /= 2 {
		for i := step; i < len(arr); i++ {
			for j := i; j >= step && arr[j] < arr[j-step]; j -= step {
				// 同组内小数前移
				swap(arr, j, j-step)
			}
		}
	}
	return arr
}

选择排序

func sortSelect(arr []int) {
	for i := 0; i < len(arr); i++ {
		m := i
        // 无序区间内选择最小值下标 m
		for j := i; j < len(arr); j++ {
			if arr[j] < arr[m] {
				m = j
			}
		}
		if m != i {
			swap(arr, i, m)
		}
	}
}

堆排序

func sortHeap(arr []int) []int {
	h := MinHeap{}
	h.build(arr)
	res := make([]int, 0, len(arr))
	for len(h.heap) > 0 {
		res = append(res, h.pop())
	}
	return res
}

type MinHeap struct {
	heap []int
}

// 上浮操作：节点与大于节点值的父节点交换
func (h *MinHeap) up(k int) {
	if k == 0 {
		return
	}
    // 节点 k 比 父节点 pk 小时，交换两节点，实现节点上浮
	pk := (k - 1) / 2
	for k >= 0 && h.heap[pk] > h.heap[k] {
		h.heap[pk], h.heap[k] = h.heap[k], h.heap[pk]
		k = pk
		pk = (k - 1) / 2
	}
}


// 下沉操作：节点与子节点中较小值交换
func (h *MinHeap) down(k int) {
	if len(h.heap) <= 1 {
		return
	}
	lastNonLeaf := (len(h.heap) - 1 - 1) / 2
	for k <= lastNonLeaf {
		// 默认左侧节点为子节点中值最小节点
		child := 2*k + 1
		childV := h.heap[child]
        // 检查右侧节点，若比默认节点小，替换之
		right := child + 1
		if right < len(h.heap) && h.heap[right] < childV {
			child = right
			childV = h.heap[right]
		}
        // 若节点比子节点大，下沉
		if h.heap[k] > childV {
			h.heap[k], h.heap[child] = h.heap[child], h.heap[k]
			k = child
		} else {
			break
		}
	}
}

// 添加操作：加至末尾节点，后上浮尾节点
func (h *MinHeap) add(v int) {
	h.heap = append(h.heap, v)
	h.up(len(h.heap) - 1)
}

// 弹出操作：头节点弹出，尾节点放置头节点，后下沉头节点
func (h *MinHeap) pop() int {
	if len(h.heap) == 0 {
		panic("empty heap")
	}
	res := h.heap[0]
	// swap last and first
	h.heap[0] = h.heap[len(h.heap)-1]
	// remove last
	h.heap = h.heap[:len(h.heap)-1]
	if len(h.heap) > 0 {
		// re-heap
		h.down(0)
	}
	return res
}

// 构建树：从最后一个非叶子节点开始，执行下沉操作
func (h *MinHeap) build(arr []int) {
	h.heap = arr
	for i := (len(h.heap) - 2) / 2; i >= 0; i-- {
		h.down(i)
	}
}

归并排序

func sortMerge(arr []int) []int {
	if len(arr) <= 1 {
		return arr
	}
	mid := len(arr) / 2
	return merge(sortMerge(arr[:mid]), sortMerge(arr[mid:]))
}

func merge(left, right []int) []int {
	res := make([]int, 0, len(left)+len(right))
    // 每次处理一个元素
	i, j := 0, 0
	for i < len(left) && j < len(right) {
		if left[i] < right[j] {
			res = append(res, left[i])
			i++
		} else {
			res = append(res, right[j])
			j++
		}
	}
	if i < len(left) {
		res = append(res, left[i:]...)
	}
	if j < len(right) {
		res = append(res, right[j:]...)
	}
	return res
}

class Solution {

    public int[] sortArray(int[] nums) {
        if (nums == null || nums.length <= 1) {
            return nums;
        }
        sort(nums, 0, nums.length);
        return nums;
    }

    // [left, right)
    private void sort(int[] nums, int left, int right) {
        if (right - left <= 1) {
            return;
        }
        int mid = left + (right - left) / 2;
        sort(nums, left, mid);
        sort(nums, mid, right);
        merge(nums, left, mid, right);
    }

    private void merge(int[] nums, int left, int mid, int right) {
        int[] arr = new int[right - left];
        int p = 0;
        int i = left, j = mid;
        while (i < mid && j < right) {
            if (nums[i] < nums[j]) {
                arr[p] = nums[i++];
            } else {
                arr[p] = nums[j++];
            }
            p++;
        }
        while (i < mid) {
            arr[p++] = nums[i++];
        }
        while (j < right) {
            arr[p++] = nums[j++];
        }
        System.arraycopy(arr, 0, nums, left, right-left);
    }

}

计数排序

func sortCount(arr []int) []int {
	// 建立排序桶 [min, max]
	minV, maxV := arr[0], arr[0]
	for _, v := range arr {
		if v < minV {
			minV = v
		}
		if v > maxV {
			maxV = v
		}
	}
	buckets := make([]int, maxV-minV+1)
	for _, v := range arr {
        // 需要 min 偏移
		buckets[v-minV]++
	}

	// 桶中提取数据
	res := make([]int, 0, len(arr))
	for i, count := range buckets {
		for ; count > 0; count-- {
            // 加回偏移量 minV
			res = append(res, i+minV)
		}
	}
	return res
}

基数排序（桶排序的一种）

基数排序：根据键值的每位数字来分配桶
计数排序：每个桶只存储单一键值
桶排序：每个桶存储一定范围的数值，桶内使用其他排序算法

func sortRadix(arr []int) []int {
    // 默认 10 进制，故桶为 [0, 9] 
    // 最大值位数默认
	maxV := arr[0]
	for _, v := range arr {
		if v > maxV {
			maxV = v
		}
	}
	maxLen := len(strconv.Itoa(maxV))

    // 按低位至高位排序
	step := 1
	for i := 0; i < maxLen; i++ {
		var buckets [10][]int
		// 插入排序桶
		for _, v := range arr {
			bitV := v / step % 10
			buckets[bitV] = append(buckets[bitV], v)
		}
		// 更新数组
		idx := 0
		for _, bucket := range buckets {
			for _, v := range bucket {
				arr[idx] = v
				idx++
			}
		}
		step *= 10
	}
	return arr
}

Redis主从与集群模式

Mon, 16 Aug 2021 09:55:00 +0800

Redis集群相关

主从模式

主节点Redis配置 redis-6090.conf：

port 6090
protected-mode no

从节点Redis配置 redis-6091.conf：

port 6091
protected-mode no
slaveof 127.0.0.1 6090

从节点Redis配置 redis-6092.conf：

port 6092
protected-mode no
slaveof 127.0.0.1 6090

Redis Sentinel配置模版 redis-sentinel-cluster.tmpl：

port ${PORT}
sentinel monitor mymaster 127.0.0.1 6090 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000

Redis Sentinel 实例的配置文件 redis-sentinel-cluster-config.sh，生成 5000-5002 3个实例：

#!/bin/bash

for port in `seq 5000 5002`; do \
  mkdir -p ./redis-sentinel-cluster/${port}/conf
  PORT=${port} envsubst < ./redis-sentinel-cluster.tmpl > ./redis-sentinel-cluster/${port}/conf/sentinel.conf
  chmod 777 ./redis-sentinel-cluster/${port}/conf/sentinel.conf
  mkdir -p ./redis-sentinel-cluster/${port}/data
done

使用 Docker Compose 部署：

version: "3"

services:
  redis7000:
    image: redis:alpine
    container_name: redis6090
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-6090/conf:/usr/local/etc/redis
      - ./redis-6090/data:/data
  redis7001:
    image: redis:alpine
    container_name: redis6091
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-6091/conf:/usr/local/etc/redis
      - ./redis-6091/data:/data
  redis7002:
    image: redis:alpine
    container_name: redis6092
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-6092/conf:/usr/local/etc/redis
      - ./redis-6092/data:/data

  sentinel5000:
    image: redis:alpine
    container_name: sentinel5000
    command: "redis-server /usr/local/etc/redis/sentinel.conf --sentinel"
    network_mode: host
    volumes:
      - ./redis-sentinel-cluster/5000/conf:/usr/local/etc/redis
      - ./redis-sentinel-cluster/5000/data:/data
  sentinel5001:
    image: redis:alpine
    container_name: sentinel5001
    command: "redis-server /usr/local/etc/redis/sentinel.conf --sentinel"
    network_mode: host
    volumes:
      - ./redis-sentinel-cluster/5001/conf:/usr/local/etc/redis
      - ./redis-sentinel-cluster/5001/data:/data
  sentinel5002:
    image: redis:alpine
    container_name: sentinel5002
    command: "redis-server /usr/local/etc/redis/sentinel.conf --sentinel"
    network_mode: host
    volumes:
      - ./redis-sentinel-cluster/5002/conf:/usr/local/etc/redis
      - ./redis-sentinel-cluster/5002/data:/data

Cluster 模式

参考资料

Redis集群配置文件

Redis配置模版 redis-cluster.tmpl：

port ${PORT}
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

各个Redis实例配置生成脚本 redis-cluster-config.sh，脚本生成端口号 7000-7005 6个Redis实例：

#!/bin/bash

for port in `seq 7000 7005`; do
  mkdir -p ./redis-cluster/${port}/conf
  PORT=${port} envsubst < ./redis-cluster.tmpl > ./redis-cluster/${port}/conf/redis.conf
  chmod 666 ./redis-cluster/${port}/conf/redis.conf
  mkdir -p ./redis-cluster/${port}/data
done

Redis集群启动

使用 Docker Compose 部署：

version: "3"

services:
  redis7000:
    image: redis:alpine
    container_name: redis7000
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-cluster/7000/conf:/usr/local/etc/redis
      - ./redis-cluster/7000/data:/data
  redis7001:
    image: redis:alpine
    container_name: redis7001
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-cluster/7001/conf:/usr/local/etc/redis
      - ./redis-cluster/7001/data:/data
  redis7002:
    image: redis:alpine
    container_name: redis7002
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-cluster/7002/conf:/usr/local/etc/redis
      - ./redis-cluster/7002/data:/data
  redis7003:
    image: redis:alpine
    container_name: redis7003
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-cluster/7003/conf:/usr/local/etc/redis
      - ./redis-cluster/7003/data:/data
  redis7004:
    image: redis:alpine
    container_name: redis7004
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-cluster/7004/conf:/usr/local/etc/redis
      - ./redis-cluster/7004/data:/data
  redis7005:
    image: redis:alpine
    container_name: redis7005
    command: "redis-server /usr/local/etc/redis/redis.conf"
    network_mode: host
    volumes:
      - ./redis-cluster/7005/conf:/usr/local/etc/redis
      - ./redis-cluster/7005/data:/data

容器启动完毕后，连接至任意一个Redis示例，开始初始化集群：

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
                           127.0.0.1:7002 127.0.0.1:7003 \
                           127.0.0.1:7004 127.0.0.1:7005 \
                           --cluster-replicas 1

执行完毕后将构建一个3主3从的Redis集群。

测试集群

对于Redis集群，命令行模式需要增加 -c 选项，否则无法操作Redis，即 redis-cli -c -p 7000

连接到端口号为7000的Redis，查看节点信息：

127.0.0.1:7000> cluster nodes

output> 2fc17db8bb8818ec8ac50f40f07bfbdf1ee8d56c 127.0.0.1:7002@17002 master - 0 1629082981000 3 connected 10923-16383
fc92c1709a88bdca8e86b9b5a27d035609e46059 127.0.0.1:7001@17001 master - 0 1629082982000 2 connected 5461-10922
3bd1b8cbe2e0e30a77dfa2b92dd70ed834bfcba4 127.0.0.1:7000@17000 myself,master - 0 1629082981000 1 connected 0-5460
cf361fc2b72ca2bfd070b282754ad19da7640485 127.0.0.1:7003@17003 slave 3bd1b8cbe2e0e30a77dfa2b92dd70ed834bfcba4 0 1629082982590 1 connected
fddc21542112cdcd8857bbfc77481b673e3a543e 127.0.0.1:7004@17004 slave fc92c1709a88bdca8e86b9b5a27d035609e46059 0 1629082981586 2 connected
2a5e09014c852fd15c2e54d500c532ec6ee2490e 127.0.0.1:7005@17005 slave 2fc17db8bb8818ec8ac50f40f07bfbdf1ee8d56c 0 1629082982088 3 connected

现在让7000的实例离线30秒：

redis-cli -p 7000 DEBUG sleep 30

现在看一下集群信息，可以看到主节点已经由7000自动切换到了7003，：

127.0.0.1:7000> cluster nodes
2fc17db8bb8818ec8ac50f40f07bfbdf1ee8d56c 127.0.0.1:7002@17002 master - 0 1629083362653 3 connected 10923-16383
fc92c1709a88bdca8e86b9b5a27d035609e46059 127.0.0.1:7001@17001 master - 0 1629083361000 2 connected 5461-10922
3bd1b8cbe2e0e30a77dfa2b92dd70ed834bfcba4 127.0.0.1:7000@17000 myself,slave cf361fc2b72ca2bfd070b282754ad19da7640485 0 1629083362000 7 connected
cf361fc2b72ca2bfd070b282754ad19da7640485 127.0.0.1:7003@17003 master - 0 1629083362000 7 connected 0-5460
fddc21542112cdcd8857bbfc77481b673e3a543e 127.0.0.1:7004@17004 slave fc92c1709a88bdca8e86b9b5a27d035609e46059 0 1629083362553 2 connected
2a5e09014c852fd15c2e54d500c532ec6ee2490e 127.0.0.1:7005@17005 slave 2fc17db8bb8818ec8ac50f40f07bfbdf1ee8d56c 0 1629083361647 3 connected

可以看到7000已经切换为了salve模式，7003切换成了master模式。

使用Go语言测试读写：

package main

import (
	"context"
	"fmt"
	"github.com/go-redis/redis/v8"
)

var ctx = context.Background()

func redisAddrList() []string {
	ip := "localhost"
	addrList := []string{}
	for port := 7000; port < 7006; port++ {
		addrList = append(addrList, fmt.Sprintf("%s:%d", ip, port))
	}
	return addrList
}

func NewRedisClient() *redis.ClusterClient {
	return redis.NewClusterClient(&redis.ClusterOptions{
		Addrs: redisAddrList(),
	})
}

func main() {
	rdb := NewRedisClient()
	err := rdb.Set(ctx, "foo", "bar", 0).Err()
	if err != nil {
		panic(err)
	}

	res, err := rdb.Get(ctx, "foo").Result()
	if err != nil {
		panic(err)
	}

	fmt.Println(res)
}

扩充节点

当前节点和槽分配情况：

~/docker-app/redis-cluster » redis-cli -p 7000 cluster nodes     
9a353c214cb677b1e996573b4550875fb0754b65 127.0.0.1:7000@17000 myself,slave 0a50949e09aa0ebce0dbad1c8b0f301adbd62780 0 1736083233000 7 connected
0a50949e09aa0ebce0dbad1c8b0f301adbd62780 127.0.0.1:7003@17003 master - 0 1736083234560 7 connected 0-5460
3b8b46eac23b396e1d0b1adcfb70d086ee539da5 127.0.0.1:7004@17004 slave bd393aeacf7de92c8a3816cbfb1c497a310a418b 0 1736083235665 2 connected
bd393aeacf7de92c8a3816cbfb1c497a310a418b 127.0.0.1:7001@17001 master - 0 1736083235000 2 connected 5461-10922
3c21e101ea340698769de95812a98275693d3b2d 127.0.0.1:7002@17002 slave 57011e62426c833268c7f01b248b0e8f44af443a 0 1736083234660 8 connected
57011e62426c833268c7f01b248b0e8f44af443a 127.0.0.1:7005@17005 master - 0 1736083235563 8 connected 10923-16383

在工作目录下复制一个redis配置 cp -r 7005 7006，并将其端口号配置为7006，使用docker启动临时redis

docker run -d \
    --name redis7006 \
    --network host \
    -v ./redis-cluster/7006/conf:/usr/local/etc/redis \
    -v ./redis-cluster/7006/data:/data \
    redis:6.2 redis-server /usr/local/etc/redis/redis.conf

确认启动成功后，执行命令添加新的 redis node 到集群中去：

~/docker-app/redis-cluster » redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000                                                                                               ing@ing-pc
>>> Adding node 127.0.0.1:7006 to cluster 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
S: 9a353c214cb677b1e996573b4550875fb0754b65 127.0.0.1:7000
   slots: (0 slots) slave
   replicates 0a50949e09aa0ebce0dbad1c8b0f301adbd62780
M: 0a50949e09aa0ebce0dbad1c8b0f301adbd62780 127.0.0.1:7003
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 3b8b46eac23b396e1d0b1adcfb70d086ee539da5 127.0.0.1:7004
   slots: (0 slots) slave
   replicates bd393aeacf7de92c8a3816cbfb1c497a310a418b
M: bd393aeacf7de92c8a3816cbfb1c497a310a418b 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 3c21e101ea340698769de95812a98275693d3b2d 127.0.0.1:7002
   slots: (0 slots) slave
   replicates 57011e62426c833268c7f01b248b0e8f44af443a
M: 57011e62426c833268c7f01b248b0e8f44af443a 127.0.0.1:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:7006 to make it join the cluster.
[OK] New node added correctly.

此时再次查看节点信息，可以看到7006已经加入到cluster中，：

~/docker-app/redis-cluster » redis-cli -p 7000 cluster nodes                                                                                                                          ing@ing-pc
9a353c214cb677b1e996573b4550875fb0754b65 127.0.0.1:7000@17000 myself,slave 0a50949e09aa0ebce0dbad1c8b0f301adbd62780 0 1736083995000 7 connected
0a50949e09aa0ebce0dbad1c8b0f301adbd62780 127.0.0.1:7003@17003 master - 0 1736083997000 7 connected 0-5460
3b8b46eac23b396e1d0b1adcfb70d086ee539da5 127.0.0.1:7004@17004 slave bd393aeacf7de92c8a3816cbfb1c497a310a418b 0 1736083995380 2 connected
 127.0.0.1:7006@17006 master - 0 1736083996384 0 connected
bd393aeacf7de92c8a3816cbfb1c497a310a418b 127.0.0.1:7001@17001 master - 0 1736083997390 2 connected 5461-10922
3c21e101ea340698769de95812a98275693d3b2d 127.0.0.1:7002@17002 slave 57011e62426c833268c7f01b248b0e8f44af443a 0 1736083996585 8 connected
57011e62426c833268c7f01b248b0e8f44af443a 127.0.0.1:7005@17005 master - 0 1736083996000 8 connected 10923-16383

如果需要给7006指定一个slave，则启动7007端口的redis后，执行命令以下命令：

$ redis-cli -p 7000 --cluster add-node 127.0.0.1:7007 127.0.0.1:7000 --cluster-slave --cluster-master-id d1c7e05fefad29888e1e4eb3a19574b94a280932

其中 –cluster-slave 指明新节点为slave，d1c7e05fefad29888e1e4eb3a19574b94a280932是7007节点id

新加入的节点是没有自动分配slot的，可以查看确认：

~/docker-app/redis-cluster » redis-cli  -p 7000 -c cluster slots                                                                                                                  1 ↵ ing@ing-pc
1) 1) (integer) 0
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 7003
      3) "0a50949e09aa0ebce0dbad1c8b0f301adbd62780"
   4) 1) "127.0.0.1"
      2) (integer) 7000
      3) "9a353c214cb677b1e996573b4550875fb0754b65"
2) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7001
      3) "bd393aeacf7de92c8a3816cbfb1c497a310a418b"
   4) 1) "127.0.0.1"
      2) (integer) 7004
      3) "3b8b46eac23b396e1d0b1adcfb70d086ee539da5"
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7005
      3) "57011e62426c833268c7f01b248b0e8f44af443a"
   4) 1) "127.0.0.1"
      2) (integer) 7002
      3) "3c21e101ea340698769de95812a98275693d3b2d"

开始为7006节点分配slot

~/docker-app/redis-cluster » redis-cli -p 7000 --cluster reshard 127.0.0.1:7000                                                                                                       ing@ing-pc
>>> Performing Cluster Check (using node 127.0.0.1:7000)
S: 9a353c214cb677b1e996573b4550875fb0754b65 127.0.0.1:7000
   slots: (0 slots) slave
   replicates 0a50949e09aa0ebce0dbad1c8b0f301adbd62780
M: 0a50949e09aa0ebce0dbad1c8b0f301adbd62780 127.0.0.1:7003
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 3b8b46eac23b396e1d0b1adcfb70d086ee539da5 127.0.0.1:7004
   slots: (0 slots) slave
   replicates bd393aeacf7de92c8a3816cbfb1c497a310a418b
M: d1c7e05fefad29888e1e4eb3a19574b94a280932 127.0.0.1:7006
   slots: (0 slots) master
M: bd393aeacf7de92c8a3816cbfb1c497a310a418b 127.0.0.1:7001
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 3c21e101ea340698769de95812a98275693d3b2d 127.0.0.1:7002
   slots: (0 slots) slave
   replicates 57011e62426c833268c7f01b248b0e8f44af443a
M: 57011e62426c833268c7f01b248b0e8f44af443a 127.0.0.1:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?

接下来会要求输入分配多少slot到哪个node，我们选择slot数量4096，分配到7006节点d1c7e05fefad29888e1e4eb3a19574b94a280932

[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? d1c7e05fefad29888e1e4eb3a19574b94a280932
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: all

Ready to move 4096 slots.
  Source nodes:
    M: 0a50949e09aa0ebce0dbad1c8b0f301adbd62780 127.0.0.1:7003
       slots:[0-5460] (5461 slots) master
       1 additional replica(s)
    M: bd393aeacf7de92c8a3816cbfb1c497a310a418b 127.0.0.1:7001
       slots:[5461-10922] (5462 slots) master
       1 additional replica(s)
    M: 57011e62426c833268c7f01b248b0e8f44af443a 127.0.0.1:7005
       slots:[10923-16383] (5461 slots) master
       1 additional replica(s)
  Destination node:
    M: d1c7e05fefad29888e1e4eb3a19574b94a280932 127.0.0.1:7006
       slots: (0 slots) master
  Resharding plan:
    Moving slot 5461 from bd393aeacf7de92c8a3816cbfb1c497a310a418b
    Moving slot 5462 from bd393aeacf7de92c8a3816cbfb1c497a310a418b
    ....

执行完毕再次确认slot分配情况

~/docker-app/redis-cluster » redis-cli -p 7000 -c cluster slots                                                                                                                       ing@ing-pc
1) 1) (integer) 0
   2) (integer) 1364
   3) 1) "127.0.0.1"
      2) (integer) 7006
      3) "d1c7e05fefad29888e1e4eb3a19574b94a280932"
2) 1) (integer) 1365
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 7003
      3) "0a50949e09aa0ebce0dbad1c8b0f301adbd62780"
   4) 1) "127.0.0.1"
      2) (integer) 7000
      3) "9a353c214cb677b1e996573b4550875fb0754b65"
3) 1) (integer) 5461
   2) (integer) 6826
   3) 1) "127.0.0.1"
      2) (integer) 7006
      3) "d1c7e05fefad29888e1e4eb3a19574b94a280932"
4) 1) (integer) 6827
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7001
      3) "bd393aeacf7de92c8a3816cbfb1c497a310a418b"
   4) 1) "127.0.0.1"
      2) (integer) 7004
      3) "3b8b46eac23b396e1d0b1adcfb70d086ee539da5"
5) 1) (integer) 10923
   2) (integer) 12287
   3) 1) "127.0.0.1"
      2) (integer) 7006
      3) "d1c7e05fefad29888e1e4eb3a19574b94a280932"
6) 1) (integer) 12288
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7005
      3) "57011e62426c833268c7f01b248b0e8f44af443a"
   4) 1) "127.0.0.1"
      2) (integer) 7002
      3) "3c21e101ea340698769de95812a98275693d3b2d"

删除节点

停止7006容器 docker stop 7006，使用CLUSTER FORGET移除节点：

~/docker-app/redis-cluster » redis-cli -p 7000 -c CLUSTER FORGET d1c7e05fefad29888e1e4eb3a19574b94a280932                                                                             ing@ing-pc
OK

~/docker-app/redis-cluster » redis-cli -p 7000 -c CLUSTER NODES                                                                                                                       ing@ing-pc
9a353c214cb677b1e996573b4550875fb0754b65 127.0.0.1:7000@17000 myself,slave 0a50949e09aa0ebce0dbad1c8b0f301adbd62780 0 1736085038000 7 connected
0a50949e09aa0ebce0dbad1c8b0f301adbd62780 127.0.0.1:7003@17003 master - 0 1736085039571 7 connected 1365-5460
3b8b46eac23b396e1d0b1adcfb70d086ee539da5 127.0.0.1:7004@17004 slave bd393aeacf7de92c8a3816cbfb1c497a310a418b 0 1736085039673 2 connected
bd393aeacf7de92c8a3816cbfb1c497a310a418b 127.0.0.1:7001@17001 master - 0 1736085038667 2 connected 6827-10922
3c21e101ea340698769de95812a98275693d3b2d 127.0.0.1:7002@17002 slave 57011e62426c833268c7f01b248b0e8f44af443a 0 1736085039068 8 connected
57011e62426c833268c7f01b248b0e8f44af443a 127.0.0.1:7005@17005 master - 0 1736085039000 8 connected 12288-16383

~/docker-app/redis-cluster » redis-cli -p 7000 -c CLUSTER SLOTS                                                                                                                       ing@ing-pc
1) 1) (integer) 1365
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 7003
      3) "0a50949e09aa0ebce0dbad1c8b0f301adbd62780"
   4) 1) "127.0.0.1"
      2) (integer) 7000
      3) "9a353c214cb677b1e996573b4550875fb0754b65"
2) 1) (integer) 6827
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7001
      3) "bd393aeacf7de92c8a3816cbfb1c497a310a418b"
   4) 1) "127.0.0.1"
      2) (integer) 7004
      3) "3b8b46eac23b396e1d0b1adcfb70d086ee539da5"
3) 1) (integer) 12288
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7005
      3) "57011e62426c833268c7f01b248b0e8f44af443a"
   4) 1) "127.0.0.1"
      2) (integer) 7002
      3) "3c21e101ea340698769de95812a98275693d3b2d"

发现此时slot缺失了一部分，查看cluster状态会发现集群此时处于 fail

~/docker-app/redis-cluster » redis-cli -p 7000 -c CLUSTER INFO                                                                                                                        ing@ing-pc
cluster_state:fail
cluster_slots_assigned:12288
cluster_slots_ok:12288
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:9
cluster_my_epoch:7
cluster_stats_messages_ping_sent:7170
cluster_stats_messages_pong_sent:7271
cluster_stats_messages_update_sent:2
cluster_stats_messages_sent:14443
cluster_stats_messages_ping_received:7270
cluster_stats_messages_pong_received:11255
cluster_stats_messages_meet_received:1
cluster_stats_messages_fail_received:3
cluster_stats_messages_auth-req_received:1
cluster_stats_messages_received:18530

原因是7006在持有slot的状态下被移除了集群，此时我们需要再次reshard

当然你可以设置 cluster-require-full-coverage=no 让集群强制在缺失slot的情况下继续提供服务

执行命令修复集群

~/docker-app/redis-cluster » redis-cli --cluster fix 127.0.0.1:7000                                                                                                               1 ↵ ing@ing-pc
127.0.0.1:7003 (0a50949e...) -> 0 keys | 4096 slots | 1 slaves.
127.0.0.1:7001 (bd393aea...) -> 0 keys | 4096 slots | 1 slaves.
127.0.0.1:7005 (57011e62...) -> 1 keys | 4096 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:7000)
S: 9a353c214cb677b1e996573b4550875fb0754b65 127.0.0.1:7000
   slots: (0 slots) slave
   replicates 0a50949e09aa0ebce0dbad1c8b0f301adbd62780
M: 0a50949e09aa0ebce0dbad1c8b0f301adbd62780 127.0.0.1:7003
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 3b8b46eac23b396e1d0b1adcfb70d086ee539da5 127.0.0.1:7004
   slots: (0 slots) slave
   replicates bd393aeacf7de92c8a3816cbfb1c497a310a418b
M: bd393aeacf7de92c8a3816cbfb1c497a310a418b 127.0.0.1:7001
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 3c21e101ea340698769de95812a98275693d3b2d 127.0.0.1:7002
   slots: (0 slots) slave
   replicates 57011e62426c833268c7f01b248b0e8f44af443a
M: 57011e62426c833268c7f01b248b0e8f44af443a 127.0.0.1:7005
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.

>>> Fixing slots coverage...
The following uncovered slots have no keys across the cluster:
[0-1364],[5461-6826],[10923-12287]
Fix these slots by covering with a random node? (type 'yes' to accept): yes
>>> Covering slot 5470 with 127.0.0.1:7005
>>> Covering slot 11756 with 127.0.0.1:7005
...

执行命令检查集群状态

~/docker-app/redis-cluster » redis-cli --cluster check 127.0.0.1:7000  
 ...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

~/docker-app/redis-cluster » redis-cli -c -p 7000 CLUSTER INFO                                                                                                                    1 ↵ ing@ing-pc
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:14
cluster_my_epoch:13
cluster_stats_messages_ping_sent:8616
cluster_stats_messages_pong_sent:8700
cluster_stats_messages_update_sent:2
cluster_stats_messages_sent:17318
cluster_stats_messages_ping_received:8699
cluster_stats_messages_pong_received:12698
cluster_stats_messages_meet_received:1
cluster_stats_messages_fail_received:3
cluster_stats_messages_auth-req_received:1
cluster_stats_messages_update_received:2
cluster_stats_messages_received:21404

算法基础-堆

Fri, 13 Aug 2021 09:47:50 +0800

堆，亦称为优先队列

最小堆

设堆大小为 size，对于下标为k的节点：

父节点下标为 (k - 1) / 2
左子节点下标为 2 * k + 1，右子节点下标为 2 * k + 2
最后一个非叶子节点的下标为 (size - 2) / 2 (即最后一个节点 size - 1的父节点 ((size - 1) - 1) / 2)

go语言参考实现：

type MinHeap struct {
	heap []int
}

func NewMinHeap() *MinHeap {
	return &MinHeap{heap: []int{}}
}

func (h *MinHeap) up(k int) {
	if k == 0 {
		return
	}
	for pk := (k - 1) / 2; k >= 0 && h.heap[pk] > h.heap[k]; {
		h.heap[pk], h.heap[k] = h.heap[k], h.heap[pk]
		k = pk
		pk = (k - 1) / 2
	}
}

func (h *MinHeap) down(k int) {
	if len(h.heap) == 1 {
		return
	}
	last := (len(h.heap) - 2) / 2
	for k <= last {
		child := 2*k + 1 // left child
		childV := h.heap[child]
		right := child + 1
		if right < len(h.heap) && h.heap[right] < childV {
			child = right
			childV = h.heap[right]
		}
		if h.heap[k] > childV {
			h.heap[k], h.heap[child] = h.heap[child], h.heap[k]
			k = child
		} else {
			break
		}
	}
}

func (h *MinHeap) add(v int) {
	h.heap = append(h.heap, v)
	h.up(len(h.heap) - 1)
}

func (h *MinHeap) pop() (int, bool) {
	if len(h.heap) == 0 {
		return 0, false
	}
	res := h.heap[0]
	h.heap[0] = h.heap[len(h.heap)-1]
	h.heap = h.heap[:len(h.heap)-1]
	if len(h.heap) > 0 {
		h.down(0)
	}
	return res, true
}

func (h *MinHeap) heapify(arr []int) {
	h.heap = arr
	for i := (len(h.heap) - 2) / 2; i >= 0; i-- {
		h.down(i)
	}
}