Buffered I/O那么大家肯定好奇, 如何查看系统的 page cache 中都缓存了哪些文件, 以及各个文件缓存了多少个 page 页, 那些 page 页被缓存 ?
page cache/proc/{pid}/maps/proc/{pid}/fd/*pgcacherpgcacher分析文件在 page cache 缓存占用信息的实现原理
如何分析文件在 page cache 缓存使用情况
查看分析某个文件在 page cache 缓存中的统计原理是这样的.
syscall.Mincorepage那么文件的 page cache 内存使用情况，只需要使用被缓存的 page 的数量乘以系统的 page size 即可, page size 默认为 4KB.
# getconf PAGESIZE
4096
pgcacherGetFileMincoreimport (
"fmt"
"os"
"unsafe"
"golang.org/x/sys/unix"
)
type Mincore struct {
Cached int64
Miss int64
}
func GetFileMincore(f *os.File, size int64) (*Mincore, error) {
if int(size) == 0 {
return nil, nil
}
mmap, err := unix.Mmap(int(f.Fd()), 0, int(size), unix.PROT_NONE, unix.MAP_SHARED)
if err != nil {
return nil, fmt.Errorf("could not mmap: %v", err)
}
vecsz := (size + int64(os.Getpagesize()) - 1) / int64(os.Getpagesize())
vec := make([]byte, vecsz)
mmap_ptr := uintptr(unsafe.Pointer(&mmap[0]))
size_ptr := uintptr(size)
vec_ptr := uintptr(unsafe.Pointer(&vec[0]))
ret, _, err := unix.Syscall(unix.SYS_MINCORE, mmap_ptr, size_ptr, vec_ptr)
if ret != 0 {
return nil, fmt.Errorf("syscall SYS_MINCORE failed: %v", err)
}
defer unix.Munmap(mmap)
value := new(Mincore)
for _, b := range vec {
if b%2 == 1 {
value.Cached++
} else {
value.Miss++
}
}
return value, nil
}
如何计算进程打开文件的 page cache 占用情况 ?
pcstat/proc/{pid}/maps/proc/{pid}/fd/*pgcacherlsof -p pidmmap + mincore/proc/{pid}/fd/*当然倒是有个笨方法, 直接遍历 rocksDB 数据目录的所有文件, 求出该进程的 page cache 使用情况.
如何计算当前系统里哪些文件空间占用最多 ?
拿到当前系统的所有进程列表, 然后遍历轮询拿到各个进程的已打开文件列表, 接着使用 mmap + mincore 求出文件在 page cache 的缓存信息. 再拿到所有进程所有文件的 page cache 缓存信息后, 做个排序输出即可.
pgcacher 使用文档
安装
source code compilation
git clone https://github.com/rfyiamcool/pgcacher.git
cd pgcacher
make build
sudo cp pgcacher /usr/local/bin/
pgcacher -h
github releases
pgcacher/usr/local/binuse binary directly
test pass on ubuntu, centos 7.x and centos 8.x.
wget xiaorui-cc.oss-cn-hangzhou.aliyuncs.com/files/pgcacher
chmod 777 pgcacher
\cp pgcacher /usr/local/bin
使用过程
pgcacher 命令帮助文档.
pgcacher <-json <-pps>|-terse|-default> <-nohdr> <-bname> file file file
-limit limit the number of files displayed, default: 500
-depth set the depth of dirs to scan, default: 0
-worker concurrency workers, default: 2
-pid show all open maps for the given pid
-top scan the open files of all processes, show the top few files that occupy the most memory space in the page cache, default: false
-lease-size ignore files smaller than the lastSize, such as '10MB' and '15GB'
-exclude-files exclude the specified files by wildcard, such as 'a*c?d' and '*xiaorui*,rfyiamcool'
-include-files only include the specified files by wildcard, such as 'a*c?d' and '*xiaorui?cc,rfyiamcool'
-json output will be JSON
-pps include the per-page information in the output (can be huge!)
-terse print terse machine-parseable output
-histo print a histogram using unicode block characters
-nohdr don't print the column header in terse or default format
-bname use basename(file) in the output (use for long paths)
-plain return data with no box characters
-unicode return data with unicode box characters
pgcacher 各种场景的使用过程.
查询该进程的 page cahce 使用占比, 附加了一个并发参数, 开启 5 个线程并发扫描.
# sudo pgcacher -pid=29260 -worker=5
+-------------------+----------------+-------------+----------------+-------------+---------+
| Name | Size │ Pages │ Cached Size │ Cached Pages│ Percent │
|-------------------+----------------+-------------+----------------+-------------+---------|
| /root/rui/file4g | 3.906G | 1024000 | 3.906G | 1024000 | 100.000 |
| /root/rui/file3g | 2.930G | 768000 | 2.930G | 768000 | 100.000 |
| /root/rui/file2g | 1.953G | 512000 | 1.953G | 512000 | 100.000 |
| /root/rui/file1g | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
| /root/rui/open_re | 1.791M | 459 | 1.791M | 459 | 100.000 |
|-------------------+----------------+-------------+----------------+-------------+---------|
│ Sum │ 9.767G │ 2560459 │ 9.767G │ 2560459 │ 100.000 │
+-------------------+----------------+-------------+----------------+-------------+---------+
查询多个文件在 page cache 的使用情况.
# dd if=/dev/urandom of=file1g bs=1M count=1000
# dd if=/dev/urandom of=file2g bs=1M count=2000
# dd if=/dev/urandom of=file3g bs=1M count=3000
# dd if=/dev/urandom of=file4g bs=1M count=4000
# cat file1g file2g file3g file4g > /dev/null
# sudo pgcacher file1g file2g file3g file4g
+--------+----------------+-------------+----------------+-------------+---------+
| Name | Size │ Pages │ Cached Size │ Cached Pages│ Percent │
|--------+----------------+-------------+----------------+-------------+---------|
| file4g | 3.906G | 1024000 | 3.906G | 1024000 | 100.000 |
| file3g | 2.930G | 768000 | 2.930G | 768000 | 100.000 |
| file2g | 1.953G | 512000 | 1.953G | 512000 | 100.000 |
| file1g | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
|--------+----------------+-------------+----------------+-------------+---------|
│ Sum │ 9.766G │ 2560000 │ 9.766G │ 2560000 │ 100.000 │
+--------+----------------+-------------+----------------+-------------+---------+
查看该目录下的所有文件在 page cache 缓存里的使用情况.
# sudo pgcacher /root/xiaorui.cc/*
+------------+----------------+-------------+----------------+-------------+---------+
| Name | Size │ Pages │ Cached Size │ Cached Pages│ Percent │
|------------+----------------+-------------+----------------+-------------+---------|
| file4g | 3.906G | 1024000 | 3.906G | 1024000 | 100.000 |
| file3g | 2.930G | 768000 | 2.930G | 768000 | 100.000 |
| file2g | 1.953G | 512000 | 1.953G | 512000 | 100.000 |
| testfile | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
| file1g | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
| pgcacher | 2.440M | 625 | 2.440M | 625 | 100.000 |
| open_re | 1.791M | 459 | 1.791M | 459 | 100.000 |
| cache.go | 19.576K | 5 | 19.576K | 5 | 100.000 |
| open_re.go | 644B | 1 | 644B | 1 | 100.000 |
| nohup.out | 957B | 1 | 957B | 1 | 100.000 |
|------------+----------------+-------------+----------------+-------------+---------|
│ Sum │ 10.746G │ 2817091 │ 10.746G │ 2817091 │ 100.000 │
+------------+----------------+-------------+----------------+-------------+---------+
查看当前系统下所有进程的已打开文件在 page cache 缓存里的使用情况.
# sudo pgcacher -top -limit 3
+------------------+----------------+-------------+----------------+-------------+---------+
| Name | Size │ Pages │ Cached Size │ Cached Pages│ Percent │
|------------------+----------------+-------------+----------------+-------------+---------|
| /root/rui/file4g | 3.906G | 1024000 | 3.906G | 1024000 | 100.000 |
| /root/rui/file3g | 2.930G | 768000 | 2.930G | 768000 | 100.000 |
| /root/rui/file2g | 1.953G | 512000 | 1.953G | 512000 | 100.000 |
|------------------+----------------+-------------+----------------+-------------+---------|
│ Sum │ 8.789G │ 2304000 │ 8.789G │ 2304000 │ 100.000 │
+------------------+----------------+-------------+----------------+-------------+---------+
递归遍历 4 层查看 aaa 目录所有子文件在 page cache 缓存里的使用情况.
# sudo pgcacher -depth=4 aaa/
+---------------------+----------------+-------------+----------------+-------------+---------+
| Name | Size │ Pages │ Cached Size │ Cached Pages│ Percent │
|---------------------+----------------+-------------+----------------+-------------+---------|
| aaa/a2g | 1.953G | 512000 | 1.953G | 512000 | 100.000 |
| aaa/bbb/ccc/ddd/d2g | 1.953G | 512000 | 1.940G | 508531 | 99.322 |
| aaa/bbb/ccc/c1g | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
| aaa/bbb/ccc/c2g | 1.953G | 512000 | 1000.000M | 256000 | 50.000 |
| aaa/bbb/ccc/ddd/d1g | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
| aaa/a1g | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
| aaa/bbb/bbb1g | 1000.000M | 256000 | 1000.000M | 256000 | 100.000 |
| aaa/bbb/bbb2g | 1.953G | 512000 | 1000.000M | 256000 | 50.000 |
|---------------------+----------------+-------------+----------------+-------------+---------|
│ Sum │ 11.719G │ 3072000 │ 9.752G │ 2556531 │ 83.220 │
+---------------------+----------------+-------------+----------------+-------------+---------+
总结
如何分析文件在 page cache 缓存中的使用情况pgcacher