linux 相关

基本操作

linux 分区

fdisk/mkfs/mount

使用 fdisk 给磁盘分区并挂载

列出磁盘信息

sudo fdisk -l [device]

进入交互模式

sudo fdisk [device]

m #帮助
p #打印当前分区表
g #创建gpt分区表
n #创建分区

fdisk 创建分区后

• 通过mkfs.ext4 /dev/sdb1创建文件 ext4 系统（p.s. 注意：不要使用 sdb，而是 sdb1 指定分区）

mkfs.bfs     mkfs.cramfs  mkfs.ext2    mkfs.ext3    mkfs.ext4    mkfs.fat     mkfs.minix   mkfs.msdos   mkfs.ntfs    mkfs.vfat

• 通过lsblk -f显示 UUID
  • -f表示打印文件系统

• 在/etc/fstab中添加挂载条目，然后mount -a表示挂载 fstab 中的所有条目 示例

  UUID=b7ee63d0-6cea-4615-b7ec-63449c8fa566 /mnt/Disk1    ext4    defaults 0 0
  

• 修改挂载点目录权限，默认只有 root 用户可以访问

swap

sudo dd if=/dev/zero of=/swapfile bs=1G count=16
sudo mkswap swapfile
sudo swapon swapfile

#fstab 
/swapfile none swap sw 0 0

mount

Fstab - Community Help Wiki (ubuntu.com)

mount 只能由 root 用户操作，导致 mount 的目录所有者为 root，普通用户无法访问

ext4 mount 选项中也没有指定 uid 的方式挂载（不像smb挂载时）ext4(5) - Linux manual page (man7.org)

其实上面根本不是问题，解决办法是：mount 之后使用 chown 命令将挂载的目录改为普通用户。由于用户信息是写入文件系统的，所以之后挂载时文件权限仍然是普通用户。

扩展分区 expand partition

ext4可以在线扩展分区（注意shrink是不行的）

方法列举：

• 使用gparted（简单，但是遇到device busy的情况也束手无策，不如命令行时知道问题出在哪）
• fdisk：需要先删除分区（注意不要删除文件系统signature），然后重新创建，保证起始sector相同。
• parted：有resize命令，扩展分区时更容易。但是由于每步操作都会立即执行，没有fdisk那样有安全感，最好还是不使用。
• resize2fs：默认是扩展所有空间。

fdisk

yfy@u22-game:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        98G   17G   76G  19% /
/dev/sda2       512M  6.1M  506M   2% /boot/efi

root@u22-game:/home/yfy# fdisk /dev/sda

Command (m for help): p

Disk /dev/sda: 120 GiB, 128849018880 bytes, 251658240 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 43AAA442-6CCE-4AC2-9389-0EE427839D73

Device       Start       End   Sectors  Size Type
/dev/sda1     2048      4095      2048    1M BIOS boot
/dev/sda2     4096   1054719   1050624  513M EFI System
/dev/sda3  1054720 209713151 208658432 99.5G Linux filesystem

Command (m for help): F
Unpartitioned space /dev/sda: 20 GiB, 21475868160 bytes, 41945055 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

    Start       End  Sectors Size
209713152 251658206 41945055  20G

Command (m for help): d
Partition number (1-3, default 3):

Partition 3 has been deleted.

Command (m for help): n
Partition number (3-128, default 3):
First sector (1054720-251658206, default 1054720):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (1054720-251658206, default 251658206): +110G

Created a new partition 3 of type 'Linux filesystem' and of size 110 GiB.
Partition #3 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: N

Command (m for help): w

The partition table has been altered.
Syncing disks.

root@u22-game:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        98G   17G   76G  19% /

root@u22-game:~# resize2fs /dev/sda3
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/sda3 is mounted on /; on-line resizing required
old_desc_blocks = 13, new_desc_blocks = 14
The filesystem on /dev/sda3 is now 28835840 (4k) blocks long.

root@u22-game:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3       108G   17G   86G  17% /

parted How to extend an EXT4 partition using parted and resize2fs - Knowledgebase - Cloud Services Store Limited resize2fs: 6.3. Resizing an Ext4 File System Red Hat Enterprise Linux 6 | Red Hat Customer Portal

千万注意 start sector 需要保持一样。fdisk 有时侯默认并不会保持不变。

Device or resource busy 怎么办

Command (m for help): w
The partition table has been altered.
Failed to remove partition 3 from system: Device or resource busy
Failed to remove partition 4 from system: Device or resource busy
Failed to update system information about partition 2: Device or resource busy
Failed to add partition 3 to system: Device or resource busy

The kernel still uses the old partitions. The new table will be used at the next reboot.
Syncing disks.

我重启后发现，分区已经更新了。resize2fs 即可

mkswap

mkswap /dev/sdaX
swapon /dev/sdaX

swapon --show

UUID=xxx none swap sw 0 0

linux 挂载

losetup

losetup is used to associate loop devices with regular files or block devices, to detach loop devices, and to query the status of a loop device.

losetup   # 打印所有挂载的loopback设备
losetup -f --show default.img   # 找到第一个未使用的loop设备，并打印，然后挂载default.img
losetup -d loopdev      # detach指定loop设备
-r, --read-only   # 只读挂载

磁盘镜像

用于挂载包含分区表的磁盘镜像

sudo kpartx -av 20240702_NanoKVM_Rev1_1_0.img  # 创建 /dev/mapper/loopXpY

sudo kpartx -dv 20240702_NanoKVM_Rev1_1_0.img

sudoer 设置

vim /etc/sudoers

将用户添加到 admin 或 sudo 用户组即可

免密码：

%sudo   ALL=(ALL:ALL) NOPASSWD: ALL

sudoers 每项含义（待续）

linux sshd 设置

允许 ssh 登录 root

先改为 yes，然后 ssh-copy-id 后，改为禁用密码

PermitRootLogin prohibit-password
#PermitRootLogin yes

禁用密码登录

PasswordAuthentication no

其它选项

PermitRootLogin prohibit-password
PubkeyAuthentication yes    # 允许公钥登录
PubkeyAcceptedKeyTypes=+ssh-rsa
PasswordAuthentication no   # 禁用密码登录
ChallengeResponseAuthentication no

Accepted password for yfy from 192.168.36.215 port 50342 ssh2

Accepted publickey for yfy from 192.168.36.180 port 26464 ssh2: RSA SHA256:z9yheNzateaIikTVkTXN1lmMPRIsp+H0ssbhU4Q8Kfg
pam_unix(sshd:session): session opened for user yfy(uid=1000) by (uid=0)

move running process to tmux

Move a running process into a tmux session | ./xai.sh

测试磁盘速度

dd

使用 dd 简单测试速度。可以区分 usb 2.0 和 3.0；可以见识 PCIe 4.0 ssd 的速度。但是要进行更准确的测速，最好使用 fio。

注意需要开启 direct，否则需要从 userspace 复制到 kernel buffer，严重影响速度。block size 对速度影响很大。 Why is dd with the 'direct' (O_DIRECT) flag so dramatically faster?

• bs 影响很大
• /dev/null不需要使用oflag=direct

root@n5105-pve ➜  dd of=/dev/null if=/mnt/wd-passport/fio.img bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 21.2453 s, 98.7 MB/s
root@n5105-pve ➜  dd if=/dev/zero of=/mnt/wd-passport/fio.img bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 21.7685 s, 96.3 MB/s
root@n5105-pve ➜  dd if=/dev/zero of=/mnt/wd-passport/fio.img bs=1M count=2000 oflag=direct
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 20.9127 s, 100 MB/s

fio

To benchmark persistent disk performance, use Flexible I/O tester (FIO) instead of other disk benchmarking tools such as dd. By default, dd uses a very low I/O queue depth, so it is difficult to ensure that the benchmark is generating a sufficient number of I/Os and bytes to accurately test disk performance.

• I/O type
  • direct, buffered, atomic. : O_DIRECT, O_ATOMIC
  • readwrite=str, rw=str
    • read: Sequential reads.
    • write: Sequential writes.
    • randread: Random reads.
    • randwrite: Random writes.
    • rw,readwrite: Sequential mixed reads and writes
    • randrw: Random mixed reads and writes.
    • trim, randtrim, trimwrite
• Block size
  • bs: The block size in bytes used for I/O units
• Buffer and Memory
  • sync=str
    • sync: Use synchronous file IO. O_SYNC
    • dsync: Use synchronous data IO. For the majority of I/O engines, this means using O_DSYNC.
• I/O size
  • size: The total size of file I/O for each thread of this job. Fio will run until this many bytes has been transferred, unless runtime is limited by other options (such as runtime, for instance, or increased/decreased by io_size).
• I/O engine
  • ioengine=str
    • psync: Basic pread(2) or pwrite(2) I/O. Default on all supported operating systems except for Windows
    • libaio: Linux native asynchronous I/O. Note that Linux may only support queued behavior with non-buffered I/O (set direct=1 or buffer=0). This engine defines engine specific options.
• I/O depth
  • iodepth=int
    • Number of I/O units to keep in flight against the file. Note that increasing iodepth beyond 1 will not affect synchronous ioengines (except for small degrees when verify_async is in use). Even async engines may impose OS restrictions causing the desired depth not to be achieved. This may happen on Linux when using libaio and not setting direct=1', since buffered I/O is not async on that OS. Keep an eyeon the I/O depth distribution in the fio output to verify that the achieved depth is as expected. Default: 1.
• chatgpt4:
  • 当 iodepth 大于 1 时，fio 会尝试同时发起多个 I/O 请求，以此来模拟并发 I/O 负载。这对于评估存储系统在高并发场景下的性能非常有用。
  • 对于同步 I/O 引擎（如 psync），iodepth 大于 1 的设置影响不大，因为同步 I/O 操作会按顺序一个接一个地执行。但是，在使用 verify_async 选项时，即使是同步引擎，将 iodepth 设置得稍微大于 1 也可能有轻微的影响。
  • 对于异步 I/O 引擎（如 libaio），增加 iodepth 可以显著提高并发度和吞吐量。然而，操作系统和其它因素可能限制实际达到的深度，特别是在不使用直接 I/O（direct=1）的情况下。在 Linux 中，如果不设置 direct=1，libaio 使用的是缓冲 I/O，而这在 Linux 上并不是异步的。
  • fio 输出中的 I/O 深度分布可以帮助你验证实际达到的深度是否符合预期。

stonewall wait_for_previous Wait for preceding jobs in the job file to exit, before starting this one. Can be used to insert serialization points in the job file. A stone wall also implies starting a new reporting group, see group_reporting. Optionally you can use stonewall=0 to disable or stonewall=1 to enable it.

配置文件

• global 公共部分，影响后面每一个 job
• 默认 job 是并行的，需要添加 stonewall等待前面一个完成
• filename 指定共同使用一个文件（否则会根据 job 名自动生成）

sudo fio --directory=/root/ --filename=test.bin ssd-win-likely.fio

vim ssd-win-likely.fio #模仿windows下diskmark，测顺序读写和4k随机
[global]
ioengine=libaio
size=10g
direct=1
runtime=60

[seq-read-4m]
bs=4M
iodepth=32
rw=read
stonewall

[seq-write-4m]
bs=4M
iodepth=32
rw=write
stonewall

[rand-read-4k]
bs=4k
iodepth=4
rw=randread
stonewall

[rand-write-4k]
bs=4k
iodepth=4
rw=randwrite
stonewall

机械硬盘例子

• dd read 时需要添加-iflag=direct，否则测得速度 11GB/s
  • O_DIRECT：绕过系统的页缓存，更准确地衡量磁盘本身的性能（
• 顺序读时，测试/dev/sda1 和 test.img 速度是不同的 120 -> 170 MB/s
• iodepth 影响 1 -> 8: 155 -> 180 MB/s
• 在我测试的机械硬盘上，增加 jobnums 提升没有 iodepth 明显

fio --filename=/dev/sda1 --bs=1M --direct=1 --iodepth=8 --numjobs 1 --group_reporting --size=1g --runtime=60 --name SeqRead --rw=read --ioengine=libaio --readonly
SeqRead: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=166MiB/s][r=166 IOPS][eta 00m:00s]
SeqRead: (groupid=0, jobs=1): err= 0: pid=884774: Wed Mar 13 17:09:47 2024
  read: IOPS=168, BW=168MiB/s (177MB/s)(1024MiB/6078msec)
    slat (usec): min=28, max=339, avg=46.65, stdev=27.34
    clat (msec): min=30, max=240, avg=45.46, stdev=25.37
     lat (msec): min=30, max=240, avg=45.51, stdev=25.37
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   40], 10.00th=[   40], 20.00th=[   40],
     | 30.00th=[   40], 40.00th=[   40], 50.00th=[   42], 60.00th=[   43],
     | 70.00th=[   43], 80.00th=[   45], 90.00th=[   45], 95.00th=[   46],
     | 99.00th=[  234], 99.50th=[  241], 99.90th=[  241], 99.95th=[  241],
     | 99.99th=[  241]
   bw (  KiB/s): min=88064, max=202752, per=99.72%, avg=172032.00, stdev=40726.61, samples=12
   iops        : min=   86, max=  198, avg=168.00, stdev=39.77, samples=12
  lat (msec)   : 50=96.09%, 100=1.56%, 250=2.34%
  cpu          : usr=0.20%, sys=4.67%, ctx=1444, majf=0, minf=2059
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s), io=1024MiB (1074MB), run=6078-6078msec

Disk stats (read/write):
  sda: ios=2108/1, merge=4/0, ticks=106583/0, in_queue=106584, util=95.24%

例子

aigo P7000Z 2TB pcie 4.0 (标称读 7350, 写 6750 MB/s) 输出

单独分区

root@ryzen-pve ➜  fio fio --filename=/dev/nvme0n1p5 ssd-win-likely.fio
seq-read-4m: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
seq-write-4m: (g=1): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
rand-read-4k: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write-4k: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write-4k-q32: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.33
Starting 5 processes
Jobs: 1 (f=0): [_(4),f(1)][100.0%][w=533MiB/s][w=136k IOPS][eta 00m:00s]
seq-read-4m: (groupid=0, jobs=1): err= 0: pid=220879: Mon Mar 11 21:01:39 2024
  read: IOPS=1760, BW=7043MiB/s (7385MB/s)(10.0GiB/1454msec)
    slat (usec): min=74, max=1561, avg=99.41, stdev=117.35
    clat (usec): min=2627, max=35094, avg=17836.27, stdev=1911.51
     lat (usec): min=2712, max=36273, avg=17935.68, stdev=1922.51
    clat percentiles (usec):
     |  1.00th=[ 9503],  5.00th=[17695], 10.00th=[17695], 20.00th=[17957],
     | 30.00th=[17957], 40.00th=[17957], 50.00th=[17957], 60.00th=[17957],
     | 70.00th=[17957], 80.00th=[17957], 90.00th=[17957], 95.00th=[17957],
     | 99.00th=[25822], 99.50th=[30802], 99.90th=[34341], 99.95th=[34866],
     | 99.99th=[34866]
   bw (  MiB/s): min= 6934, max= 7136, per=99.90%, avg=7035.49, stdev=142.14, samples=2
   iops        : min= 1733, max= 1784, avg=1758.50, stdev=36.06, samples=2
  lat (msec)   : 4=0.35%, 10=0.70%, 20=97.34%, 50=1.60%
  cpu          : usr=0.55%, sys=18.10%, ctx=2524, majf=7, minf=32778
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq-write-4m: (groupid=1, jobs=1): err= 0: pid=220884: Mon Mar 11 21:01:39 2024
  write: IOPS=1598, BW=6392MiB/s (6703MB/s)(10.0GiB/1602msec); 0 zone resets
    slat (usec): min=90, max=508, avg=139.03, stdev=26.46
    clat (usec): min=7196, max=38619, avg=19847.21, stdev=1776.10
     lat (usec): min=7313, max=38753, avg=19986.24, stdev=1777.09
    clat percentiles (usec):
     |  1.00th=[19268],  5.00th=[19268], 10.00th=[19268], 20.00th=[19268],
     | 30.00th=[19268], 40.00th=[19268], 50.00th=[19268], 60.00th=[19530],
     | 70.00th=[19530], 80.00th=[20055], 90.00th=[20317], 95.00th=[23725],
     | 99.00th=[23987], 99.50th=[31327], 99.90th=[37487], 99.95th=[38011],
     | 99.99th=[38536]
   bw (  MiB/s): min= 6335, max= 6448, per=100.00%, avg=6394.61, stdev=56.28, samples=3
   iops        : min= 1583, max= 1612, avg=1598.33, stdev=14.57, samples=3
  lat (msec)   : 10=0.23%, 20=76.21%, 50=23.55%
  cpu          : usr=16.18%, sys=7.18%, ctx=2560, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand-read-4k: (groupid=2, jobs=1): err= 0: pid=220895: Mon Mar 11 21:01:39 2024
  read: IOPS=88.1k, BW=344MiB/s (361MB/s)(2048MiB/5950msec)
    slat (nsec): min=1700, max=316263, avg=2375.65, stdev=1120.83
    clat (usec): min=8, max=5976, avg=42.63, stdev=17.74
     lat (usec): min=16, max=5998, avg=45.01, stdev=17.82
    clat percentiles (usec):
     |  1.00th=[   39],  5.00th=[   39], 10.00th=[   40], 20.00th=[   40],
     | 30.00th=[   41], 40.00th=[   41], 50.00th=[   42], 60.00th=[   43],
     | 70.00th=[   43], 80.00th=[   45], 90.00th=[   47], 95.00th=[   50],
     | 99.00th=[   64], 99.50th=[   67], 99.90th=[   87], 99.95th=[  109],
     | 99.99th=[  182]
   bw (  KiB/s): min=350364, max=355856, per=100.00%, avg=352845.55, stdev=1586.19, samples=11
   iops        : min=87591, max=88964, avg=88211.27, stdev=396.59, samples=11
  lat (usec)   : 10=0.01%, 20=0.02%, 50=95.51%, 100=4.41%, 250=0.05%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 10=0.01%
  cpu          : usr=9.82%, sys=33.38%, ctx=263427, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rand-write-4k: (groupid=3, jobs=1): err= 0: pid=221144: Mon Mar 11 21:01:39 2024
  write: IOPS=209k, BW=818MiB/s (857MB/s)(2048MiB/2505msec); 0 zone resets
    slat (nsec): min=1800, max=86641, avg=2893.21, stdev=904.34
    clat (usec): min=3, max=147467, avg=15.82, stdev=421.90
     lat (usec): min=6, max=147469, avg=18.71, stdev=421.90
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   13],
     | 70.00th=[   13], 80.00th=[   13], 90.00th=[   14], 95.00th=[   17],
     | 99.00th=[   20], 99.50th=[   25], 99.90th=[  396], 99.95th=[ 1106],
     | 99.99th=[ 2245]
   bw (  KiB/s): min=599918, max=956832, per=100.00%, avg=853261.50, stdev=170280.21, samples=4
   iops        : min=149979, max=239208, avg=213315.25, stdev=42570.30, samples=4
  lat (usec)   : 4=0.01%, 10=0.53%, 20=98.61%, 50=0.57%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.19%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=17.97%, sys=63.86%, ctx=1540, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rand-write-4k-q32: (groupid=4, jobs=1): err= 0: pid=221156: Mon Mar 11 21:01:39 2024
  write: IOPS=128k, BW=502MiB/s (526MB/s)(2048MiB/4082msec); 0 zone resets
    slat (nsec): min=1780, max=92291, avg=2970.76, stdev=1119.23
    clat (usec): min=8, max=220221, avg=245.77, stdev=2559.24
     lat (usec): min=12, max=220223, avg=248.74, stdev=2559.23
    clat percentiles (usec):
     |  1.00th=[   106],  5.00th=[   115], 10.00th=[   116], 20.00th=[   117],
     | 30.00th=[   118], 40.00th=[   119], 50.00th=[   120], 60.00th=[   121],
     | 70.00th=[   122], 80.00th=[   124], 90.00th=[   130], 95.00th=[   147],
     | 99.00th=[   437], 99.50th=[  1483], 99.90th=[ 32900], 99.95th=[ 35914],
     | 99.99th=[117965]
   bw (  KiB/s): min= 2648, max=967056, per=98.42%, avg=505654.00, stdev=477841.95, samples=8
   iops        : min=  662, max=241764, avg=126413.50, stdev=119460.49, samples=8
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.01%, 100=0.71%, 250=97.01%
  lat (usec)   : 500=1.36%, 750=0.07%, 1000=0.06%
  lat (msec)   : 2=0.46%, 4=0.02%, 10=0.01%, 50=0.27%, 100=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=10.44%, sys=39.40%, ctx=385, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=7043MiB/s (7385MB/s), 7043MiB/s-7043MiB/s (7385MB/s-7385MB/s), io=10.0GiB (10.7GB), run=1454-1454msec

Run status group 1 (all jobs):
  WRITE: bw=6392MiB/s (6703MB/s), 6392MiB/s-6392MiB/s (6703MB/s-6703MB/s), io=10.0GiB (10.7GB), run=1602-1602msec

Run status group 2 (all jobs):
   READ: bw=344MiB/s (361MB/s), 344MiB/s-344MiB/s (361MB/s-361MB/s), io=2048MiB (2147MB), run=5950-5950msec

Run status group 3 (all jobs):
  WRITE: bw=818MiB/s (857MB/s), 818MiB/s-818MiB/s (857MB/s-857MB/s), io=2048MiB (2147MB), run=2505-2505msec

Run status group 4 (all jobs):
  WRITE: bw=502MiB/s (526MB/s), 502MiB/s-502MiB/s (526MB/s-526MB/s), io=2048MiB (2147MB), run=4082-4082msec

Disk stats (read/write):
  nvme0n1: ios=544852/1011542, merge=0/0, ticks=373885/471120, in_queue=845097, util=91.87%

zfs dataset, compression=off

root@ryzen-pve ➜  fio fio --filename=/pool0/test/fio.img ssd-win-likely.fio
seq-read-4m: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
seq-write-4m: (g=1): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
rand-read-4k: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write-4k: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write-4k-q32: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.33
Starting 5 processes
Jobs: 1 (f=1): [_(4),w(1)][97.8%][w=271MiB/s][w=69.3k IOPS][eta 00m:01s]
seq-read-4m: (groupid=0, jobs=1): err= 0: pid=57763: Mon Mar 11 23:41:55 2024
  read: IOPS=550, BW=2201MiB/s (2308MB/s)(10.0GiB/4653msec)
    slat (usec): min=858, max=76147, avg=1814.40, stdev=1574.46
    clat (usec): min=2, max=135775, avg=55723.26, stdev=11488.49
     lat (usec): min=865, max=137572, avg=57537.66, stdev=11732.56
    clat percentiles (msec):
     |  1.00th=[   29],  5.00th=[   49], 10.00th=[   50], 20.00th=[   51],
     | 30.00th=[   52], 40.00th=[   53], 50.00th=[   54], 60.00th=[   56],
     | 70.00th=[   58], 80.00th=[   59], 90.00th=[   63], 95.00th=[   66],
     | 99.00th=[  127], 99.50th=[  131], 99.90th=[  136], 99.95th=[  136],
     | 99.99th=[  136]
   bw (  MiB/s): min= 1832, max= 2520, per=98.72%, avg=2172.51, stdev=252.75, samples=9
   iops        : min=  458, max=  630, avg=543.00, stdev=63.30, samples=9
  lat (usec)   : 4=0.04%, 1000=0.04%
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.27%, 20=0.31%, 50=15.27%
  lat (msec)   : 100=82.42%, 250=1.52%
  cpu          : usr=0.00%, sys=92.97%, ctx=1245, majf=0, minf=32779
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq-write-4m: (groupid=1, jobs=1): err= 0: pid=69794: Mon Mar 11 23:41:55 2024
  write: IOPS=696, BW=2786MiB/s (2921MB/s)(10.0GiB/3676msec); 0 zone resets
    slat (usec): min=695, max=34437, avg=1431.68, stdev=1340.37
    clat (usec): min=2, max=121796, avg=44214.86, stdev=17037.26
     lat (usec): min=1049, max=124310, avg=45646.54, stdev=17349.80
    clat percentiles (msec):
     |  1.00th=[   24],  5.00th=[   26], 10.00th=[   27], 20.00th=[   32],
     | 30.00th=[   34], 40.00th=[   37], 50.00th=[   41], 60.00th=[   45],
     | 70.00th=[   50], 80.00th=[   54], 90.00th=[   70], 95.00th=[   83],
     | 99.00th=[   92], 99.50th=[  112], 99.90th=[  120], 99.95th=[  122],
     | 99.99th=[  123]
   bw (  MiB/s): min= 2492, max= 3088, per=97.73%, avg=2722.52, stdev=207.48, samples=7
   iops        : min=  623, max=  772, avg=680.43, stdev=51.94, samples=7
  lat (usec)   : 4=0.04%
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.23%, 20=0.39%, 50=71.80%
  lat (msec)   : 100=26.64%, 250=0.78%
  cpu          : usr=10.37%, sys=76.05%, ctx=3339, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand-read-4k: (groupid=2, jobs=1): err= 0: pid=83243: Mon Mar 11 23:41:55 2024
  read: IOPS=44.7k, BW=174MiB/s (183MB/s)(2048MiB/11739msec)
    slat (usec): min=3, max=7135, avg=21.40, stdev=37.28
    clat (nsec): min=1490, max=14725k, avg=67768.36, stdev=89963.24
     lat (usec): min=6, max=14826, avg=89.17, stdev=108.61
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   30], 10.00th=[   43], 20.00th=[   45],
     | 30.00th=[   48], 40.00th=[   58], 50.00th=[   60], 60.00th=[   61],
     | 70.00th=[   62], 80.00th=[   65], 90.00th=[   71], 95.00th=[  182],
     | 99.00th=[  334], 99.50th=[  449], 99.90th=[  644], 99.95th=[ 1012],
     | 99.99th=[ 2573]
   bw (  KiB/s): min=23187, max=272800, per=97.63%, avg=174420.39, stdev=69350.81, samples=23
   iops        : min= 5796, max=68200, avg=43605.04, stdev=17337.76, samples=23
  lat (usec)   : 2=0.01%, 10=0.01%, 20=2.26%, 50=30.78%, 100=60.63%
  lat (usec)   : 250=4.03%, 500=2.14%, 750=0.07%, 1000=0.03%
  lat (msec)   : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=3.54%, sys=78.68%, ctx=14965, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rand-write-4k: (groupid=3, jobs=1): err= 0: pid=86147: Mon Mar 11 23:41:55 2024
  write: IOPS=42.3k, BW=165MiB/s (173MB/s)(2048MiB/12393msec); 0 zone resets
    slat (usec): min=5, max=53972, avg=22.47, stdev=88.41
    clat (nsec): min=1030, max=54037k, avg=71553.91, stdev=159275.72
     lat (usec): min=7, max=54066, avg=94.03, stdev=186.91
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[   23], 10.00th=[   24], 20.00th=[   26],
     | 30.00th=[   33], 40.00th=[   40], 50.00th=[   48], 60.00th=[   61],
     | 70.00th=[   75], 80.00th=[   99], 90.00th=[  149], 95.00th=[  184],
     | 99.00th=[  330], 99.50th=[  408], 99.90th=[  766], 99.95th=[ 1336],
     | 99.99th=[ 2343]
   bw (  KiB/s): min=95843, max=376008, per=97.41%, avg=164831.38, stdev=81612.27, samples=24
   iops        : min=23960, max=94002, avg=41207.75, stdev=20403.10, samples=24
  lat (usec)   : 2=0.01%, 10=0.01%, 50=51.73%, 100=28.46%, 250=17.77%
  lat (usec)   : 500=1.74%, 750=0.19%, 1000=0.03%
  lat (msec)   : 2=0.06%, 4=0.01%, 10=0.01%, 100=0.01%
  cpu          : usr=4.78%, sys=85.06%, ctx=7976, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rand-write-4k-q32: (groupid=4, jobs=1): err= 0: pid=134803: Mon Mar 11 23:41:55 2024
  write: IOPS=49.8k, BW=194MiB/s (204MB/s)(2048MiB/10530msec); 0 zone resets
    slat (usec): min=6, max=3191, avg=18.89, stdev=22.34
    clat (nsec): min=1350, max=6289.1k, avg=623232.18, stdev=438263.27
     lat (usec): min=7, max=6358, avg=642.12, stdev=451.25
    clat percentiles (usec):
     |  1.00th=[  241],  5.00th=[  253], 10.00th=[  269], 20.00th=[  318],
     | 30.00th=[  379], 40.00th=[  433], 50.00th=[  498], 60.00th=[  578],
     | 70.00th=[  660], 80.00th=[  783], 90.00th=[ 1139], 95.00th=[ 1500],
     | 99.00th=[ 2442], 99.50th=[ 2802], 99.90th=[ 3458], 99.95th=[ 3785],
     | 99.99th=[ 4817]
   bw (  KiB/s): min=119688, max=355104, per=99.77%, avg=198711.14, stdev=71924.36, samples=21
   iops        : min=29922, max=88776, avg=49677.76, stdev=17981.10, samples=21
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=4.34%, 500=46.12%, 750=27.41%, 1000=9.30%
  lat (msec)   : 2=10.66%, 4=2.14%, 10=0.03%
  cpu          : usr=5.88%, sys=90.77%, ctx=2681, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2201MiB/s (2308MB/s), 2201MiB/s-2201MiB/s (2308MB/s-2308MB/s), io=10.0GiB (10.7GB), run=4653-4653msec

Run status group 1 (all jobs):
  WRITE: bw=2786MiB/s (2921MB/s), 2786MiB/s-2786MiB/s (2921MB/s-2921MB/s), io=10.0GiB (10.7GB), run=3676-3676msec

Run status group 2 (all jobs):
   READ: bw=174MiB/s (183MB/s), 174MiB/s-174MiB/s (183MB/s-183MB/s), io=2048MiB (2147MB), run=11739-11739msec

Run status group 3 (all jobs):
  WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=2048MiB (2147MB), run=12393-12393msec

Run status group 4 (all jobs):
  WRITE: bw=194MiB/s (204MB/s), 194MiB/s-194MiB/s (204MB/s-204MB/s), io=2048MiB (2147MB), run=10530-10530msec

samsung pm9a1 zfs dataset compression=off

root@ryzen-pve ➜  fio fio --filename=/rpool/test/fio.img ssd-win-likely.fio
seq-read-4m: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
seq-write-4m: (g=1): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
rand-read-4k: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write-4k: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write-4k-q32: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.33
Starting 5 processes
Jobs: 1 (f=1): [_(4),w(1)][100.0%][w=284MiB/s][w=72.7k IOPS][eta 00m:00s]
seq-read-4m: (groupid=0, jobs=1): err= 0: pid=184273: Mon Mar 11 23:44:05 2024
  read: IOPS=621, BW=2485MiB/s (2606MB/s)(10.0GiB/4121msec)
    slat (usec): min=831, max=12071, avg=1606.85, stdev=272.83
    clat (usec): min=2, max=74006, avg=49318.40, stdev=4649.68
     lat (usec): min=1035, max=86078, avg=50925.25, stdev=4761.72
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[47449], 10.00th=[47973], 20.00th=[47973],
     | 30.00th=[48497], 40.00th=[48497], 50.00th=[49021], 60.00th=[49546],
     | 70.00th=[50070], 80.00th=[50594], 90.00th=[52167], 95.00th=[53740],
     | 99.00th=[60031], 99.50th=[65274], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[73925]
   bw (  MiB/s): min= 2143, max= 2597, per=99.06%, avg=2461.56, stdev=137.78, samples=8
   iops        : min=  535, max=  649, avg=615.25, stdev=34.68, samples=8
  lat (usec)   : 4=0.04%
  lat (msec)   : 2=0.04%, 4=0.12%, 10=0.23%, 20=0.31%, 50=70.16%
  lat (msec)   : 100=29.10%
  cpu          : usr=0.07%, sys=99.08%, ctx=580, majf=0, minf=32781
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq-write-4m: (groupid=1, jobs=1): err= 0: pid=191809: Mon Mar 11 23:44:05 2024
  write: IOPS=670, BW=2683MiB/s (2813MB/s)(10.0GiB/3817msec); 0 zone resets
    slat (usec): min=667, max=22092, avg=1487.61, stdev=1056.33
    clat (usec): min=2, max=155391, avg=46053.12, stdev=20622.13
     lat (usec): min=1709, max=159557, avg=47540.73, stdev=21111.71
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   32], 10.00th=[   33], 20.00th=[   34],
     | 30.00th=[   35], 40.00th=[   35], 50.00th=[   36], 60.00th=[   38],
     | 70.00th=[   56], 80.00th=[   59], 90.00th=[   69], 95.00th=[   85],
     | 99.00th=[  140], 99.50th=[  146], 99.90th=[  155], 99.95th=[  155],
     | 99.99th=[  157]
   bw (  MiB/s): min= 1840, max= 3198, per=98.34%, avg=2638.15, stdev=642.29, samples=7
   iops        : min=  460, max=  799, avg=659.29, stdev=160.33, samples=7
  lat (usec)   : 4=0.04%
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.12%, 20=0.20%, 50=66.41%
  lat (msec)   : 100=30.00%, 250=3.16%
  cpu          : usr=8.39%, sys=64.49%, ctx=9529, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rand-read-4k: (groupid=2, jobs=1): err= 0: pid=197068: Mon Mar 11 23:44:05 2024
  read: IOPS=43.7k, BW=171MiB/s (179MB/s)(2048MiB/12004msec)
    slat (usec): min=3, max=11718, avg=21.97, stdev=77.90
    clat (nsec): min=1000, max=15963k, avg=69205.71, stdev=188845.88
     lat (usec): min=4, max=18618, avg=91.17, stdev=242.55
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   30], 10.00th=[   41], 20.00th=[   44],
     | 30.00th=[   48], 40.00th=[   56], 50.00th=[   58], 60.00th=[   59],
     | 70.00th=[   61], 80.00th=[   64], 90.00th=[   70], 95.00th=[  172],
     | 99.00th=[  326], 99.50th=[  420], 99.90th=[ 1029], 99.95th=[ 4424],
     | 99.99th=[ 8979]
   bw (  KiB/s): min= 2824, max=346528, per=99.96%, avg=174638.37, stdev=87492.72, samples=24
   iops        : min=  706, max=86632, avg=43659.58, stdev=21873.19, samples=24
  lat (usec)   : 2=0.01%, 10=0.01%, 20=2.24%, 50=30.92%, 100=60.85%
  lat (usec)   : 250=3.51%, 500=2.35%, 750=0.02%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.03%, 10=0.05%, 20=0.01%
  cpu          : usr=3.26%, sys=74.40%, ctx=15963, majf=0, minf=16
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=524288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rand-write-4k: (groupid=3, jobs=1): err= 0: pid=198074: Mon Mar 11 23:44:05 2024
  write: IOPS=53.0k, BW=207MiB/s (217MB/s)(2048MiB/9897msec); 0 zone resets
    slat (usec): min=5, max=28448, avg=17.86, stdev=95.73
    clat (nsec): min=1040, max=28542k, avg=57219.95, stdev=171032.03
     lat (usec): min=7, max=28566, avg=75.08, stdev=200.03
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[   24], 10.00th=[   24], 20.00th=[   25],
     | 30.00th=[   26], 40.00th=[   29], 50.00th=[   34], 60.00th=[   44],
     | 70.00th=[   53], 80.00th=[   67], 90.00th=[   89], 95.00th=[  135],
     | 99.00th=[  289], 99.50th=[  367], 99.90th=[ 2409], 99.95th=[ 3425],
     | 99.99th=[ 5080]
   bw (  KiB/s): min=67664, max=391400, per=100.00%, avg=212579.53, stdev=107195.48, samples=19
   iops        : min=16916, max=97850, avg=53144.89, stdev=26798.83, samples=19
  lat (usec)   : 2=0.01%, 10=0.01%, 50=67.54%, 100=24.96%, 250=6.02%
  lat (usec)   : 500=1.14%, 750=0.08%, 1000=0.03%
  lat (msec)   : 2=0.10%, 4=0.09%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=5.51%, sys=79.49%, ctx=4215, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rand-write-4k-q32: (groupid=4, jobs=1): err= 0: pid=207247: Mon Mar 11 23:44:05 2024
  write: IOPS=62.1k, BW=243MiB/s (254MB/s)(2048MiB/8439msec); 0 zone resets
    slat (usec): min=5, max=12157, avg=15.01, stdev=27.59
    clat (nsec): min=1430, max=16166k, avg=499501.55, stdev=426869.67
     lat (usec): min=7, max=16286, avg=514.51, stdev=439.20
    clat percentiles (usec):
     |  1.00th=[  239],  5.00th=[  247], 10.00th=[  255], 20.00th=[  281],
     | 30.00th=[  314], 40.00th=[  351], 50.00th=[  388], 60.00th=[  437],
     | 70.00th=[  510], 80.00th=[  619], 90.00th=[  783], 95.00th=[  947],
     | 99.00th=[ 2638], 99.50th=[ 3032], 99.90th=[ 3949], 99.95th=[ 4555],
     | 99.99th=[13304]
   bw (  KiB/s): min=159454, max=351896, per=96.67%, avg=240239.69, stdev=56171.60, samples=16
   iops        : min=39863, max=87974, avg=60059.87, stdev=14042.95, samples=16
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=7.16%, 500=61.83%, 750=19.43%, 1000=7.39%
  lat (msec)   : 2=2.28%, 4=1.82%, 10=0.08%, 20=0.01%
  cpu          : usr=5.55%, sys=91.46%, ctx=1963, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2485MiB/s (2606MB/s), 2485MiB/s-2485MiB/s (2606MB/s-2606MB/s), io=10.0GiB (10.7GB), run=4121-4121msec

Run status group 1 (all jobs):
  WRITE: bw=2683MiB/s (2813MB/s), 2683MiB/s-2683MiB/s (2813MB/s-2813MB/s), io=10.0GiB (10.7GB), run=3817-3817msec

Run status group 2 (all jobs):
   READ: bw=171MiB/s (179MB/s), 171MiB/s-171MiB/s (179MB/s-179MB/s), io=2048MiB (2147MB), run=12004-12004msec

Run status group 3 (all jobs):
  WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=2048MiB (2147MB), run=9897-9897msec

Run status group 4 (all jobs):
  WRITE: bw=243MiB/s (254MB/s), 243MiB/s-243MiB/s (254MB/s-254MB/s), io=2048MiB (2147MB), run=8439-8439msec

生活相关

禁止休眠

How to Disable Suspend and Hibernation Modes In Linux (tecmint.com)

sudo systemctl status sleep.target suspend.target hibernate.target hybrid-sleep.target

禁止关闭笔记本盖子休眠

vim /etc/systemd/logind.conf

HandleLidSwitch=lock   # 从suspend改为lock

man logind.conf

Controls how logind shall handle the system power, reboot and sleep keys and the lid switch to trigger actions such as system power-off, reboot or suspend. Can be one of "ignore", "poweroff", "reboot", "halt", "kexec", "suspend", "hibernate", "hybrid-sleep", "suspend-then-hibernate", "lock", and "factory-reset".

If "ignore", systemd-logind will never handle these keys. If "lock", all running sessions will be screen-locked; otherwise, the specified action will be taken in the respective event. Only input devices with the "power-switch" udev tag will be watched for key/lid switch events.

HandlePowerKey= defaults to "poweroff", HandleRebootKey= defaults to "reboot", HandleSuspendKey= defaults to "suspend", HandleHibernateKey= defaults to "hibernate", HandlePowerKeyLongPress= defaults to "ignore", HandleRebootKeyLongPress= defaults to "poweroff", HandleSuspendKeyLongPress= defaults to "hibernate", HandleHibernateKeyLongPress= defaults to "ignore". HandleLidSwitch= defaults to "suspend". HandleLidSwitchExternalPower= is completely ignored by default (for backwards compatibility) — an explicit value must be set before it will be used to determine behaviour. HandleLidSwitchDocked= defaults to "ignore". If the system is inserted in a docking station, or if more than one display is connected, the action specified by HandleLidSwitchDocked= occurs; if the system is on external power the action (if any) specified by HandleLidSwitchExternalPower= occurs; otherwise the HandleLidSwitch= action occurs.

连接 wifi

iw,iwconfig 等底层工具

• iw
• iwlist
• wpa_supplicant: 用于连接 WPA 加密的 wifi

不使用 networkmanger 等工具，需要

• iw 连接 wifi
  • ifup interface
• 开启 dhcpclient 获得 ip

Network configuration/Wireless - ArchWiki (archlinux.org)

█▓▒░root@king3399█▓▒░ Mon Aug 07 03:07:46pm
~/ iw dev wlan0 station dump
Station 3c:cd:57:f5:da:85 (on wlan0)
        inactive time:  0 ms
        rx bytes:       26025
        rx packets:     153
        tx bytes:       22842
        tx packets:     121
        tx failed:      0
        signal:         -63 dBm
        tx bitrate:     234.0 MBit/s
        rx bitrate:     390.0 MBit/s
        authorized:     yes
        authenticated:  yes
        associated:     yes
        WMM/WME:        yes
        TDLS peer:      no
        DTIM period:    1
        beacon interval:100
        short slot time:yes
        connected time: 60 seconds
        current time:   1691392108815 ms
█▓▒░root@king3399█▓▒░ Mon Aug 07 03:08:28pm
~/ iw dev wlan0 link
Connected to 3c:cd:57:f5:da:85 (on wlan0)
        SSID: ACSA-706-5G
        freq: 5240
        RX: 29153 bytes (171 packets)
        TX: 24932 bytes (142 packets)
        signal: -60 dBm
        rx bitrate: 390.0 MBit/s
        tx bitrate: 234.0 MBit/s

        bss flags:      short-slot-time
        dtim period:    1
        beacon int:     100

扫描 wifi

iwlist wlan0 scan |grep ESSID    # 列出wifi

iwd

pve 设置 wifi 和普通 debian 差不多，但是应该避免使用 networkmanager 等高级的网络服务。因为这些是为桌面环境设计的，不适合 pve。WLAN - Proxmox VE

Setting up the Wi-Fi itself is not different in Proxmox VE than with a default Debian installation. But avoid installing advanced, network daemons like NetworkManager as those are normally suited for desktops only and may interfere with Proxmox VEs network requirements.

debian 使用 wifi 方法总结：WiFi/HowToUse - Debian Wiki

• iwd 作为 NetworkManager, and systemd-networkd 等的 backend
• 可以自己使用 iwd + systemd-resolved 打造一个最轻量的网络栈

iwd - ArchWiki (archlinux.org)

如果想要单独使用 iwd，需要 /etc/iwd/main.conf 开启网络配置（否则不会 dhcp 获得 v4 地址）

[General]
EnableNetworkConfiguration=true

[IPv4]
Address=192.168.36.216
Netmask=255.255.255.0
Gateway=192.168.36.1
Broadcast=192.168.36.255
DNS=192.168.36.1

NetworkManger

GUI：基于 GTK 的nm-connection-editor CLI：

• nmtui：使用起来很简单，可以编辑有线无线配置，激活连接等。
• nmcli：更加底层的命令行，功能还是很强大的

nmcli 基本使用

nmcli conn
nmcli conn edit xxx

WPA2 Enterprise 连接配置（eduroam）

WPA2 Enterprise 导致的 wifi 无法连接问题(7) Cannot Connect to College WIFI using NetworkManager : archlinux (reddit.com)

Failed to determine AP security information

12.10 - Connect to a WPA2-Enterprise Connection via CLI (no desktop) - Ask Ubuntu

nmcli con add type wifi ifname wlan0 con-name eduroam ssid eduroam 

You may edit the following settings: connection, 802-11-wireless (wifi), 802-11-wireless-security (wifi-sec), 802-1x, ethtool, match, ipv4, ipv6, hostname, tc, proxy
nmcli> set 802-1x.eap peap
nmcli> set 802-1x.phase2-auth mschapv2
nmcli> set 802-1x.identity xxxxx
nmcli> set 802-1x.password xxxxx
nmcli> set wifi-sec.key-mgmt wpa-eap
nmcli> save
Connection 'eduroam' (ae79dc6c-b660-469d-9ecb-3a29504de969) successfully updated.
nmcli> activate

netplan 连接 wifi 配置

How to configure wifi in Ubuntu (linuxhint.com)

# netplan yaml
network:
  version: 2
  renderer: networkd
  wifis:
      wlan0:
        optional: true
        dhcp4: true
        dhcp6: true
        access-points:
          "ustcnet": {}   # 没有密码
          "ssid2":
            password: "aaa"

通过 iwconfig 查看是否连接成功，出现 ESSID 则为成功

➜  ~ iwconfig wlan0
wlan0     IEEE 802.11  ESSID:"P2W_5G"
          Mode:Managed  Frequency:5.745 GHz  Access Point: 1C:40:E8:11:89:CF
          Bit Rate=234 Mb/s   Tx-Power=22 dBm
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on
          Link Quality=70/70  Signal level=-35 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:7   Missed beacon:0

音乐

USB Bluetooth passthrough KVM : VFIO (reddit.com) The 5 Best Command Line Music Players for Linux (tecmint.com)

sudo apt install cmus
cmus

5 # 浏览当前文件夹

使用 vlc

cvlc file.mp3

蓝牙

Bluetooth - ArchWiki (archlinux.org)

命令行工具bluetoothctl

1. (optional) Select a default controller with select _MAC_address_.
2. (optional) Enter power on to turn the power to the controller on if the device is set to off. It is on by default; see #Default adapter power state.
3. Enter devices to get the MAC address of the device with which to pair.
4. Enter device discovery mode with scan on command if device is not yet on the list.
5. Turn the agent on with agent on or choose a specific agent: if you press tab twice after agent you should see a list of available agents. A bluetooth agent is what manages the Bluetooth 'pairing code'. It can either respond to a 'pairing code' coming in, or can send one out. The default-agent should be appropriate in most cases.[1]
6. Enter pair _MAC_address_ to do the pairing (tab completion works).
7. If using a device without a PIN, one may need to manually trust the device before it can reconnect successfully. Enter trust _MAC_address_ to do so.
8. Enter connect _MAC_address_ to establish a connection.

网络

network 相关

常见网络配置方法

NetworkConfiguration - Debian Wiki

• ifupdown
  • debian 默认的方法
  • 配置文件为：/etc/network/interfaces
  • ifupdown2 是新版，支持更加灵活的配置
• NetworkManger
  • 目标是尽可能保证连接到网络。使用自动的规则配置网络，比如检测到有线设备就会连接
  • 常用于桌面发行版
• systemd
  • 相关的服务：systemd-networkd, systemd-resolved
  • 配置文件位于/etc/system/network，使用类似于 ini 格式
• netplan
  • 常见于 ubuntu，底层可以基于 systemd-networkd 或 NetworkManager（服务器版和桌面版）
  • 是另一层抽象，使用自己的配置文件/etc/netplan/，使用 yaml 格式。

networking - How exactly are NetworkManager, networkd, netplan, ifupdown2, and iproute2 interacting? - Unix & Linux Stack Exchange

网络工具

Chapter 5. Network setup (debian.org)

• iproute2
• iptables
• ethtool
• mtr
• nmap
• tcpdump
• dnsutils: nslookup, dig

dhcp

测试 dhcp 是否工作

• netplan（或者重启容器） + tcpdump + wireshark

• -d: debug 模式，前台运行
• -n: 不设置接口
• -s: 指定 dhcp server（否则是广播）

 dhclient -d -n -w -s 192.168.35.1

dns

dig

dig [@server] [-p port] [-t type] [-4] [-6] [+trace] name

• @ 指定 DNS 查询使用的服务器名称或 IP，IP 地址可以是用点分隔的 IPv4 地址也可以是冒号分隔的 IPv6 地址。当参数指定的值是服务器的主机名时，dig 命令会在查询该域名服务器前先解析该主机名；
• -p 指定 DNS 查询使用的端口号，默认情况下 DNS 查询使用标准的 53 端口，若使用非端口则需要通过 -p 参数指定，可使用此选项来测试已配置为侦听非标准端口号上的 DNS 服务器；
• -t 指定 DNS 查询的记录类型，常用的类型包括：A/AAAA/NS/MX/CNAME 等，缺省查询类型是 A；
• -4 指定 dig 命令仅使用 IPv4 查询传输；
• -6 指定 dig 命令仅使用 IPv6 查询传输；
• +trace 跟踪从根名称服务器开始的迭代查询过程，缺省情况不使用跟踪。启用跟踪时，dig 命令会执行迭代查询以解析要查询的名称，显示来自用于解析查询的每个服务器的应答。

指定解析类型

• NS 记录：用来指定域名由哪个 DNS 服务器进行解析；
• CNAME 记录：用来定义域名的别名，方便实现将多个域名解析到同一个 IP 地址；
• A 记录：用来指定主机名对应的 IPv4 地址；
• AAAA 记录：用来指定主机名对应的 IPv6 地址；
• MX 记录：用来指定收件人域名的邮件服务器，SMTP 协议会根据 MX 记录的值来决定邮件的路由过程；
• PTR 记录：常用于反向地址解析，将 IP 地址解析到对应的名称；
• SOA 记录：称为起始授权机构记录，不同于 NS 记录用于标识多台域名解析服务器，SOA 记录用于在多台 NS 记录中哪一台是主 DNS 服务器。

USTC lug 加群链接

dig jointele.ustclug.org TXT

;; ANSWER SECTION:
jointele.ustclug.org.   600     IN      TXT     "(base64) aHR0cHM6Ly90Lm1lLytYOXg1V09qTVpVbGlZVGRs"

查看上游 dns

有多种 dns 设置方式：

• 直接编辑/etc/resolve 设置上游 dns
• systemd-resolve 服务（自动设置 dns）

systemd-resolve --status

从 ubuntu22.04 开始

resolvectl status

systemd-resolve command not found in Ubuntu 22.04 desktop - Ask Ubuntu

iptables

持久化

iptables-save > /etc/iptables/rules.v4  # 
ip6tables-save > /etc/iptables/rules.v6 
iptables-restore < /etc/iptables/rules.v4

安装 iptabels-persistent

创建 socks5 代理

有时候我们需要临时通过一台服务器上网。而代理协议比较方便的有 HTTP 和 Socks5。socks5 支持 TCP 和 UDP，使用用途更广一些。

• dante-server
  • 网站搜索推荐的。但是
    • 不支持 ipv6
    • 配置增加了很多东西，所以配置相对复杂
      • rule，控制哪些客户端可以连接 server，以及客户端可以 request 哪些地址

看来还是直接使用 ss-server 算了。

遇到的问题

bridge iptable 不生效

bridge-nf-call-iptables - BOOLEAN
 1 : pass bridged IPv4 traffic to iptables' chains.
 0 : disable this.
 Default: 1

系统维护

setcap

授予特定二进制文件在不提升用户权限的情况下绑定低端口

sudo setcap 'cap_net_bind_service=+ep' /usr/bin/python3.10 # 不能是软链接

ldap

LDAP - ACSA Docs (acsalab.com)

systemd

sudo systemctl list-units --type=service --state=running

ssh 证书

SSH Certificate Authority - LUG @ USTC (ustclug.org) Managing servers with OpenSSH Certificate Authority - iBug

• client 验证 server 身份，需要将 server 公钥添加到 known_hosts 中。
  • 当有多个服务器时，client 需要同意多次（ssh 时的 yes）
  • 当 server 公钥变更时，还得修改 known_hosts
• server 验证 client 身份，同样需要将 client public key 添加到 authorized_keys 中。
  • 多个服务端和客户端，需要将每个 client 公钥复制到每个 server。
  • 修改一个 client 需要修改所有 server

SSH CA(Certificate Authority) 是一对受信任的密钥，用于签发证书。

An SSH Certificate Authority (CA) is a trusted key pair that issues certificates.

签发的证书可以用来授权 client 或者 server。（通过信任 host CA，client 可以信任所有 server，只要它们有 host CA 签名的证书。同样，通过信任 user CA，server 可以信任所有的 client，只要它们有 user CA 签名的证书。）

Certificates can be used for authentication on both the server side and the client side

SSH CA 和 X.509 证书系统不同的是，证书无法签发新的证书（即无法嵌套）

host CA 和 user CA 可以是不同的，也可以是相同的。

SSH CA

创建 CA

和创建一个通常的密钥对一样。

 sudo ssh-keygen -f yfy_ca -t ed25519 -C "yfy CA"

• -t dsa | ecdsa | ecdsa-sk | ed25519 | ed25519-sk | rsa
• -C comment：添加到公钥结尾

签发证书

ssh-keygen -s /path/to/ssh_ca \
           -I certificate_identity \
           -n principals \
          [-O options] \
          [-V validity_interval] \
           public_key_file

使用 host CA 授权 server 端

给 server 的公钥 ssh_host_rsa_key.pub 签名

ssh-keygen -s /path/to/ssh_ca \
           -I blog \
           -h \
           -n blog.s.ustclug.org,blog.p.ustclug.org,10.254.0.15,202.141.176.98,202.141.160.98 \
           ssh_host_rsa_key.pub

• -h 表示签发的是 host 证书（server 端）
• -n 用于限制该证书只在提供的域名或者 ip 上有效。这样即使该证书被偷了，偷窃者也无法伪造我们的服务器。

添加到 sshd 配置中，这样 client 连接时会尝试提供该证书

# /etc/ssh/sshd_config
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

客户端信任 CA 机构

• 将 CA 公钥保存在 my_ca.pub 中

printf "@cert-authority * " | cat - my_ca.pub >> ~/.ssh/known_hosts

• 保存在 dns 中
  • 利用{command1 && command2 }将两条命令合并，将其输出重定向

{printf "@cert-authority * " && dig ca.yfycloud.top TXT +noall +answer | awk -F'TXT' '{print $NF}' |xargs} >> ~/.ssh/known_hosts

使用 nslookup(windows bash 下测试)

(printf "@cert-authority * " && nslookup -q=txt ca.yfycloud.top |grep CA | xargs) >> ~/.ssh/known_hosts

使用 user CA 授权 client 端

服务端信任 user CA

TrustedUserCAKeys /etc/ssh/ssh_user_ca

给指定用户签发证书

ssh-keygen -s my_ca -I <user name> -n root,ubuntu id_rsa.pub

• 将生成的 id_rsa-cert.pub 放至 id_rsa.pub 同一目录，ssh 连接服务端时会自动读取。
• -n 用于限制允许登录的用户

遇到问题

mobaxterm连接服务器时报错：No supported authentication methods available (server sent: publickey) 但是我明明是指定的使用ssh key登录，在命令行中也确实是能够用指定的私钥登录的。

后面发现问题是mobaxterm发现了我有一个证书，使用该证书去登录了。但是该证书签名时只principal只指定了root，从而导致登不上。 服务端日志如下，刚开始指定登录root成功，第二次指定fyyuan登录失败

Mar 29 11:39:26 snode1 sshd[1143073]: Accepted publickey for root from 2001:da8:d800:c017:ed94:1e42:a843:27f9 port 23902 ssh2: ED25519-CERT SHA256:oL5HVEHu4AQj4e0d03kjT39A6p5t+3j5SKUKOqLKjwI ID Yuan Fuyan (serial 0) CA ECDSA SHA256:e/hc4wrIaub3Q8UGmQYQI3A2fiiHF/RevFKo0UCG4Tc
Mar 29 11:39:26 snode1 sshd[1143073]: pam_unix(sshd:session): session opened for user root by (uid=0)
Mar 29 11:39:45 snode1 sshd[1144083]: error: Certificate invalid: name is not a listed principal
Mar 29 11:39:45 snode1 sshd[1144083]: error: Received disconnect from 2001:da8:d800:c017:ed94:1e42:a843:27f9 port 23929:14: No supported authentication methods available [preauth]
Mar 29 11:39:45 snode1 sshd[1144083]: Disconnected from authenticating user fyyuan 2001:da8:d800:c017:ed94:1e42:a843:27f9 port 23929 [preauth]

包管理器 apt

apt show pkg -a #查看包的所有仓库  # APT-Sources:

--no-install-recommends

查询文件

linux - List of files installed from apt package - Server Fault

apt install apt-file
apt-file update

列出包的内容

• 已安裝包：dpkg -L deb
• 未安裝包：apt-file list $package
• 有 deb 文件：dpkg --contents $package.deb

查询文件位于哪个包中

dpkg -S /usr/bin/nvidia-xconfig
apt-file search /usr/bin/nvidia-xconfig

源优先级

按照 nvidia 官网下载了本地版本的 cuda deb，安装后在/var/cuda-repo-ubuntu2004-12-1-local可以看到 deb 包。但是 apt install 时仍然采用了 nvidia 仓库的。 解决方法：

• 直接注释掉/etc/apt/source.list中的 nvidia 仓库，然后 apt update
  • 也有可能 source.list 中没有 developer.download.nvidia.com，那么禁用/etc/apt/source.list.d/下的在线 cuda 仓库。如 mv cuda-ubuntu2004-x86_64.list cuda-ubuntu2004-x86_64.list.bak。

vim /etc/apt/preference.d/cuda-repository-pin-600

# Priority 越小优先级越高
Package: nsight-compute
Pin: origin *ubuntu.com*
Pin-Priority: -1

Package: nsight-systems
Pin: origin *ubuntu.com*
Pin-Priority: -1

Package: *
Pin: release l=NVIDIA CUDA
Pin-Priority: 600

➜  ~ apt-cache policy gnome-shell-extension-ubuntu-dock
gnome-shell-extension-ubuntu-dock:
  Installed: (none)
  Candidate: 72~ubuntu5.22.04.2.1
  Version table:
     72~ubuntu5.22.04.2.1 500
        500 http://mirrors.ustc.edu.cn/ubuntu jammy-updates/main amd64 Packages
     72~ubuntu5 500
        500 http://mirrors.ustc.edu.cn/ubuntu jammy/main amd64 Packages

用户管理

• adduser：交互式添加用户
  • useradd 为更底层的命令
• usermod -aG sudo user：将用户添加到 sudo 组
• gpasswd -d user group: 将用户从组中删除

日志/journalctl

How To Use Journalctl to View and Manipulate Systemd Logs | DigitalOcean

journalctl --list-boots
journalctl -b # 自上次reboot开始log
journalctl -b -1 # 上一次reboot

journalctl --since "2015-01-10" --until "2015-01-11 03:00"  # YYYY-MM-DD HH:MM:SS
journalctl --since yesterday
journalctl -S yesterday

journalctl -k # 内核

sytemd.time(7)

time span

Time spans refer to time durations

•   usec, us, µs
•   msec, ms
•   seconds, second, sec, s
•   minutes, minute, min, m
•   hours, hour, hr, h
•   days, day, d
•   weeks, week, w
•   months, month, M (defined as 30.44 days)
•   years, year, y (defined as 365.25 days)

example

2 h
2hours
48hr
1y 12month
55s500ms
300ms20s 5day

timestamp

Timestamps refer to specific, unique points in time

一般显示格式：Fri 2012-11-23 23:02:15 CET

包含毫秒格式（sub-second remainder is expected separated by a full stop from the seconds component.）

2014-03-25 03:59:56.654563

特殊关键字："now", "today", "yesterday", and "tomorrow"

相对时间，- 表示当前时间减去 timespan，+ 则是加上。也可以在结尾使用 ago 和 left（注意需要一个空格）

+3h30min → Fri 2012-11-23 21:45:22

@ 表示相对于 UNIX epoch

Finally, a timespan prefixed with "@" is evaluated relative to the UNIX time epoch 1st Jan, 1970, 00:00.

@1395716396 → Tue 2014-03-25 03:59:56

             Fri 2012-11-23 11:12:13 → Fri 2012-11-23 11:12:13
                 2012-11-23 11:12:13 → Fri 2012-11-23 11:12:13
             2012-11-23 11:12:13 UTC → Fri 2012-11-23 19:12:13
                          2012-11-23 → Fri 2012-11-23 00:00:00
                            12-11-23 → Fri 2012-11-23 00:00:00
                            11:12:13 → Fri 2012-11-23 11:12:13
                               11:12 → Fri 2012-11-23 11:12:00
                                 now → Fri 2012-11-23 18:15:22
                               today → Fri 2012-11-23 00:00:00
                           today UTC → Fri 2012-11-23 16:00:00
                           yesterday → Fri 2012-11-22 00:00:00
                            tomorrow → Fri 2012-11-24 00:00:00
           tomorrow Pacific/Auckland → Thu 2012-11-23 19:00:00
                            +3h30min → Fri 2012-11-23 21:45:22
                                 -5s → Fri 2012-11-23 18:15:17
                           11min ago → Fri 2012-11-23 18:04:22
                         @1395716396 → Tue 2014-03-25 03:59:56

➜  ~ systemd-analyze timestamp now
  Original form: now
Normalized form: Mon 2024-06-03 18:09:45 CST
       (in UTC): Mon 2024-06-03 10:09:45 UTC
   UNIX seconds: @1717409385.708355
       From now: 165us ago

calendar event

Calendar events may be used to refer to one or more points in time in a single expression. They form a superset of the absolute timestamps explained above

格式和 timestamps 类似

Thu,Fri 2012-*-1,5 11:12:13

规则

• * 表示通配符
• , 逗号分隔表示多个值
• .. 表示连续的一段值
• 省略 date 表示今天，省略 time 表示 00:00:00
• start/repetition：从 start 开始，增加 repetition 的倍数的所有值。
• date 中可以使用 ~ 表示倒数的第几天。如 *-02~01 表示 2 月最后一天。
  • "Mon *-05~07/1" means "the last Monday in May."。~07/1 表示从最后一周开始的每一天，mon 则限制了周一。

一些规则的简写

               minutely → *-*-* *:*:00
                 hourly → *-*-* *:00:00
                  daily → *-*-* 00:00:00
                monthly → *-*-01 00:00:00
                 weekly → Mon *-*-* 00:00:00
                 yearly → *-01-01 00:00:00
              quarterly → *-01,04,07,10-01 00:00:00
           semiannually → *-01,07-01 00:00:00

示例

             Sat,Thu,Mon..Wed,Sat..Sun → Mon..Thu,Sat,Sun *-*-* 00:00:00
                 Mon,Sun 12-*-* 2,1:23 → Mon,Sun 2012-*-* 01,02:23:00
                               Wed *-1 → Wed *-*-01 00:00:00
                      Wed..Wed,Wed *-1 → Wed *-*-01 00:00:00
                            Wed, 17:48 → Wed *-*-* 17:48:00
           Wed..Sat,Tue 12-10-15 1:2:3 → Tue..Sat 2012-10-15 01:02:03
                           *-*-7 0:0:0 → *-*-07 00:00:00
                                 10-15 → *-10-15 00:00:00
                   monday *-12-* 17:00 → Mon *-12-* 17:00:00
             Mon,Fri *-*-3,1,2 *:30:45 → Mon,Fri *-*-01,02,03 *:30:45
                  12,14,13,12:20,10,30 → *-*-* 12,13,14:10,20,30:00
                       12..14:10,20,30 → *-*-* 12..14:10,20,30:00
             mon,fri *-1/2-1,3 *:30:45 → Mon,Fri *-01/2-01,03 *:30:45
                        03-05 08:05:40 → *-03-05 08:05:40
                              08:05:40 → *-*-* 08:05:40
                                 05:40 → *-*-* 05:40:00
                Sat,Sun 12-05 08:05:40 → Sat,Sun *-12-05 08:05:40
                      Sat,Sun 08:05:40 → Sat,Sun *-*-* 08:05:40
                      2003-03-05 05:40 → 2003-03-05 05:40:00
            05:40:23.4200004/3.1700005 → *-*-* 05:40:23.420000/3.170001
                        2003-02..04-05 → 2003-02..04-05 00:00:00
                  2003-03-05 05:40 UTC → 2003-03-05 05:40:00 UTC
                            2003-03-05 → 2003-03-05 00:00:00
                                 03-05 → *-03-05 00:00:00
                                hourly → *-*-* *:00:00
                                 daily → *-*-* 00:00:00
                             daily UTC → *-*-* 00:00:00 UTC
                               monthly → *-*-01 00:00:00
                                weekly → Mon *-*-* 00:00:00
               weekly Pacific/Auckland → Mon *-*-* 00:00:00 Pacific/Auckland
                                yearly → *-01-01 00:00:00
                              annually → *-01-01 00:00:00
                                 *:2/3 → *-*-* *:02/3:00

日志文件

• /run/log/journal/MACHINE-ID/：存储在内存中，重启消失
• /var/log/journal/MACHINE-ID：存储在磁盘中

查看大小：

journalctl --disk-usage  # 日志大小，包含归档日志

清除：

• vacuum：删除归档的日志（不会删除 active 的）
• rotate：将 active 归档。可以合并到上条命令

journalctl --rotate

journalctl --vacuum-time=1months  # 一个月以前的旧日志。格式：1m, 2h, 2weeks, 4months
journalctl --vacuum-size=100M     # 只保留新的100M大小的日志

设置最大日志大小：

/etc/systemd/journald.conf
SystemMaxUse=100M

SystemMaxUse=, RuntimeMaxUse=

Enforce size limits on the journal files stored. The options prefixed with "System" apply to the journal files when stored on a persistent file system, more specifically /var/log/journal. The options prefixed with "Runtime" apply to the journal files when stored on a volatile in-memory file system, more specifically /run/log/journal.

MaxFileSec=

The maximum time to store entries in a single journal file before rotating to the next one.

查看重启原因

https://unix.stackexchange.com/a/278166

last -x |tac |less  # last显示最后logged用户，tac将文本逆序

• 正常顺序：
  • reboot 没有对应的 shutdown 表示非正常关机

shutdown system down  ... <-- first the system shuts down   
reboot   system boot  ... <-- afterwards the system boots
runlevel (to lvl 3)  # runlevel3表示无图形界面，开机桌面环境可能是runlevel 5

查看 crontab 日志

grep CRON /var/log/syslog

jou -u cron

更换内核

编译

8.10. Compiling a Kernel (debian-handbook.info)

disabling the debug information

$ make deb-pkg LOCALVERSION=-falcot KDEB_PKGVERSION=$(make kernelversion)-1
[...]
$ ls ../*.deb
../linux-headers-5.10.46-falcot_5.10.46-1_amd64.deb
../linux-image-5.10.46-falcot_5.10.46-1_amd64.deb
../linux-image-5.10.46-falcot-dbg_5.10.46-1_amd64.deb
../linux-libc-dev_5.10.46-1_amd64.deb

The whole process requires around 20 GB of free space, at least 8 GB of RAM, and several hours of compilation (utilizing one core) for a standard amd64 Debian kernel. These requirements can be drastically reduced by disabling the debug information using CONFIG_DEBUG_INFO=n, but this will make it impossible to trace kernel errors (“oops”) using gdb and also stop the creation of the linux-image-version-dbg package.

# 获得当前官方kernel编译配置，如/boot/config-5.19.0-35-generic
kernel_config=$(ls /boot/config-* | grep generic | sort -Vr | head -n 1) 
cp "${kernel_config}" .config

# 基于当前.confg，只询问新内核中新功能的配置
make oldconfig 

# 编译
make -j "$(nproc)" bindeb-pkg LOCALVERSION="${kernel_localversion}" || exit

# 安装
sudo dpkg -i ../linux-image-${kernel_version}${kernel_localversion}*.deb ../linux-headers-${kernel_version}${kernel_localversion}*.deb

# 更新grub，通过GRUB_DEFAULT选择编译的内核
vim /etc/default/grub
grub_line="GRUB_DEFAULT=\"Advanced options for Ubuntu>Ubuntu, with Linux ${kernel_version}${kernel_localversion}\""
sudo sed -i -e "s/^#GRUB_DEFAULT=.*/\0\n${grub_line}/" "${grub_config}"

sudo update-grub

missing certificate

Building a custom kernel based on these settings will fail if debian/certs/debian-uefi-certs.pem does not exist. This file can either be obtained from the kernel team's Git repository and placed into the source tree, or it can be replaced by your own certificate(s), or you'll have to disable the setting via CONFIG_SYSTEM_TRUSTED_KEYS="".

sed -i -e '/^CONFIG_SYSTEM_TRUSTED_KEYS=/ c\CONFIG_SYSTEM_TRUSTED_KEYS=\"\"' .config
sed -i -e '/^CONFIG_SYSTEM_REVOCATION_KEYS=/ c\CONFIG_SYSTEM_REVOCATION_KEY=\"\"' .config

dkms

使用 patched kernel 后，没有了 zfs，无法启动 lxc。 Ubuntu 20.04, building patched kernel results in no ZFS support - Ask Ubuntu

https://debian-handbook.info/browse/stable/sect.kernel-compilation.html#sect.modules-build

• 一些模块不能打包到 linux 内核中
• 使用 dkms 自动化编译这些模块。需要安装了对应的linux-headers-*

Compiling External Modules Some modules are maintained outside of the official Linux kernel. To use them, they must be compiled alongside the matching kernel. While we could manually extract the tarball and build the module, in practice we prefer to automate all this using the DKMS framework (Dynamic Kernel Module Support). Most modules offer the required DKMS integration in a package ending with a -dkms suffix.  In our case, installing dahdi-dkms is all that is needed to compile the kernel module for the current kernel provided that we have the linux-headers-* package matching the installed kernel. For instance, if you use linux-image-amd64, you would also install linux-headers-amd64.

sudo apt install zfs-dkms

dkms status
# zfs/2.1.5, 5.19.17, x86_64: installed

dkms build

dkms install
dkms uninstall  zfs/2.1.5

dkms add
dkms remove zenpower/$(VERSION)   --all

例子：Ta180m/zenpower3: Migrated to Gitea (github.com)

• 安装 header
  • pve 使用自己的内核，因此也是安装自己的 header

apt install linux-headers-$(uname -r)

#pve
apt install proxmox-headers-$(uname -r)

dkms-install:
        dkms --version >> /dev/null
        mkdir -p $(DKMS_ROOT_PATH)
        cp $(CURDIR)/dkms.conf $(DKMS_ROOT_PATH)
        cp $(CURDIR)/Makefile $(DKMS_ROOT_PATH)
        cp $(CURDIR)/zenpower.c $(DKMS_ROOT_PATH)

        sed -e "s/@CFLGS@/${MCFLAGS}/" \
            -e "s/@VERSION@/$(VERSION)/" \
            -i $(DKMS_ROOT_PATH)/dkms.conf

        dkms add zenpower/$(VERSION)
        dkms build zenpower/$(VERSION)
        dkms install zenpower/$(VERSION)

dkms-uninstall:
        dkms remove zenpower/$(VERSION) --all
        rm -rf $(DKMS_ROOT_PATH)

删除内核

dpkg --get-selections | grep 5.19.17

apt remove xxx

或者手动删除

cd /boot/
rm -rf *3.19.8*

update-grub

内核模块

find /lib/modules/$(uname -r) -type f -name '*.ko'

modprobe xxx
modprobe -r xxx # 删除


lsmod |grep xxx
rmmod xxx

1. depmod: 用于更新内核模块依赖信息。depmod 命令会扫描系统中的内核模块，并为它们生成一个依赖关系文件，通常在 /lib/modules/ 目录下。这个文件通常被命名为 modules.dep，它列出了每个模块及其依赖的其他模块，以确保内核模块正确加载和卸载。

2. lsmod: 用于列出当前加载到内核中的模块。lsmod 命令会显示已加载模块的列表，以及它们的大小、使用次数和依赖关系。

3. modprobe: 用于动态加载和卸载内核模块。modprobe 命令可以根据模块名称自动加载模块，并处理其依赖关系。它还可以用于卸载模块，或者通过参数指定模块选项。

4. insmod: 用于手动加载一个内核模块。insmod 命令可以直接加载指定的模块文件，但不会处理模块的依赖关系。因此，通常建议使用 modprobe 来加载模块，以确保正确处理依赖关系。

5. rmmod: 用于从内核中卸载一个已加载的模块。rmmod 命令接受模块名称作为参数，并会尝试卸载该模块，前提是它没有被其他模块所使用。

6. modinfo: 用于获取有关内核模块的信息。modinfo 命令可以列出有关模块的信息，如作者、描述、参数等。

7. ls /lib/modules/: 这个命令用于查看系统中可用的内核模块。在 /lib/modules/ 目录下，你可以找到与已安装内核版本相关的模块文件和依赖信息。

硬件相关

关闭 beep/bell

Turn off beep / bell on linux terminal - Linux Tutorials - Learn Linux Configuration

查看 CPU,cache

查看 cache 大小

• vim /proc/cpuinfo
  • cache_size 不知道具体含义，L2/per core？
  • cache_alignment 为 L1 cache line 大小？

cache size      : 512 KB
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual

• lscpu
  • 很详细，可以看到 cpu flag, numa, cache

Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    32 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15

• getconf -a | grep CACHE
  • 可以看到 cache line 大小

LEVEL1_ICACHE_SIZE                 32768
LEVEL1_ICACHE_ASSOC
LEVEL1_ICACHE_LINESIZE             64
LEVEL1_DCACHE_SIZE                 32768
LEVEL1_DCACHE_ASSOC                8
LEVEL1_DCACHE_LINESIZE             64
LEVEL2_CACHE_SIZE                  524288
LEVEL2_CACHE_ASSOC                 8
LEVEL2_CACHE_LINESIZE              64
LEVEL3_CACHE_SIZE                  33554432
LEVEL3_CACHE_ASSOC                 0
LEVEL3_CACHE_LINESIZE              64
LEVEL4_CACHE_SIZE
LEVEL4_CACHE_ASSOC
LEVEL4_CACHE_LINESIZE

lspci

apt install pciutils

常用选项

lspci -k  #查看内核驱动

lspci -nn #查看设备id

lspci -vv # 查看pcie链路速度：LnkCap, LnkSta
- 2.5 GT/s = PCI-e gen 1
- 5 GT/s = PCI-e gen 2
- 8 GT/s = PCI-e gen 3

指定设备

-s [[[[<domain>]:]<bus>]:][<device>][.[<func>]]

• domain: 0 to ffff
• bus: 0-ff
• device: 0-ff
• function: 0-7
• 可以省去或者设置为*表示任意值

-d [vendor]:[device][:class[:prog-if]]

• vendor: e.g. 10de
• device: : e.g. 1c02

查看树状结构

lspci -tv

• 03.1：是 slot and function number of the PCIe root hub
• [08]表示在其后面的设备的 bus 为 08
• 00.0 为 device 和 function 编号

+-03.1-[08]--+-00.0  NVIDIA Corporation GP106 [GeForce GTX 1060 3GB]
           |            \-00.1  NVIDIA Corporation GP106 High Definition Audio Controller

监控 (温度、功耗)

SSD 温度

Linux find NVMe SSD temperature using command line - nixCraft (cyberciti.biz)

apt install nvme-cli

sudo nvme smart-log /dev/nvme0

CPU 频率、功耗

lscpu -e
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ      MHZ
  0    0      0    0 0:0:0:0           是 4850.1948 2200.0000 3023.077
  1    0      0    1 1:1:1:0           是 4850.1948 2200.0000 3167.647
  2    0      0    2 2:2:2:0           是 4850.1948 2200.0000 2529.517
  3    0      0    3 3:3:3:0           是 4850.1948 2200.0000 2895.147
  4    0      0    4 4:4:4:0           是 4850.1948 2200.0000 3068.640
  5    0      0    5 5:5:5:0           是 4850.1948 2200.0000 2814.150
  6    0      0    6 6:6:6:0           是 4850.1948 2200.0000 3097.454
  7    0      0    7 7:7:7:0           是 4850.1948 2200.0000 3148.684
  8    0      0    0 0:0:0:0           是 4850.1948 2200.0000 2938.599
  9    0      0    1 1:1:1:0           是 4850.1948 2200.0000 3376.774
 10    0      0    2 2:2:2:0           是 4850.1948 2200.0000 4420.665
 11    0      0    3 3:3:3:0           是 4850.1948 2200.0000 2996.655
 12    0      0    4 4:4:4:0           是 4850.1948 2200.0000 3100.166
 13    0      0    5 5:5:5:0           是 4850.1948 2200.0000 2894.990
 14    0      0    6 6:6:6:0           是 4850.1948 2200.0000 3244.797
 15    0      0    7 7:7:7:0           是 4850.1948 2200.0000 2897.287

CPU frequency scaling - ArchWiki (archlinux.org)

查看功耗

sudo turbostat

音频

ALSA

yfy@TianYi310 ➜  /dev aplay -l
**** PLAYBACK 硬體裝置清單 ****
card 0: PCH [HDA Intel PCH], device 0: CX20751/2 Analog [CX20751/2 Analog]
  子设备: 1/1
  子设备 #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 3: HDMI 0 [HDMI 0]
  子设备: 1/1
  子设备 #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 7: HDMI 1 [HDMI 1]
  子设备: 1/1
  子设备 #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 8: HDMI 2 [HDMI 2]
  子设备: 1/1
  子设备 #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 9: HDMI 3 [HDMI 3]
  子设备: 1/1
  子设备 #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 10: HDMI 4 [HDMI 4]
  子设备: 1/1
  子设备 #0: subdevice #0

root@p2w \u@\h:\w\$ cat /proc/asound/modules
 0 snd_usb_audio

root@p2w \u@\h:\w\$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: Lanseyaoji [Lanseyaoji], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

arecord, aplay

arecord, aplay - command-line sound recorder and player for ALSA soundcard driver

arecord -d 10 -f cd -t wav -D copy foobar.wav will record foobar.wav as a 10-second, CD-quality wave file

aplay 播放貌似只支持 WAVE 格式，mp3 播放是噪音

yfy@TianYi310 ➜  Downloads aplay 'Laco - NEXUS.mp3'
正在播放 原始資料 'Laco - NEXUS.mp3' : Signed 16 bit Little Endian, 频率8000Hz， Mono
^C被信号 中断...退出
yfy@TianYi310 ➜  Downloads aplay filename.wav
正在播放 WAVE 'filename.wav' : Signed 16 bit Little Endian, 频率44100Hz， Stereo

aplay -l 没有设备

但是 root 用户有。这也导致 pulseaudio list sinks 时只有伪设备。

alsa - How do I get sound with an HDA card? - Ask Ubuntu audio - ALSA does not find sound devices as user but can as root after kernel upgrade - Super User

没有尝试参考 1，而是直接修改了/dev/snd 的权限 777（添加用户到 audio 组没有作用），问题解决。

PulseAudio

system-wide 模式和 per-user 模式

推荐使用 per-user 模式，system-wide 模式不安全。Running PulseAudio as System-Wide Daemon – PulseAudio (www.freedesktop.org)

per-user 模式，每个用户需要使用时启动 pulseaudio。system 模式适合没有普通用户的嵌入式设备

运行 server

Running (www.freedesktop.org)

有桌面环境情况下一般情况下会自动启动。也可以手动启动

pulseaudio --kill # 关闭pulseaudio server

pulseaudio # 运行在前台，ctrl-C结束。便于实时查看日志
pulseaudio --daemonize # daemon模式，运行在后台
pulseaudio -vv --log-time # 显示更多log

~/.config/pulse/优先级高于/etc/pulse/

• default.pa : The default PulseAudio Startup Script Commands in this file run when the daemon is started or restarted. Run man default.pa for documentation.
• daemon.conf: The PulseAudio daemon configuration file Run man pulse-daemon.conf for documentation.
• system.pa：仅用于 system-wide 模式

控制命令 pactl

CLI (www.freedesktop.org)

通过pactl命令和 pulseaudio daemon 进行交互

pactl list sources # 查看输入设备
pactl list sinks # 查看输出设备

# 调节音量
pactl set-sink-mute <device> toggle/true/false  # device tab自动补全
pactl set-sink-volume <device> 30%

让麦克风输出到扬声器

sound - How to hear my voice in speakers with a mic? - Ask Ubuntu

pactl load-module module-loopback latency_msec=1

# stop
pactl unload-module module-loopback

另一个办法直接使用 alsa 命令。

arecord -f cd - | aplay - # 延迟很高
arecord -f cd - | tee output.wav | aplay - #  play while saving

arecord --buffer-time=1 - | aplay --buffer-time=1 - # 改善延迟

将指定 source 输出到指定 sink

Modules – PulseAudio (www.freedesktop.org)

pactl load-module module-loopback latency_msec=1 source=2 sink=0  # 数字为index

录指定应用声音

p.s 关键点

• 直接对输出到扬声器的声音录制，自己就听不到声音了。使用 combined，对输出到 headphone 的声音复制一份，然后对其进行录制

参考：Linux audio recording guide (ro-che.info)

1. Create a “null” sink that you will be recording. Let’s call it recording.
2. Create a combined sink that will send its input to both headphones and the recording sink. Otherwise, you will be able to record a stream but not hear it yourself. So, let’s call this sink combined.
3. Direct the sound from the specific applications you want to record into the combined sink.
4. Record the monitor of the recording sink to a file.

添加 combined 设备

pacmd load-module module-null-sink sink_name=recording sink_properties=device.description=recording
pacmd load-module module-combine-sink sink_name=combined sink_properties=device.description=combined \
  slaves=recording,alsa_output.pci-0000_00_1f.3.analog-stereo

将 combined 设置为默认输出，然后开始录制

parecord --channels=1 -d recording.monitor filename.wav

• parecord 录制没有压缩，只能使用 parecord 录制吗？

音频网络串流

参考

• linux 到安卓设备：How to stream my GNU/Linux audio output to Android devices over WI-FI? - Super User
• windows 到 linux：Streaming audio from Windows to Linux using PulseAudio - Super User
  • windows 上安装音频驱动，linux 上安装 scream receiver：duncanthrax/scream: Virtual network sound card for Microsoft Windows (github.com)
• windows 到 android

• 手机到 laptop, 通过 pulseaudio：How to stream audio from your phone to your laptop with PulseAudio | Hacker News (ycombinator.com)

pulseaudio

Network Setup – PulseAudio (www.freedesktop.org)

direct 模式

最简单的方式

• 在需要实际播放声音的设备上（server 端），开启module-native-protocol-tcp允许网络连接。

• 然后在客户端上，设置 PULSE_SERVER。这样客户端的声音就会交给服务端输出

  • 可以通过环境变量PULSE_SERVER
  • Alternatively you can modify ~/.pulse/client.conf or /etc/pulse/client.conf and set default-server.

  export PULSE_SERVER=tcp:<server_ip>:4713
  

android 上有 pulse server 软件：XServer XSDL - Google Play 上的应用

• 延迟挺低的

tunnel 模式

通过 tunnel，可以创建一个新的 sink，将所有音频通过网络发送到另一个 server 上（source 也一样）。好处是可以无缝切换使用本地还是远程设备。

Modules – PulseAudio (www.freedesktop.org)

假设 A，B 两个机器，A 有麦克风，B 有扬声器。可以通过以下配置，让 A 获得 B 的扬声器，B 获得 A 的麦克风。

# A
load-module module-native-protocol-tcp auth-anonymous=1
load-module module-tunnel-sink server=192.168.36.215

# B
load-module module-native-protocol-tcp auth-anonymous=1
load-module module-tunnel-source server=192.168.36.230

RTP

RTP is the Realtime Transfer Protocol. It is a well-known protocol for transferring audio and video data over IP. Two related protocols are SDP and SAP. SDP is the Session Description Protocol and can be used to describe RTP sessions. SAP is the Session Announcement Protocol and can be used to announce RTP sessions that are described with SDP. (Modern SIP based VoIP phones use RTP/SDP for their sessions, too) All three protocols are defined in IETF RFCs (RFC3550, RFC3551, RFC2327, RFC2327). They can be used in both multicast and unicast fashions. PulseAudio exclusively uses multicast RTP/SDP/SAP containing audio data.

• 可以用于 1 对多，多对多等场景。
• RTP 基于 IP 层的协议延迟可能更低？

• pulseaudio 目前只支持 LAN 内。没法在跨 LAN 的两台设备中实现。

The current implementation is designed to be used exclusively in local area networks, though Internet use is theoretically supported. Only uncompressed audio is supported, hence you won't be able to transmit more than a few streams at the same time over a standard LAN.

Rygel

Your best bet is probably to setup a proper media server (like Rygel, which works well with Pulseaudio) to transcode the raw audio to something like MP3 and stream that instead.

其它 module

module-simple-protocol-tcp

An implementation of a simple protocol which allows playback by using simple tools like netcat. Just connect to the listening socket of this module and write the audio data to it, or read it from it for playback, resp. recording.

vlc 进行 http 串流

• 优点：客户端不用安装软件，只需要打开浏览器
• 缺点：延迟有 3-4s，不适用于播放视频场景

关键点：source 中默认有一个 monitor 的源，该源就可以获得输出到扬声器的声音

cvlc -vvv pulse://XXXX --sout '#transcode{acodec=mp3,ab=128,channels=2}:standard{access=http,dst=0.0.0.0:8888/pc.mp3}'

设备上

http://your.local.ip.address:8888/pc.mp3

各种串流

windows to linux

linux to anything

视频

v4l2

列出所有设备

root@tianyi ➜  ~ v4l2-ctl --list-devices
USB2 Video: USB2 Video (usb-0000:00:14.0-4):
        /dev/video2
        /dev/video3
        /dev/media1

EasyCamera: EasyCamera (usb-0000:00:14.0-5):
        /dev/video0
        /dev/video1
        /dev/media0

笔记本自带摄像头

root@tianyi ➜  ~ v4l2-ctl -d /dev/video0 --all
Driver Info:
        Driver name      : uvcvideo
        Card type        : EasyCamera: EasyCamera
        Bus info         : usb-0000:00:14.0-5
        Driver version   : 6.2.16
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format

Priority: 2
Video input : 0 (Input 1: ok)
Format Video Capture:
        Width/Height      : 640/480
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Bytes per Line    : 1280
        Size Image        : 614400
        Colorspace        : sRGB
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Default (maps to Limited Range)
        Flags             :
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 640, Height 480
        Default     : Left 0, Top 0, Width 640, Height 480
        Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 640, Height 480, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 640, Height 480, Flags:
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 0

采集卡

root@tianyi ➜  ~ v4l2-ctl -d /dev/video2 --all
Driver Info:
        Driver name      : uvcvideo
        Card type        : USB2 Video: USB2 Video
        Bus info         : usb-0000:00:14.0-4
        Driver version   : 6.2.16
        Capabilities     : 0x84a00001
                Video Capture
                Metadata Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Priority: 2
Video input : 0 (Input 1: ok)
Format Video Capture:
        Width/Height      : 2560/1600
        Pixel Format      : 'MJPG' (Motion-JPEG)
        Field             : None
        Bytes per Line    : 0
        Size Image        : 8192000
        Colorspace        : sRGB
        Transfer Function : Rec. 709
        YCbCr/HSV Encoding: ITU-R 601
        Quantization      : Default (maps to Full Range)
        Flags             :
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 2560, Height 1600
        Default     : Left 0, Top 0, Width 2560, Height 1600
        Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 2560, Height 1600, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 2560, Height 1600, Flags:
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 30.000 (30/1)
        Read buffers     : 0