181 lines
6.1 KiB
Markdown
181 lines
6.1 KiB
Markdown
# ariane
|
|
## Hard disks
|
|
| | V1 | V2 | V3 | V4 |
|
|
| :-: | :-: | :-: | :-: | :-: |
|
|
|**H1**|00 /dev/sda `6TB` <br> `WD-WX21D36PP32E`|04 |08 |12 |
|
|
|**H2**|01 /dev/sdb `6TB` <br> `WD-WX21D36PPLPH`|05 |09 |13 |
|
|
|**H3**|02 /dev/sdc `6TB` <br> `WD-WX21D36PP0K1`|06 |10 |14 |
|
|
|**H4**|03 /dev/sdd `6TB` <br> `WD-WXB1HB4MJCMM`|07 |11 |15 |
|
|
## Debian setup install steps
|
|
```
|
|
Boot Image Debian Netinstall on USB stick, Advanced Options->Expert Install
|
|
Language: English, Location: other->Europe->Austria
|
|
Locale: en_US.UTF-8, Additional Locale: de_AT.UTF-8, System Locale: en_US.UTF-8
|
|
Keyboard: German
|
|
Detect and mount CD-ROM, Load installer components: no extra
|
|
auf KISTL statische IP 192.168.86.7/24 für ariane.fet.htu.tuwien.ac.at einrichten
|
|
Detect network hardware, Configure Network: enp8s0: Hostname: ariane, IP 192.168.86.7/24 domain fet.htu.tuwien.ac.at
|
|
Setup Users and Passwords: shadow, user petra
|
|
Configure the clock: NTP Server: tutimea.tuwien.ac.at
|
|
Detect disks, partition disks: manual
|
|
RAID1 md0 with both SSDs https://blog.sleeplessbeastie.eu/2013/10/04/how-to-configure-software-raid1-during-installation-process/
|
|
/dev/nvmeXn1p1 32GB primary Volume für / in btrfs 32GB
|
|
/dev/nvmeXn1p2 80 GB primary Volume
|
|
ZFS RAIDZ-2 on is already on 4 x 6TB disks = 12TB 0 spares
|
|
partition1 ariane-root btrfs noatime, discard
|
|
Generic Kernel
|
|
Mirror: gd.tuwien.ac.at, allow backported
|
|
install with ssh server and standard sys utilities
|
|
Install Grub on /dev/nvme0n1 (to removable media path)
|
|
Install Grub on /dev/nvme1n1
|
|
reboot, log in as root
|
|
edit /etc/ssh/sshd_config set PermitRootlogin to yes
|
|
service sshd reload
|
|
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
|
|
ssh-copy-id root@ariane from client
|
|
edit /etc/ssh/sshd_config set PermitRootlogin to Prohibit-Password
|
|
```
|
|
## Install ZFS
|
|
```shell
|
|
apt-get install zfsutils-linux
|
|
modprobe zfs
|
|
systemctl restart zfs-import-cache
|
|
systemctl restart zfs-import-scan
|
|
systemctl restart zfs-mount
|
|
systemctl restart zfs-share
|
|
```
|
|
## Initial ZFS Setup
|
|
### Find disk serial numbers
|
|
```shell
|
|
for i in a b c d e f g h i; do echo -n "/dev/sd$i: "; hdparm -I /dev/sd$i | awk '/Serial Number/ {print $3}'; done
|
|
nvme list
|
|
```
|
|
### Wipe disks
|
|
```shell
|
|
mdadm --zero-superblock --force /dev/sda
|
|
sgdisk --zap-all /dev/sdX
|
|
```
|
|
### Partition disks
|
|
```shell
|
|
sgdisk -n1:0:0 -t1:BF01 /dev/sdX
|
|
```
|
|
### Format LUKS containers (and set passphrase)
|
|
```shell
|
|
cryptsetup luksFormat /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PP32E-part1
|
|
cryptsetup luksFormat /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPLPH-part1
|
|
cryptsetup luksFormat /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PP0K1-part1
|
|
cryptsetup luksFormat /dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4MJCMM-part1
|
|
```
|
|
### Create zpools
|
|
```shell
|
|
zfs_mount.sh mount
|
|
zpool create -o ashift=12 -o autoexpand=on -o autoreplace=on -O atime=off -O compression=lz4 \
|
|
-O acltype=posixacl -O xattr=sa zv1 raidz2 \
|
|
/dev/mapper/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PP32E-part1 \
|
|
/dev/mapper/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPLPH-part1 \
|
|
/dev/mapper/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PP0K1-part1 \
|
|
/dev/mapper/ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4MJCMM-part1
|
|
zpool create -o ashift=12 -O atime=off -O compression=lz4 ssd mirror /dev/disk/by-id/nvme-INTEL_SSDPEKKW128G7_BTPY63830DUM128A-part2 /dev/disk/by-id/nvme-INTEL_SSDPEKKW128G7_BTPY63750L9G128A-part2
|
|
```
|
|
### Create datasets
|
|
```shell
|
|
zfs create zv1/homes
|
|
zfs create zv1/daten
|
|
zfs create zv1/fotos
|
|
chown root:nogroup /zv1/homes
|
|
chown root:2000 /zv1/daten/ /zv1/fotos/
|
|
zfs create -o canmount=off -o setuid=off -o exec=off ssd/var
|
|
zfs create -o com.sun:auto-snapshot=false -o mountpoint=/var/lib/nfs ssd/var/nfs
|
|
zfs create -o com.sun:auto-snapshot=false -o exec=on ssd/var/cache
|
|
zfs create ssd/var/log
|
|
zfs create -o exec=on ssd/var/lxc
|
|
mv /var/cache/* /ssd/var/cache/
|
|
zfs set mountpoint=/var/cache/ ssd/var/cache
|
|
mv /var/log/* /ssd/var/log/
|
|
zfs set mountpoint=/var/log ssd/var/log
|
|
mv /var/lib/lxc/* /ssd/var/lxc/
|
|
zfs set mountpoint=/var/lib/lxc ssd/var/lxc
|
|
zfs create -o com.sun:auto-snapshot=false zv1/sojus
|
|
zfs create -o com.sun:auto-snapshot=false zv1/daten/Scans
|
|
```
|
|
### Set dataset quota
|
|
```shell
|
|
zfs set quota=1T zv1/homes zv1/daten zv1/fotos
|
|
zfs set quota=3T zv1/sojus
|
|
zfs set quota=5G zv1/daten/Scans
|
|
```
|
|
### If intend using ACL someday
|
|
```shell
|
|
zfs set xattr=sa zv1
|
|
zfs set acltype=posixacl zv1
|
|
zfs set aclinherit=passthrough zv1
|
|
```
|
|
#### If ACL for ZFS NFS share is implemented someday (not done!)
|
|
```shell
|
|
setfacl -R --set u::rwX,g::rwX,o::rwX /zv1/{daten,fotos}/
|
|
setfacl -d -R --set u::rwX,g::rwX,o::rwX /zv1/{daten,fotos}/
|
|
```
|
|
### Transfer data
|
|
```shell
|
|
# https://github.com/rubo77/rsync-homedir-excludes
|
|
rsync --stats -avhP --exclude-from=/var/tmp/ignorelist root@atlas:/home/ /zv1/homes/
|
|
rsync --stats -avhP --exclude Pictures root@atlas:/mnt/save/fet_daten/ /zv1/daten/
|
|
rsync --stats -avhP root@atlas:/mnt/save/fet_daten/Pictures/ /zv1/fotos/
|
|
```
|
|
### Share via NFS
|
|
```shell
|
|
zfs set sharenfs="rw=@192.168.86.0/24,insecure,sync,no_subtree_check,root_squash,no_wdelay" zv1/homes
|
|
zfs set sharenfs="rw=@192.168.86.0/24,insecure,sync,no_subtree_check,root_squash,crossmnt" zv1/daten
|
|
zfs set sharenfs="rw=@192.168.86.0/24,insecure,sync,no_subtree_check,root_squash,crossmnt" zv1/fotos
|
|
showmount -e
|
|
```
|
|
### Before reinstalling
|
|
```shell
|
|
zfs export
|
|
```
|
|
### After reinstalling
|
|
```shell
|
|
./ansible-playbook -i hosts/production site.yml --tags common --limit ariane
|
|
ssh ariane apt-get update && apt-get upgrade && reboot
|
|
./ansible-playbook -i hosts/production site.yml --tags zfs --limit ariane
|
|
zfs import
|
|
zpool import zv1
|
|
zfs mount -a
|
|
```
|
|
## ZFS Maintenance
|
|
```shell
|
|
zpool status -v
|
|
zpool list -v
|
|
zpool get all
|
|
zpool history
|
|
zpool scrub zv1
|
|
zfs list -o space
|
|
```
|
|
### show and delete all snapshots
|
|
```shell
|
|
zfs list -t snapshot
|
|
zfs list -H -o name -t snapshot | xargs -n1 zfs destroy
|
|
```
|
|
## MDADM Maintenance
|
|
```shell
|
|
mdadm --misc --detail /dev/md0
|
|
echo check > /sys/block/md0/md/sync_action
|
|
watch -n 0.1 cat /proc/mdstat
|
|
```
|
|
## Start System
|
|
```shell
|
|
zfs_mount.sh mount # Enter Passphrase
|
|
# start all LXC containers
|
|
```
|
|
## Reboot System
|
|
```shell
|
|
# stop all LXC containers
|
|
zfs_mount.sh reboot
|
|
```
|
|
## Shutdown System
|
|
```shell
|
|
# stop all LXC containers
|
|
zfs_mount.sh unmount
|
|
halt -p
|
|
```
|