Btrfs subvolumes support

Chrysostomus commented

2017-02-06 14:51:31 +00:00

(Migrated from gitlab2.manjaro.org)

I wish to port btrfs subvolume functions from previous manjaro cli-installer, so that installation to btrfs subvolumes can be done without manual mounting.

ghost commented

2017-02-18 11:14:43 +00:00

(Migrated from gitlab2.manjaro.org)

I you no one else volunteered I could give it a try.

Chrysostomus commented

2017-02-18 11:22:05 +00:00

(Migrated from gitlab2.manjaro.org)

Thanks! Greatly appreciated!

I poached the btrfs subvolume functions from the old cli-installer https://github.com/Chrysostomus/manjaro-architect/blob/master/bin/btrfs-subvol-functions.in
They might rely on some other stuff that has not yet been ported though.

Thanks! Greatly appreciated! I poached the btrfs subvolume functions from the old cli-installer https://github.com/Chrysostomus/manjaro-architect/blob/master/bin/btrfs-subvol-functions.in They might rely on some other stuff that has not yet been ported though.

ghost commented

2017-02-21 08:19:53 +00:00

(Migrated from gitlab2.manjaro.org)

I must conclude I am a lousy shell scripter :) But you manage all right :)

This might take me some time cause I can't get the my brain about the logic.

C#, javascript, Python was a lot easier to grasp than shell :)

I must conclude I am a lousy shell scripter :) But you manage all right :) This might take me some time cause I can't get the my brain about the logic. C#, javascript, Python was a lot easier to grasp than shell :)

Chrysostomus commented

2017-02-21 11:56:26 +00:00

(Migrated from gitlab2.manjaro.org)

It's okay. The schedule for this feature is "before summer"

The logic I would think that we need is

after choosing mount options for partition, if the chosen partition is btrfs, start subvolume creation function
subvolume creation function: ask user if they want to create a subvolume on this partition. Keep asking until they choose "done".

if user chooses yes, ask for name of the subvolume. In next window, ask mountpoint for the subvolume.
mount chosen mountpoint to chosen directory with mount options from previous menu
repeat loop

when loop exits, continue to next

No idea how to handle raid options, but I find it a lesser priority

It's okay. The schedule for this feature is "before summer" The logic I would think that we need is 1) after choosing mount options for partition, if the chosen partition is btrfs, start subvolume creation function 2) subvolume creation function: ask user if they want to create a subvolume on this partition. Keep asking until they choose "done". - if user chooses yes, ask for name of the subvolume. In next window, ask mountpoint for the subvolume. - mount chosen mountpoint to chosen directory with mount options from previous menu - repeat loop 3) when loop exits, continue to next No idea how to handle raid options, but I find it a lesser priority

ghost commented

2017-02-21 12:09:50 +00:00

(Migrated from gitlab2.manjaro.org)

On Tue, 2017-02-21 at 03:56 -0800, Chrysostomus wrote:

It's okay. The schedule for this feature is "before summer"
The logic I would think that we need is
after choosing mount options for partition, if the chosen partition
is btrfs, start subvolume creation function
subvolume creation function: ask user if they want to create a
subvolume on this partition. Keep asking until they choose "done".
if user chooses yes, ask for name of the subvolume. In next window,
ask mountpoint for the subvolume.
mount chosen mountpoint to chosen directory with mount options from
previous menu
repeat loop
when loop exits, continue to next
No idea how to handle raid options, but I find it a lesser priority

There is already som btrfs support - I just installed in a vm with
btrfs

On Tue, 2017-02-21 at 03:56 -0800, Chrysostomus wrote: > It's okay. The schedule for this feature is "before summer" > The logic I would think that we need is > after choosing mount options for partition, if the chosen partition > is btrfs, start subvolume creation function > subvolume creation function: ask user if they want to create a > subvolume on this partition. Keep asking until they choose "done". > if user chooses yes, ask for name of the subvolume. In next window, > ask mountpoint for the subvolume. > mount chosen mountpoint to chosen directory with mount options from > previous menu > repeat loop > when loop exits, continue to next > No idea how to handle raid options, but I find it a lesser priority There is already som btrfs support - I just installed in a vm with btrfs

Chrysostomus commented

2017-02-21 12:28:47 +00:00

(Migrated from gitlab2.manjaro.org)

Yes, I install half of the test systems on btrfs too. But if user wants to install on btrfs subvolumes to be able to take snapshots, he needs to create and mount them manually separate of the installer.

Idea would be to to integrate subvolume creation to existing btrfs functionality.

Yes, I install half of the test systems on btrfs too. But if user wants to install on btrfs subvolumes to be able to take snapshots, he needs to create and mount them manually separate of the installer. Idea would be to to integrate subvolume creation to existing btrfs functionality.

ghost commented

2017-02-21 12:42:14 +00:00

(Migrated from gitlab2.manjaro.org)

On Tue, 2017-02-21 at 04:28 -0800, Chrysostomus wrote:

Yes, I install half of the test systems on btrfs too. But if user
wants to install on btrfs subvolumes to be able to take snapshots, he
needs to create and mount them manually separate of the installer.
Idea would be to to integrate subvolume creation to existing btrfs
functionality.

I see. I have no knowledge of btrfs so I have familiarize myself with it.

But I find a lot of red flags on Arch wiie considering btrfs - so it seems it is not a better file system.

https://wiki.archlinux.org/index.php/Btrfs#File_system_creation

It seems risky to incorporate advanced features such as snapshots when so many red flags are up.

On Tue, 2017-02-21 at 04:28 -0800, Chrysostomus wrote: > Yes, I install half of the test systems on btrfs too. But if user > wants to install on btrfs subvolumes to be able to take snapshots, he > needs to create and mount them manually separate of the installer. > Idea would be to to integrate subvolume creation to existing btrfs > functionality. I see. I have no knowledge of btrfs so I have familiarize myself with it. But I find a lot of red flags on Arch wiie considering btrfs - so it seems it is not a better file system. https://wiki.archlinux.org/index.php/Btrfs#File_system_creation It seems risky to incorporate advanced features such as snapshots when so many red flags are up.

Chrysostomus commented

2017-02-21 12:50:43 +00:00

(Migrated from gitlab2.manjaro.org)

Subvolumes, snapshots and compression are stable features. Raid is too, except for raid5/6. The unstable ones are fsck, dedublication, quotas and raid5/6. Nothing in installer guides user to try unstable features.

And swap files cannot be used to.

Most gotchas of btrfs are already automatically handled handled by the installer. Incompatible settings are disabled from grub and fsck is removed from mkinitcpio.conf.

Subvolumes, snapshots and compression are stable features. Raid is too, except for raid5/6. The unstable ones are fsck, dedublication, quotas and raid5/6. Nothing in installer guides user to try unstable features. And swap files cannot be used to. Most gotchas of btrfs are already automatically handled handled by the installer. Incompatible settings are disabled from grub and fsck is removed from mkinitcpio.conf.

ghost commented

2017-02-21 18:00:48 +00:00

(Migrated from gitlab2.manjaro.org)

I continued your thoughts from

The logic I would think that we need is

select_filesystem

btrfs on partition
creating btrfs takes a long time
display message instead of buttons

https://wiki.archlinux.org/index.php/Btrfs#Mounting_subvolumes
Simplify subvolume layouts by not using the toplevel(ID=5) as /.
Instead create a subvolume on the subvolume and use that as /

the flow
format partition with btrfs
use selected mount options

what sane default mount options for annie doe?
noatime
recovery
space_cache (default in kernel?)
most recommendations is not to use compression with /home
most recommendations is to use compression with /
but to achieve both it must be separate disks as mount options for the first mount takes precedence

mkdir -p /mnt/btrfs
mount /dev/sdX2 /btrfs

It would be sane - in case of average user - to create a default structure for system if user choose btrfs?
Just to complicate it - also the experienced user should be able to mess with the basics?

create subvolume
btrfs create subvolume /mnt/btrfs/system
btrfs create subvolume /mnt/btrfs/system/root
btrfs create subvolume /mnt/btrfs/system/home
ask for more subvolumes - if yes create
btrfs create subvolume /mnt/btrfs/user
create and subsequent volumes
ask for snapshot or subvolume
btrfs create subvolume /mnt/btrfs/user/new_vol1
btrfs create snapshot /mnt/btrfs/user/new_vol2
etc
mount system subvolumes
create install_root
mkdir -p /mnt/install_root
mount -t btrfs -o subvol=system/root /dev/sdxY /mnt/install_root
mkdir /mnt/install_root/boot[/efi]
mkdir /mnt/install_root/home
mount -t btrfs -o subvol=/system/home /dev/sdx/ /mnt/install_root/home
mount /dev/sdX1 /mnt/install_root/boot[/efi]
adding/removing subvolumes requires mount of the basic partition
using the subvolumes can be achieved without mounting the basic partition
a user is required to have knowledge about btrfs to benefit from it
are snapshots done automagically?

I continued your thoughts from >The logic I would think that we need is select_filesystem * btrfs on partition creating btrfs takes a long time display message instead of buttons https://wiki.archlinux.org/index.php/Btrfs#Mounting_subvolumes Simplify subvolume layouts by not using the toplevel(ID=5) as /. Instead create a subvolume on the subvolume and use that as / the flow format partition with btrfs use selected mount options * what sane default mount options for annie doe? noatime recovery space_cache (default in kernel?) most recommendations is not to use compression with /home most recommendations is to use compression with / but to achieve both it must be separate disks as mount options for the first mount takes precedence mkdir -p /mnt/btrfs mount /dev/sdX2 /btrfs It would be sane - in case of average user - to create a default structure for system if user choose btrfs? Just to complicate it - also the experienced user should be able to mess with the basics? * create subvolume btrfs create subvolume /mnt/btrfs/system btrfs create subvolume /mnt/btrfs/system/root btrfs create subvolume /mnt/btrfs/system/home * ask for more subvolumes - if yes create btrfs create subvolume /mnt/btrfs/user create and subsequent volumes ask for snapshot or subvolume btrfs create subvolume /mnt/btrfs/user/new_vol1 btrfs create snapshot /mnt/btrfs/user/new_vol2 etc * mount system subvolumes create install_root mkdir -p /mnt/install_root mount -t btrfs -o subvol=system/root /dev/sdxY /mnt/install_root mkdir /mnt/install_root/boot[/efi] mkdir /mnt/install_root/home mount -t btrfs -o subvol=/system/home /dev/sdx/ /mnt/install_root/home mount /dev/sdX1 /mnt/install_root/boot[/efi] * adding/removing subvolumes requires mount of the basic partition * using the subvolumes can be achieved without mounting the basic partition a user is required to have knowledge about btrfs to benefit from it are snapshots done automagically?

ghost commented

2017-02-23 16:25:43 +00:00

(Migrated from gitlab2.manjaro.org)

@Chrysostomus
I must give in - I cannot get the bigger picture - it is much too confusing for me with variables where I do not know where the value comes from and why.

This shell scripting takes a special brain which I do not have.

At least I tried - keep up the good work 😺 t

@Chrysostomus I must give in - I cannot get the bigger picture - it is much too confusing for me with variables where I do not know where the value comes from and why. This shell scripting takes a special brain which I do not have. At least I tried - keep up the good work :smiley_cat: t

Chrysostomus commented

2017-02-23 16:26:30 +00:00

(Migrated from gitlab2.manjaro.org)

Thanks, I appreciate the effort!

Chrysostomus commented

2017-03-24 02:50:41 +00:00

(Migrated from gitlab2.manjaro.org)

I noticed now that the Acidburn0zzz branch has btrfs implemented. I'll take a look if we can import that...

Chrysostomus commented

2017-04-15 07:43:24 +00:00

(Migrated from gitlab2.manjaro.org)

I have been thinking about this. It feels too complicated to properly implement btrfs subvolumes in the current mount partitions phase, because the phase is split across several functions and the times I can work with this are few and far between.

I think it would be, for the time being, make a separate "mount btrfs subvolumes" menu item. It would be ran after basic partition mounting. It would

check if /mnt is on btrfs volume. If not, refuse to do anything and give explanation.
Offer automatic and manual mode. Automatic creates and mounts volumes @, @home, @var, @cache and @snapshots.
Manual mode ask for names for subvolumes, separated by spaces. Pressing Enter creates those volumes.
After subvolume creation unmount btrfs root and loop to ask a mountpoint for each of these subvolumes. Use mount options of the original btrfs root.
In the end, maybe offer to add devices to the btrfs volume?

What do you guys think? @oberon2007 @papajoker ?

I have been thinking about this. It feels too complicated to properly implement btrfs subvolumes in the current mount partitions phase, because the phase is split across several functions and the times I can work with this are few and far between. I think it would be, for the time being, make a separate "mount btrfs subvolumes" menu item. It would be ran after basic partition mounting. It would 1) check if /mnt is on btrfs volume. If not, refuse to do anything and give explanation. 2) Offer automatic and manual mode. Automatic creates and mounts volumes @, @home, @var, @cache and @snapshots. 3) Manual mode ask for names for subvolumes, separated by spaces. Pressing Enter creates those volumes. 4) After subvolume creation unmount btrfs root and loop to ask a mountpoint for each of these subvolumes. Use mount options of the original btrfs root. 5) In the end, maybe offer to add devices to the btrfs volume? What do you guys think? @oberon2007 @papajoker ?

Chrysostomus commented

2017-04-16 00:06:20 +00:00

(Migrated from gitlab2.manjaro.org)

something like this:

btrfs_subvolumes()

{
    if [[ $(lsblk -lno FSTYPE,MOUNTPOINT | awk '/ \/mnt$/ {print $1}') == btrfs ]]; then 
        DIALOG " Your root volume is not formatted in btrfs " --infobox "\nMount partitions first!\n " 0 0
        return 0
    fi
    #1) save mount options and name of the root partition 
    mount | grep "on /mnt " | grep -Po '(?<=\().*(?=\))' > /tmp/.root_mount_options
    lsblk -lno MOUNTPOINT,NAME | awk '/^\/mnt / {print $2}' > /tmp/.root_partition
    #2) choose automatic or manual mode
    DIALOG " Choose mode " --menu "\n$_Note\nAutomatic mode is deigned to\nallow integration with snapper,\nnon-recursive snapshots,\nseparating system and user data\nand restoring snapshots without losing data. " 0 0 2 \
      "1" "automatic" \
      "2" "manual" 2>/tmp/.subvol_mode

    if [[ $(cat /tmp/.subvol_mode) != "" ]]; then
        if [[ $(cat /tmp/.subvol_mode) -eq 2 ]]; then
            # Create subvolumes manually
            DIALOG " Create subvolumes " --inputbox "\nInput names of the subvolumes separated by spaces." 0 0 "" 2>/tmp/.subvols || return 0
            cd /mnt
            for subvol in $(cat /tmp/subvols); do
                btrfs subvolume create $subvol
            done
            cd
            # Mount subvolumes
            umount /mnt
            for sub in $(btrfs subvolume list -p /mnt); do
                DIALOG " Mount this subvolume " --inputbox "\nInput mountpoint of the subvolume\nas it would appear in installed system\n(without prepending /mnt)\n." 0 0 "" 2>/tmp/.mountp || return 0
                mount -o "$(cat /tmp/.root_mount_options)",subvol="$sub" /dev/"$(cat /tmp/.root_partition)" /mnt"$(cat /tmp/.mountp)"
            done
        else
            DIALOG " This creates subvolumes @ for /,\n@home for /home,\n@cache for /var/cache and subvolume snapshots. " --infobox "\nMount partitions first!\n " 0 0
            # Create subvolumes automatically
            cd /mnt
            btrfs subvolume create @
            btrfs subvolume create @home
            btrfs subvolume create @cache
            btrfs subvolume create snapshots
            cd
            # Mount subvolumes
            umount /mnt
            mount -o "$(cat /tmp/.root_mount_options)",subvol=@ /dev/"$(cat /tmp/.root_partition)" /mnt
            mkdir -p /mnt/home
            mkdir -p /mnt/var/cache
            mount -o "$(cat /tmp/.root_mount_options)",subvol=@home /dev/"$(cat /tmp/.root_partition)" /mnt/home
            mount -o "$(cat /tmp/.root_mount_options)",subvol=@cache /dev/"$(cat /tmp/.root_partition)" /mnt/var/cache
        fi
    else
        return 0
    fi
}

something like this: ``` btrfs_subvolumes() { if [[ $(lsblk -lno FSTYPE,MOUNTPOINT | awk '/ \/mnt$/ {print $1}') == btrfs ]]; then DIALOG " Your root volume is not formatted in btrfs " --infobox "\nMount partitions first!\n " 0 0 return 0 fi #1) save mount options and name of the root partition mount | grep "on /mnt " | grep -Po '(?<=$).*(?=$)' > /tmp/.root_mount_options lsblk -lno MOUNTPOINT,NAME | awk '/^\/mnt / {print $2}' > /tmp/.root_partition #2) choose automatic or manual mode DIALOG " Choose mode " --menu "\n$_Note\nAutomatic mode is deigned to\nallow integration with snapper,\nnon-recursive snapshots,\nseparating system and user data\nand restoring snapshots without losing data. " 0 0 2 \ "1" "automatic" \ "2" "manual" 2>/tmp/.subvol_mode if [[ $(cat /tmp/.subvol_mode) != "" ]]; then if [[ $(cat /tmp/.subvol_mode) -eq 2 ]]; then # Create subvolumes manually DIALOG " Create subvolumes " --inputbox "\nInput names of the subvolumes separated by spaces." 0 0 "" 2>/tmp/.subvols || return 0 cd /mnt for subvol in $(cat /tmp/subvols); do btrfs subvolume create $subvol done cd # Mount subvolumes umount /mnt for sub in $(btrfs subvolume list -p /mnt); do DIALOG " Mount this subvolume " --inputbox "\nInput mountpoint of the subvolume\nas it would appear in installed system\n(without prepending /mnt)\n." 0 0 "" 2>/tmp/.mountp || return 0 mount -o "$(cat /tmp/.root_mount_options)",subvol="$sub" /dev/"$(cat /tmp/.root_partition)" /mnt"$(cat /tmp/.mountp)" done else DIALOG " This creates subvolumes @ for /,\n@home for /home,\n@cache for /var/cache and subvolume snapshots. " --infobox "\nMount partitions first!\n " 0 0 # Create subvolumes automatically cd /mnt btrfs subvolume create @ btrfs subvolume create @home btrfs subvolume create @cache btrfs subvolume create snapshots cd # Mount subvolumes umount /mnt mount -o "$(cat /tmp/.root_mount_options)",subvol=@ /dev/"$(cat /tmp/.root_partition)" /mnt mkdir -p /mnt/home mkdir -p /mnt/var/cache mount -o "$(cat /tmp/.root_mount_options)",subvol=@home /dev/"$(cat /tmp/.root_partition)" /mnt/home mount -o "$(cat /tmp/.root_mount_options)",subvol=@cache /dev/"$(cat /tmp/.root_partition)" /mnt/var/cache fi else return 0 fi } ```

Chrysostomus commented

2017-04-16 15:31:19 +00:00

(Migrated from gitlab2.manjaro.org)

Making progress with this after moving it within the root volume mounting action. At least automatic mode seems to work now.

Chrysostomus commented

2017-04-17 07:06:01 +00:00

(Migrated from gitlab2.manjaro.org)

Pushed now to the online repo. Automatic and manual modes are still a bit rudimentary, but they seem to work somewhat...

Rows
Columns

Btrfs subvolumes support #9