Boot times on live media are too long #136
Labels
No labels
Done
bug
buildiso
buildpkg
buildtree
critical
deployiso
duplicate
enhancement
help wanted
in progress
invalid
manjaro-chroot
note
old-not-relevant
optional
question
sonar
todo
wontfix
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
tools/manjaro-tools#136
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Created by: NuLogicSystems
As reported by Clemens, It currently takes over 45 seconds to boot in live mode.
These services seem to be the majority of the issue:
27.822s mhwd-live.service
9.754s livecd.service
3.993s pacman-init.service
3.824s dev-sdb1.device
2.244s ModemManager.service
Created by: philmmanjaro
Hi James,
mhwd-live is around 25s, which is normal. 9s for livecd is also normal. pacman-init checks all servers. Depending on the internet-speed it might take a while to get it done. Fastest bootup is always with free drivers and no internet connection. I'll take a look at it again.
Created by: philmmanjaro
This is a normal bootup as it gets with Virtualbox
This is totally normal and expected. Never had any issues with it before.
Created by: NuLogicSystems
Well the above is what I get on bare metal, Intel core 3 with 8 gigs of ram
and Intel only graphics with no wired internet connection, using the free
driver option. 45 - 50 seconds to boot a live media is way too long and leaves a bad first impression.
Created by: NuLogicSystems
Here is the entire output of systemd-analyze blame:
Created by: philmmanjaro
Well, if I start the same image in EFI mode you have to wait around 1:28 min to have the XFCE desktop loaded. With BIOS it is around 1 minute. This is normal. EFI needs ca. 39 seconds to load the kernel image and display the first systemd output. This is a different story and was always present so far.
Created by: NuLogicSystems
This is not an issue for me personally, but Clemens is used to faster boot times in Ubuntu, and even on manjaro prior to my pre6 ISO.
Created by: NuLogicSystems
However, the MHWD service was much shorter prior to the fix for the dynamic linker cache issue.
Created by: philmmanjaro
Well, Ubuntu has all drivers pre-installed. So no package manager needs to be triggered to verify the package and install it. Additionally the configuration time is not there. To speed it up you have to create additional overlays you load with having the drivers pre-installed on them. This results in more overlay and bigger install medias as mostly the given drivers collide with each other. In the past I tried that to speed up the boot-process but was dismissed by the community. I'll check now an older install media to compare the speed against Manjaro 15.12-alpha2 I already posted above.
Created by: NuLogicSystems
Please do.
Personally, I think if the times are comparable maybe it's just more
noticeable now that it only shows these few services loading and doesn't
scroll the list of what is happening in between like it used to.
Created by: philmmanjaro
I just booted up Manjaro 0.8.0 which took around one minute to load the XFCE desktop. However systemd-analyze is there broken due missing packages. I'll try another one from 2013.
Created by: NuLogicSystems
I wonder why your getting 14 sec and on my system i'm getting 27 with the same ISO?
Could it be due to actual hardware verses virtual-box?
Created by: philmmanjaro
I really see no issues in speed changes over time:
However this image has no plymouth enabled ...
Created by: NuLogicSystems
Question, How is it that livecd.service is longer than mhwd.service on that ISO and the opposite on the newer ones?
Anyway, I see that mhwd.service has always taken around 15+ seconds in virtualbox.
I'm just wondering what do you get booting the ISO on the hardware itself?
Created by: philmmanjaro
We have one speciality with Manjaro 0.8.4:
Over time the technique changed. You can see that in the plot how long each service took. Systemd got also more and more complex over time. This delays it also. Sometimes the hardware installation was inside of livecd.service on newer ones we splitted it up. Also changing from aufs to overlayfs might have changed some stuff. However the complete bootup regarding mhwd mostly didn't changed.
Created by: Kirek
mhwd.service at a point was broken (not all xorg packages needed by mhwd where in the livecd), so mhwd did nothing because there weren't all the stuff there, and the only reason it booted at all is because the xorg server ships with the modesetting driver by default... but yes it was faster.
Created by: Kirek
We could probably slash a second or two in the livecd.service if we generate all locales when building the iso.
Created by: philmmanjaro
Also just a notice: i686 boots faster than x86_64 as those also load lib32 drivers.
Here is the current take on 0.8.12 i686:
So on my end it is just fine as always. Why does it take much longer on hardware?
Simple: a CD/DVD is slower in booting up. On USB-Sticks it depends on the transfer speed. SD cards might be faster. And with VirtualBox the ISO will be loaded directly from RAM. Hope this clears it up.
And yes, in the past we didn't generate any locales at all. This was an issue in the past. Generating all locales will result in bigger install medias and make no sense, at least for me.
Created by: udeved
Its no problem with openrc.
livecd takes about 30 secs to boot into DE.
Created by: NuLogicSystems
Clemens just posted this video to an issue tracker for netrunner rolling:
https://github.com/netrunner-rolling/netrunner-rolling-issues/issues/1
He wan't the ISO to be error free.
Created by: philmmanjaro
Ramon added some changes to make more services run parallel. However, did you tried openrc yet. Marcus told me already it is sometimes faster than systemd. Most of the posted issues are not present with this init system.
Created by: NuLogicSystems
I'll rebuild Manjaro-tools-git to incorporate the changes.
I guess I can ask if Clemens would like me to try openrc, but I'm not sure how that will go.
Please take a look at the issue tracker link I posted so you can see what he's asking me to do.
Created by: Kirek
The changes are in the manjaro-tools-livecd package:
github.com/manjaro/manjaro-tools-livecd@f2f495335cGrab it from unstable or build it yourself.
Created by: philmmanjaro
Well, we didn't change manjaro-tools but manjaro-livecd. You can find those packages in our unstable branch. I'll keep track on it.
Created by: Kirek
mhwd-live will still take a long time thought, but at least it will start a little earlier
Created by: philmmanjaro
You might also disable plymouth so Clemens sees what is going on. When he is fine with it you can put it back on ;)
Created by: NuLogicSystems
I used the manjaro-tools-git pkgbuild to build the devel branch so it builds all three packages at once. Please take a look at the link I posted.
Created by: Kirek
"switch_root: failed to unlink ro_branch: Directory not empy" message has been always in Manjaro, no idea how to supress or fix it, but its never the cause for a halted boot (but because of quiet boot it may be the last message you see).
Created by: Kirek
the manjaro-tools-livecd is a different PKGBUILD...
Created by: NuLogicSystems
I already tried disabling plymouth and removing quite from grub, it still only shows the three services.
Created by: NuLogicSystems
Oh OK, I was thinking of iso not livecd. I'll build an ISO using unstable to see the difference then.
Created by: NuLogicSystems
If you watch the video he is hitting esc to exit plymouth anyway, that's the other issue he want's me to fix,
"plymouthd: could not start boot splash: no such file or directory", I can't fix that one either and it happens on the Manjaro KDE ISO as well. I'm pretty sure this has something to do with plymouthd and systemd, but I'm not positive what causes this as it works on shutdown.
Created by: philmmanjaro
Latest Manjaro 15.12 has following times:
Well, it is faster than the last build.
Created by: philmmanjaro
So here are the latest Capella boots in both architectures using VirtualBox. I think faster we don't get anymore with the current development state:
Created by: Kirek
I spend some time checking how much time it takes to run every item in the livecd script:
Seems the big item is this line:
depmod $(cat /usr/lib/modules/$(uname -r)/extramodules/version)
What exactly it does?
Created by: philmmanjaro
Hmm, it was introduced with 0.8.2 back in 2012. I've to check if this is still needed.
Created by: philmmanjaro
@Kirek: can you bring your time code into the livecd.log. Would be nice to have it displayed like this.
Created by: philmmanjaro
Even with enhanced tuning the total boot-up time varies always around 30 - 35s:
Created by: Kirek
I doubt we can reduce this much more unless we change how we configure the graphics drivers.
Created by: udeved
Please make sure that this code is only run with systemd.
Created by: philmmanjaro
Why, this code is universal.
Created by: udeved
How? systemd-analyse is not present on openrc system.
Created by: philmmanjaro
We don't use systemd-analyze for our log files. We use date. I just posted my results by using systemd-analyze to have a proper result.
Created by: udeved
Is the systmd boot time ok now?
Created by: philmmanjaro
See the posted stats here. Yes, it is faster now.
Created by: udeved
So can we close it?
Created by: philmmanjaro
Let AJSlye decide that. From our end it is what we could do so far. Everything else would be a redesign of how mhwd works.
Created by: NuLogicSystems
I wouldn't be the one to decide this, Clemens would.
I do agree with Philip here, anything else would require me to find a way around using MHWD on the ISO and load the all the xorg drivers by default, which would probably be more work than it's worth.
Created by: udeved
I could readd a feature that can install x11 stuff in a chroot instead of iso repo and on the fly.
So there would be a x11drivers image instead of mhwd repo.
Created by: NuLogicSystems
That would not only be cool but a big source of my headaches gone.
If this could be enabled or disabled (managed) by a true/false switch in the manjaro-tools.conf file as well that would be awsome.
Created by: udeved
Right, thats the idea, a switch.
I will do this in x11 branch until merged with devel branch.
Created by: Kirek
You can try that now, you only have to remove stuff from Packages-Mhwd and put them in Packages-live and see how it goes.
Created by: udeved
The question is, do we want an optional x11 drivers layer instead of mhwd repo?
Mr AjJ needs to say.
Created by: Kirek
No, the question is: will it work ? Not only boot, and not only for graphic drivers, but also vm, networks drivers and other stuff. And calamares/thus also runs a mhwd script that depends on this mhwd repo.
It's not that simple.
Created by: udeved
Is there mhwd on netrunner? I never tested the netrunner.
Created by: Kirek
Of course it is, its based on manjaro...
Created by: NuLogicSystems
This is ultimately your distro and tools, all I can do is give you an excerpt from an email sent to me from Clemens:
"It takes 38 seconds on my intel gfx machine with 4gb ram and recent cpu.
Even 20 seconds are intolerable, so lets get rid of this and revert back to whatever any other distro but manjaro does."
If it would work then having this as an option would be useful. As far as calameres and thus, I wouldn't suggest changing the way they work. The issue is currently only on the live media, not the installed system.
Created by: NuLogicSystems
The Rolling Edition of Netrunner is currently based on Manjaro, Netrunner standard edition is based on Kubuntu. The issue is that the kubuntu based live media is currently booting faster than the Manjaro based ISO. This shouldn't be the case, Manjaro should be lighter and faster right?
Created by: udeved
I guess the mhwd repo on iso approach is simply a bit slow as it is.
Its not much that can be done there from manjaro-tools, except a mentioned optional layer.
But, I think this would need some changes in mhwd and how it configures drivers.
Created by: Kirek
As we said the problem is how mhwd does it thing, the main problem even if we preinstall all the packages-mhwd stuff, mhwd still insist on reinstalling them, so we don't gain anything unless we make mhwd a little smarter.
Created by: Kirek
@AJSlye
This problem is in the live media, in the installed system doesn't run mhwd-live, that's why we basically ignored this issue for a long time.
Created by: udeved
I noticed that the time of mhwd live script increases, if I enable prop drivers, ie the repo gets bigger, mor packages.
Only free drivers is faster.
Created by: NuLogicSystems
Right, how many chipsets does MHWD really need to look for?
I'm sure we only need to look for Intel, Nvidia, ATI/AND and possibly sis and VIA, I mean how many people actually have neomagic, trident, tseng, etc. any longer?
Created by: Kirek
Looking for the correct drivers takes no time, what it happen is that mhwd runs pacman to install the correct packages and that is what it takes time (unpacking them...)
Created by: philmmanjaro
Well, mhwd is already smart and will skip the packages. We also have the feature already included into manjaroiso from the get go. I simply disabled it due the fact that the ISO is bigger. However, there is no need to edit mhwd at all. We simply agreed to "drop" that feature for the always released I install only the needed stuff.
Created by: udeved
If I understand right, if we reintroduce an optional chroot drivers image, the iso would boot considerably faster, but the iso size increases naturally.
It would be of use for AJ.
Created by: NuLogicSystems
Yea, the ISO size doesn't really mater since the older ISO's Leszek made by extracting your release ISO's and modifying them we're already 2.4 gig. The ISO's I've been making using Manjaro-tools are only 1.9 gig, so there is room for expansion. Though I still say let's not just turn it back on but make it an option for anyone that may need it.
Created by: Kirek
I'm testing now what happens if we preinstall all the packages that mhwd needs.
Created by: udeved
@AJSlye
I will make that happen in a drvopt branch.
Just modify later the git build to use the brach, while kirek tests the mhwd behaviour
Created by: NuLogicSystems
OK, sounds good. Will there be an option I need to modify in manjaro-tools.conf or profile.conf?
Created by: philmmanjaro
Here we go for the history:
I think 0.8.1 was the only release we did so far using that feature. So how does it work?
You might have now conflicts as we didn't had Catalyst back then. For free drivers it is easy to be done. Remember, some packages are already preinstalled like ndiswrapper.
The feature to have ultra fast bootup was always there, but since 2012 disabled. I never felt the need to have it as we blew CD-size always. Back then we wanted to have the best live-cd with a most packages but all drivers on the disk not pre-installing all driver. And this goal has to be kept. This is the only feature which makes Manjaro Manjaro. Not installing unneeded stuff from the get-go. However for live-session it doesn't matter. It simply will skip the installations.
I also recommend to pre-install only nvidia and let the other packages go. Clemens only needs the free drivers preinstalled to have his ultra fast bootup.
Why I kept it from you? Well, nobody read the given code or asked me. I also never thought this will ever pop up again as we always did it the way we did it.
Created by: philmmanjaro
@AJSlye: at the moment it is disabled and not used for years. I've to dig up my old code and re-introduce that feature. It won't make it for 15.09 release. I can try but I can't promise. This was always ultra low for me and I was always against it to do so. I simply don't give a shit what others think about it. Simply like that. It always bothered me how the other distros had to uninstall things. This was designed for Chakra and adopted by Manjaro and I won't change it. I only might re-introduce the feature to have some preinstalled drivers for the live-session.
Created by: NuLogicSystems
Yes, I think for the live-session only as well, If only to give a great first impression from the live media.
The installed system wouldn't need this anyway and boots fast enough as it is.
Created by: udeved
@AJSlye
I implemented a mhwd_repo switch, true by default.
It will be in the profile.conf
Created by: Kirek
We don't really need to make major changes, it's a matter to add the drivers we want in the Package-Live list (or merge two list) this way we don't need to make extra overlay images.
Created by: philmmanjaro
No, we need additional overlays, otherwise we have all the "shit" preinstalled. That is not the way to go. You can add it for ease to the Livecd-Overlay package list. I recommend to add:
Never add them to your Desktop-Package-List.
Created by: udeved
On mhwd_repo=false list merge, mhwd with livecd, and skip mhwd repo?
I think an additiona layer is no problem, perhaps more flexible also
Created by: philmmanjaro
@udeved: no, we need mhwd overlay always. You can only add additional drivers to the Livecd-Overlay package list. Thats it. See the post above.
Created by: Kirek
This is the results, in my test it was usually between 30 and 40s before.
Created by: philmmanjaro
Why do we need the mhwd-overlay? Simple: we need it for offline installations as not always internet is present. If the driver is not present mhwd will try to download it or fails. Pre-installed drivers will simply skip the installation step in mhwd, it stills configures the drivers, though.
Created by: udeved
I was thinking we simply have the switch, and if AJ wants it to be preinstalled, he simply deactivates mhwd_repo switch, and all the drivers in Packages-Mhwd will be installed. Almost no changes, just to install instead of making repo.
Created by: NuLogicSystems
Yes, we need just pre-install the drivers on the live session, but not remove the overlay, calamares and thus need these if an internet connection is not available.
Created by: NuLogicSystems
Oops, sorry, I hit the wrong button.
Created by: udeved
Ok, then its basically no changes and you fiddle with your Livecd list then. :)
Created by: philmmanjaro
See, mhwd is already fast. It was always the package manager ;)
@udeved: No, you still don't get it. mwhd-overlay is needed for offline installs. All drivers provided by mhwd-db needs to be present in that overlay. There is no switch needed at all. manjaro-tools has the feature already. Simply add those pre-installed drivers to your Livecd-Overlay-Package list. Thats it.
Created by: udeved
See last post.
Created by: NuLogicSystems
Yes, I think I get it.
Kerik, is that all you did when you tested to get 4 sec. instead of 20+?
Created by: philmmanjaro
Yep, your to fast for me ;). Hope everybody got it now and we can close this one here. @udeved. I'll add a new present file so everybody gets it. @AJSlye simply link that then as your live-cd-package list.
Created by: Kirek
@AJSlye
Just copy everything from Packages-Mhwd to Packages-Live
You can put the whole list of Packages-Mhwd but only ONE nvidia nonfree driver (you will need to remove 304 and 320.
Created by: NuLogicSystems
OK, thanks philip.
I think if we can get 4 - 5 sec instead of 20+ Clemens will be satisfied. ;)
Created by: NuLogicSystems
I think I'd rather leave the proprietary drivers even nvidia for booting with that option from syslinux.
Hopefully only the free drivers are needed to be installed on the ISO.
Created by: Kirek
I'm thinking that maybe locale-gen can be optimized. Afaik we generate en/de locales always in the iso, so if the user select this languages at boot (or in efi its always en_US), we could skip that:
Created by: philmmanjaro
Most boot it anyway for German or English. So this won't change anything. We can still go the road and do more locales. It always results in a bigger install media. We might add a config file for that, so everybody can decide if he wants to go that road or not.
Created by: Kirek
This locales are already generated in the images, so if the user select this languages we can skip regenerating them. It will only be for those two.
Created by: philmmanjaro
Ah ok, go for it.
Created by: philmmanjaro
And this also shows how the status was back then in 2012 and now, couple years later. Keep in mind to make it always optional and fully flexible so the maintainer can decide if he wants it. Also we should start to document all those features. Else you have always to ask me or read all the given code. For me it will be also harder as now more people contribute to our tools. Hope this will also show how important it is to document. And never try to drive me nuts. I really hate to explain my past decisions and why I felt them that way.
Created by: philmmanjaro
Only two open issues for 0.9.13. We are getting close now.
Created by: udeved
Why not achieve the same but have besides mhwd repo image a drivers image optional using same mhwd list?
Created by: philmmanjaro
Why do we need another overlay? livecd-overlay is exactly the overlay for packages we have just for the live-session. Those drivers are also only needed for that session. So adding another overlay will make no sense. However, if we want to have a switch, then yes - another overlay would make sense. In the end each maintainer should decide if he wants to have a small ISO or a bigger one but fast boot-up. Keep it kiss. Why should we add something which is obviously not needed anymore.
Created by: NuLogicSystems
I agree with Phillip, if we can speed this up without making things overly complex then that is the rout to take.
Created by: philmmanjaro
I had this discussion years ago. And it was almost the same. So to solve this: we simply provide a solution both sides will get happy. On my end: I really don't need it. However we might use it for the official supported editions. LxQt and other maintainers won't use it at all.
Created by: udeved
Its clean in terms of package lists.
The thought was, why not, we use something called overlayfs.
It should give same size. I don't want to force it, but I have a branch ready so we can simply test.
Created by: NuLogicSystems
Yes, but does this branch deal with the fact that Calamares and Thus need the mhwd-overlay to install properly when offline?
Created by: udeved
Yes, we would just have an additional layer on switch, which is just the mhwd packages also installed besides the repo layer.
The only difference is additional layer, but no changes to livecd lists.
Created by: philmmanjaro
We can test your ideas for sure. However we should decide it as a team. I see your approach on this one. Re-inventing the wheel all the time over and over again is not the way to go. This is now the fast solution. If we get a smarter one, then fine. But please let this endless discussion go. I get more and more tired regarding this matter. I always designed my tools to be flexible for almost all cases.
The facts are:
Created by: philmmanjaro
@udeved: please check my provided file as it differs from the mhwd package list for given reasons. Else I'm fine with it. If it makes you happy, then go for it - but at least get somebody test it before merging it into devel. And please ask the team before you change some stuff to your liking. I always see some dudes get angry cos of that fact. We are a team and no one-man show anymore.
Created by: NuLogicSystems
OK Phillip, I'm cloning manjaro-tools-profiles now to merge your changes to my profiles.
Created by: udeved
This is the idea. Totally optional.
Listen guys, I want to provide the best solution, and not the fastest, it may not be my one, its why I said, I have it in separate branch. I am personally totally neutral on this one, because I will stay with standard for my profile.
Created by: philmmanjaro
@udeved: the package list is not the same. We need another one. Look before you post something. The idea is ok, but you should start listen to others also. And read those posts again. With that current draft you will ran into package conflicts.
And here is another bug report which will do the opposite we are currently discussing - extend the boot-time: https://github.com/manjaro/manjaro-tools/issues/140
So all in all this will and will be always a never ending story ...
Created by: NuLogicSystems
Philip, would I just remove Package-Livecd and replace it with packages-Livecd-Xorg?
Created by: udeved
I just don't understand the exclusive either or, I mean you can change livecd list, but some people may want to have preinstalled drivers for whatever reason in a separate image. Making the drivers image loading a new list is not the issue I think.
Created by: philmmanjaro
@AJSlye A soft-link to that file would be also ok. If you don't have the shared folder, then yes - replace the provided one with yours.
Created by: NuLogicSystems
Will, the softlink be followed, the name of the link changes with the file. I keep both shared and Netrunner in there own separate folder from what I download from git since I changed syslinux as well. I then merge changes to the KDE and shared profiles there with mine. This all helps me keep organized.
Created by: philmmanjaro
I've no exclusive either or topic here. I provided a fast and working solution to this matter. You can use that provided file and get rid of the normal live-session packages and call it Packages-MHWD-Live or what ever. You can't add all proprietary drivers to it. So I only added nvidia as this one is the most used driver. All the other drivers will be still installed old-school. If it is now in livecd-overlay or in another one - it would be the same.
However you have to modify isomounts and so on. So more modifications are needed to get it work the way you like it. If you want to have each proprietary driver fast-boot, several overlays are needed and a smarter boot-screen with more options. All in all it will get more complex as simply adding those drivers to Packages-Livecd.
I'm open for all but I'm slowly but soon get sick about this situation talking about it again and again to come to no point. If you have the time to go the extra mile, then do it. I don't hinder you. But please include the team into it as everybody else does too.
Created by: philmmanjaro
@AJSlye now it feels more like Kindergarden. Did you checked your profile folder? There are several files and folders symlinked to shared. On my end I've only Packages-Xfce and the profile.conf as real files. Either I'm to long on a PC or you all drive me nuts some how ...
Created by: udeved
Where is the formular to file to start a branch?
Created by: NuLogicSystems
Yes when I changed the link from Packages-Livecd to Pakages-livecd-xorg the link name will change as well. I was just wondering if that would make a difference as far as buildiso was concerned. I mean if the link changes would manjaro-tools still follow it? I'm just concerned that Manjaro tools is hardcoded to these file names and won't recognize Pakages-livecd-xorg and skip it even if it's a softlink, that's all.
Created by: philmmanjaro
@udeved for real? Do I really need to make some Popcorn now?
This is common sense to create a new branch if you want to try out something. Even in Calamares we do it.
Created by: udeved
You mistake a little joke.
Calm down please, and we can settle whatever it is per email.
Created by: philmmanjaro
I'm calm. I even don't started yet. Seems it was a long day for me. Good news is, the current git-checkout created my dev-build just fine. Will install it now to see how it went.
Created by: NuLogicSystems
Since either way should work and I'm keeping my profiles seperate from the rest, I'll just replace livecd with livecd-xorg. Thanks phillip, i'm going to test this out tonight. :)
Created by: philmmanjaro
@AJSlye hope this will make Clemens happy. What ever the outcome will be, we have found a proper solution for the boot-time matter by now. As it is a hot topic it might even pop up again. Try to boot Ubuntu from CD. It takes for ages ;)
Created by: NuLogicSystems
Yes, I know. And how is it that kubuntu is so much faster than Ubuntu at boot times, etc., this makes no sense to me. LOL
Created by: philmmanjaro
Magic ;) - Well I can't find any regressions so far (https://github.com/manjaro/manjaro-tools/issues/141). We are good to go. Will merge now the current packages with testing. manjaro-tools should be still used from git. I still have to decide how we provide a snapshot of all compatible profiles for that package. Else the community will complain again that the master profiles don't work. Maybe reintroducing the devel branch might help. However our maintainers should commit then to the right branch. This is and will be always tricky.
Created by: Kirek
I don't think we made any profile breaking change in this cycle.
And going back to boot times, right now the slower service is
pacman-init.Is there any reason we initialize the pacman keys at boot instead of doing it in buildiso?
Created by: NuLogicSystems
Man, I just started building against testing. LOL.
Never mind me, it's already in the compressing stage. I'll just build another one later.
Created by: philmmanjaro
@Kirek I've to check why I introduced pacman-init. It should be also possible to do it within creation of the root-image. If I remember it correctly we had a bug and needed to create it during bootup. It is always good to optimize it. Mostly I left a note with the commit in pacman ...
Created by: philmmanjaro
@AJSlye you simply worry to much. Create the image and tell Clemens the dudes of Manjaro might even reduce the boot-time even more. However you will have already the half time needed as before ...
Created by: NuLogicSystems
I was mostly making a joke, I had already updated manjaro-tools-git from the devel branch and my profile before starting buildiso. I think I only missed the locals commit but good enough to test boot times.
And yes, this will definitely make him happy.
Created by: philmmanjaro
@Kirek simple - security reasons: initializes a new random GnuPG key on every boot.
So in the past we had the issue that every Manjaro install media had the same key. So everybody had the same key. So we introduced it to create one on bootup and copy that key to the to-be-installed system.
We can do now this:
I'd say, fuck the 4 - 8 seconds we gain out of it for security reasons.
Created by: Kirek
Ok that makes sense, we will keep as it is.
Created by: udeved
8 secs on systemd to init pacman?
is there other service started inbetween haveged and pacman-init?
Created by: NuLogicSystems
Yes, I think that was an approx number since systemd initializes services in parallel.
Created by: udeved
Doesn't take that long on openrc with parallel boot enabled.
Created by: philmmanjaro
Well, haveged needs to be started first then it gens a random number and then pacman-key kicks in. If you don't use haveged it takes for ages. You even have to move your mouse or type buttons to get a random number generated. This also points out that openrc is in some ways the better init-system :P. @AJSlye: recommend it to Clemens and see what he thinks about it ... Does the power features fully work on KDE? @udeved ...
Created by: NuLogicSystems
I actually haven't got around to asking that one yet. I wan't to do a test build with openrc on my end first.
However, I'm pretty sure I know what he would say.
Created by: NuLogicSystems
OK, the resulting ISO is only 1990 MB, that's not so bad actually. Going to boot it now and run systemd-analyze blame.
Created by: philmmanjaro
Great. Now you have to create a plot of your boot-time and post both, the one before and after to see the difference ;) Then we can go all to bed ...
Created by: udeved
I just thought, perhaps the unit files can be somehow optimized, if the parallel funnily slows it down.
If...
Created by: philmmanjaro
systemd is sometimes a blackbox ;) Welcome to MacOS
Created by: NuLogicSystems
OK, not sure about plotting it but now get:
Result: Startup finished in 15.431s (kernel) + 9.895s (userspace) = 25.326s
Before:
Created by: NuLogicSystems
The strange thing is, now nmbd and smdb are actually loading when they we're not before, even though I had them added to my profile
Created by: philmmanjaro
Is that a good thing? To plot you would do
systemd-analyze plot > output.svgand create a jpg, png out of it ;)Created by: Kirek
systemd-analyze blame is not a full list of loaded services...
Pararellizing pacman-init, livecd and mhwd-live is one of the first optimizations I did after this issue was opened and reduced the boot times for @philmmanjaro from 42-46s to 32-35s... so there is no issue there.
Created by: NuLogicSystems
OK, I didn't know that one. I'll do that when I get back then.
Created by: NuLogicSystems
Thanks for the explanation there Kirek, that makes sense now. ;)
Created by: philmmanjaro
Yes, blame only lists those services which took the longest. However with plot you see which service were started, when, how long it actually took and in which context ...
Created by: Kirek
No, you don't need them.
Created by: NuLogicSystems
@Kirek
I had removed the question, but thanks for answering it anyway.
If I run into an issue at least now I know I can remove those.
Created by: NuLogicSystems
OK, So I removed the nvidia driver from being installed to the live-session as well as thus and the cli installer (Netrunner doesn't use these since we have our own build of Calamares) and now the ISO is back to fitting on a 2 gig usb stick with 7 megs to spare (1920 MB).
Created by: philmmanjaro
@AJSlye did you got any feedback from Clemens? I assume we can close this one here now ...
Created by: NuLogicSystems
I'm uploading a 2015.09-RC1 ISO based on these changes now, I will get feed
back possibly tomorrow.
Created by: philmmanjaro
Fine. no rush there ;)
Created by: NuLogicSystems
If you would like to test the ISO I just pushed to the Blue Systems repository, then please do so, but as always do not share this to the public unless Clemens give the go ahead.
Created by: philmmanjaro
Netrunner 15.09-rc1 (2015-09-24)
Here is my current analysis. Somehow Netrunner is slower in bootup than Manjaro KDE-Minimal 15.12-dev
Created by: NuLogicSystems
Yes, Netrunner Rolling is far from minimal.
We added smbd and nmbd to the start-systemd= line because Clemens wanted samba-mounter working out of the box. And of course zramswap as well or it wouldn't be Netrunner.
Created by: philmmanjaro
Here is the analysis from Manjaro 15.12-dev (2015-09-24):
Created by: NuLogicSystems
We also have some deviations for ksystemsettings5, plasma shell and default panel that slows down the Plasma desktop from loading a bit. Netrunner makes quite a few default settings and modifications.
Created by: NuLogicSystems
Yes, but it's only a 7 sec difference between them.
Clemens just emailed me that the 25 sec. it's taking now is acceptable compared to the 45-50 sec he was previously getting.
Also going back through this issue how is it that now your getting 5 sec for mhwd-live when all your previous posts it was around 15 sec? Unless of course you built the manjaro ISO installing the drivers as well and comparing oranges to oranges.
Created by: NuLogicSystems
Oh yea, we also added sys to the addgroups= line so that we don't need a root password for cups administration. Like I said, we make many, many modifications and changes.
Created by: NuLogicSystems
Since there are no further Issues, I'm marking this closed.
Created by: udeved
Just for info, an iso with preinstalled drivers layer and repo layer with only free drivers increases iso +10MB. 585MB vs 595MB lxqt-minimal
Created by: philmmanjaro
Free drivers are always small. The big part are those proprietary ones ...
Created by: udeved
I know, but its rather attractive for free drivers only withonly 10MB increase.
Boots fast and nice.
Created by: NuLogicSystems
udeved, why are you changing manjaro-tools this late in the game?
We can not use your switch unless it keeps the mhwd layer, calamares and thus both require the mhwd repository for offline installs. Also, this seems like a waste of your time and resources since we already have a working fix in place that required zero rewrite of manjaro-tools code.
Created by: udeved
Huh?
It is
a) a separate branch
b) nothing changes except you get an additional layer with preinstalled drivers
c) it has the same effect as adding the drivers to Packages-Livecd, no switches, only triggered if Packages-Mhwd-drivers is present
So whats the problem?
Created by: NuLogicSystems
There's not really a problem, but why wast time on something that doesn't need another layer to accomplish?
Created by: udeved
Because its much easier to test out this stuff if it is in a separate file and layer, instead of adding and removing drivers from livercd list for each build.
If somebody wants preinstalled drivers, a simple Packages-Mhwd-drivers can be symlinked or used as file.
Created by: NuLogicSystems
You don't, you just need to either softlink to Packages-Livecd or Packages-Livecd-xorg no need to add and remove anything from the lists. The only other change I made for my ISO was to remarked out the nvidia proprietary packages in Packages-Livecd-Xorg
Created by: NuLogicSystems
I'm not trying to argue with you. If Philip and the rest of the maintainers think what your changing is OK then so do I.
Created by: udeved
Yeah, you could drop your xorg specific livecd list this way.
I mean, whats the deal, absolutely nothing changes.
There is more than one method to achieve the end result you want.
The optional driver layer would exactly contain these packages minus livecd.
Created by: NuLogicSystems
Right and if the way we are doing it now works then why waste effort, that is unless the new way is somehow more optimal.
Created by: udeved
Why not? If its wasted is kind of subjective.
Nobody pays me, I do it for fun doing it, imagine that.
I could build your profile this way, and would have netrunnner additions in a separate layer.
Created by: philmmanjaro
Well, remember that we need a different set of packages since proprietrary drivers will conflict with each other. This means, if there are the same packages as with mhwd repository image then it won't work. As of now I didn't had time to check your branch yet. I'm open for the change, but it should work and make somehow some sense way we put extra time into it, as the current set already works.
Created by: udeved
There aren't, its Packages-Mhwd-drivers.
Created by: NuLogicSystems
Does this keep the mhwd-livecd repository layer required for offline installs or doesn't it?
Does this only install the free drivers to a separate layer?
Installing proprietary drivers creates an issue if older cards need legacy drivers as now it needs to remove drivers to install drivers or conflicts will occur, so more time added to boot ISO when choosing non-free option. I had already closed this topic since it was fixed, do I need to re-open it now?
Created by: udeved
Yes, it does keep the repo, I said it already twice.
It installs whatever you put in the Packages-Mhwd-drivers.
Why would you reopen? Its in a branch, that is not about to be merged the next days I think.
Created by: NuLogicSystems
Yes, but you also made changes to the master branch of manjaro-tools-iso-profiles as well, this too should have been in a separate branch I think.
Created by: NuLogicSystems
Also why not install these drivers as well as add the repository in the same mhwd-live layer or in the livecd layer like we do now only adding a switch to turn it on or off instead of adding yet another layer?
Why make it more complex than it needs to be?
Created by: udeved
Its only a matter of defining Packages-Mhwd-drivers, no separate branch needed for profiles.
I was told using the same mhwd package list is not good.
Its trivial to change the install location, but I don't see any disadvantage using additional layers.
My test iso doesn't boot slower with additional layer.
Created by: NuLogicSystems
OK, so how many layers or overlay file-systems are we up to already? How many is too many? What is the norm?
Created by: udeved
I personally don't care for any norm, and afaik there is no norm with overlayfs on iso.
We use rootfs and desktopfs, which amounts to one layer that is installed.
All the rest are live layers.
Created by: NuLogicSystems
I'm just wondering, I'm not against this BTW, I just want to be sure this is not wasted effort. What Phillip provided does the job well enough for me.
Created by: udeved
As said, I don't see it as wasted, its my time. :)
I am evaluating if I will use an additional layer for m lxqt-minimal profile.
mhwd-scripst runs fast with preinstalled drivers.
Created by: philmmanjaro
At iso-profiles we just have an additional file. So no branch is needed. For now all is fine as both ways will do the same thing in the end. @udeved just added another layer. It's a blue or red question like in the Matrix. Let him test it. However, I see this earliest to been merged, if ever, within 0.9.14 or later ...
Created by: udeved
Its simply a matter of being able to enable additional layer by dropping a file of symlink in profile.
No file, no layer additionally, but its clean instead of already growing number of package lists in shared again.
Its the same mechanism we use for all other layers too.
Created by: NuLogicSystems
OK, for now I'll just ignore these changes until I need to know how to use them. ;)
Created by: udeved
Whatever, since free drivers are only 10MB increase, I think I will go for preinstalled free drivers and prop only in mhwd repo. I'll see how that goes.
Created by: NuLogicSystems
That's basically what I did on my ISO as well with my Packages-Mhwd-Xorg file.
Created by: udeved
I just added instructions for calamares to unpack an optional drivers image
Created by: NuLogicSystems
So now there is two repositories images of packages or am I missing something?
Created by: udeved
Nope, I just tell calamares to also unpack the optional drivers image with the free drivers, since the mhwd repo only contains prop. drivers, so that I don't get missing drivers source for mhwd in any case.
Ie the mdhwd drivers image is installed too. Its experimental, haven't done a test build and install yet.
Created by: NuLogicSystems
OK but before all the drivers were in the one layer, you separated them then?
And if so, where are the network, virtual box, vmware and bumblebee packages now?
Created by: udeved
I haven't done any package list work yet, I would do it in my profile anyway first.
This optional layer stuff is not finished yet.
Created by: udeved
The idea is to get perhaps around shared/Packages-Desktop in favor of an additional layer with xorg all profiles need.
Created by: udeved
My thinking behind this, unpacking the sqfs file depends on cpu abd ram. Most systems handle it sufficiently.
However, if I try to split for example the quite big desktop image in a xorg and a desktop sqfs, it may be faster to unpack. I think the mounting of the layers isn't the issue, and also squashing the images should improve on a weaker cpu.
Created by: NuLogicSystems
Are you suggesting that what your trying to do is make the ISO boot faster using less resources than what we have now?
If so, than I say that sounds great.
Created by: philmmanjaro
What ever the outcome here will be, it has to been tested, discussed by the team and agreed that we will go this way. Also it should not conflict with the current status. So please test it thru and thru and create a merge-request later one, so we all can discuss and test it. Remember, if you change your profile and somebody else tries to build it with the old way it might be broken. Maybe a branch for iso-profiles is therefor also needed ...
Created by: udeved
A testing profile(actually a copy of my profile) is fully sufficient to juggle with xorg packages in a new list.
Created by: philmmanjaro
Ah, ok. Keep me updated on your findings ;)
Created by: udeved
You think its doable? Ie having only prop. drivers in repo, and free drivers preinstalled as well as installed on target system? atm, half the free drivers are already installed, such as the xorg-input drivers.
I personally only use xorg-input-evdev and always remove keyboard and mouse xorg driver.
Created by: philmmanjaro
Only install the graphical drivers on a live-session overlay, not on images getting unpacked to target. Else we would have installed drivers the user doesn't need at all. Free drivers don't conflict but I don't want to have them pre-installed on the target system. mhwd should detect and only install the needed. So mhwd-repository still need to hold all drivers, doesn't matter if free or non-free.
Created by: udeved
Either that, or optionally install free drivers on target.
If I create a repo with non free drivers only which depends on mhwd list, I need the free drivers installed so that I don't get error if not online. Right?
Its a trade off to consider, +10MB, but faster boot and non free driver blacklists free driver, at least nvidia does.
Created by: philmmanjaro
We have all needed drivers in the mhwd-repository as we use that repo instead the one we have on our servers. So mhwd basically does always an offline installation. All drivers are needed to be in that repository. Splitting it is not possible.
Created by: philmmanjaro
It doesn't matter in which overlay you put the pre-installed packages to have a faster boot, but the packages for the mhwd-repo should not been changed at all.