[buildiso] support pxe booting #84
Labels
No labels
Done
bug
buildiso
buildpkg
buildtree
critical
deployiso
duplicate
enhancement
help wanted
in progress
invalid
manjaro-chroot
note
old-not-relevant
optional
question
sonar
todo
wontfix
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
tools/manjaro-tools#84
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Created by: Kirek
From https://forum.manjaro.org/index.php?topic=22137.0 by patpat
Hi guys,
this post try to explain why Manjaro cannot be net booted from a PXE server.
Hopefully Manjaro developers will make the small necessary changes and soon
we'll get PXE bootable Manjaro ISOs.
Manjaro (i.e. a 64 bit flavor) initially boots a small Linux executive made of
kernel: /manjaro/boot/x86_64/manjaroiso
initrd: /manjaro/boot/x86_64/manjaro.img
In a regular boot the executive initializes everything and mounts from the CD the squashed file system
/manjaro/x86_64/*.sqf
next the initial executive "chroots" to the newly mounted filesystem and "voila"
In a net boot scenario (PXE) the initial executive cannot directly mount the squashed files
without net retrieving them first. Then it is clear for PXE boot the initial executive must include
net support (net drivers) and a few net utilities like ipconfig and wget or curl. Fortunately
including this functionality adds only ~5 MB to /manjaro/boot/x86_64/manjaro.img
Comparatively Arch Linux does include net support in its /arch/boot/x86_64/archiso.img
and it can be PXE booted as shown here:
http://vercot.com/~serva/an/NonWindowsPXE3.html#linux
Unfortunately Manjaro has removed the net drivers/support from its
/manjaro/boot/x86_64/manjaro.img
Steps for making Manjaro's ISO PXE bootable
I'm one of Serva developers I have good experience making Linux distributions PXE compatible.
Let me know if you guys need help fixing this issue.
Best,
Patrick
Created by: ppatpat
OK the first step is making a manjaro.img with net support.
I have seen the manjaro-tools at https://github.com/manjaro/manjaro-tools
/lib/util-iso-boot.sh
$1: work_dir
gen_boot_image(){
local _kernver=$(cat $1/usr/lib/modules/*/version)
chroot-run $1
/usr/bin/mkinitcpio -k ${_kernver}
-c /etc/mkinitcpio-${iso_name}.conf
-g /boot/${iso_name}.img
}
the former function chroots to the working dir and creates manjaro.img using mkinitcpio ( https://projects.archlinux.org/mkinitcpio.git/tree/man/mkinitcpio.8.txt ) based on mkinitcpio-manjaro.conf.
Then it looks like either the system where the manjaro.img is created does not include the net drivers (then these drivers never get added to the manjaro.img) or the mkinitcpio conf/preset files do not explicitly add the net kernel modules
Created by: udeved
So this is first to be solved by adding a patched ipconfig version to the repos?
Created by: ppatpat
Adding a patched version of ipconfig to Manjaro's repository is just one of the required steps described on the first post.
If you are able to patch ipconfig that would definitely help; the patch is very simple (2 lines) see here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=756633
https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1327412
At the moment what I'd really need is some Manjaro maintainer willing to assemble a /manjaro/boot/x86_64/manjaro.img identical to the one on Manjaro's ISO but including net drivers/support (like the one Arch uses). With that image I'd be able to patch and test Manjaro hooks and get Manjaro PXE booted. When we get everything working you guys can add the mods to Manjaro's build script.
If you wonder why I cannot use Arch's img the reason is they use a different kernel version...
let me know.
Best,
Patrick
Created by: udeved
Ok, but I still don't understand how we approach this without the patched ipconfig version?
Or better, what do you mean with net drivers? Where are they coming from? Kernel? ipconfig?
I see no problem to create the boot image, but I didn't get how to include the drivers.
@philmmanjaro , kirek
Do we use a different ipconfig from arch?
Why does it work on arch with likely same ipconfig and not on manjaro?
Created by: ppatpat
--/Ok, but I still don't understand how we approach this without the patched ipconfig version?
Temporarily I can use some already patched (Ubuntu based) ipconfig that works with Arch and Manjaro
--/Or better, what do you mean with net drivers? Where are they coming from? Kernel? ipconfig?
The net drivers must be added when creating /manjaro/boot/x86_64/manjaro.img (this has nothing to do with ipconfig) . AFAIK manjaro.img is created with "mkinitcpio"; its man page or manual should tell you how to add the different kernel modules to the img file.
The final manjaro.img should get at directory
/usr/lib/modules/3.16.7.4-1-MANJARO/kernel/
modules like e100.ko, e1000.ko e1000e.ko, etc, which are network drivers
--/I see no problem to create the boot image, but I didn't get how to include the drivers.
I "think" this depends on how you create mkinitcpio-manjaro.conf
--/Do we use a different ipconfig from arch?
No, you guys don't; at least Arch ipconfig also has the bug I'm mentioning here.
--/Why does it work on arch with likely same ipconfig and not on manjaro?
Because
you can see how to PXE boot Arch here:
http://vercot.com/~serva/an/NonWindowsPXE3.html
Best,
Patrick
Created by: udeved
Thx, my question is answered, so its kernel modules which need to be included in the boot image and available before any udev or dev manager starts. I know how to do this with mkinitcpio.
PS: One could add the kernel modules to the profile's mkinitcpio.conf if I understood it correctly.
This one gets copied for the mkinitcpio boot image.
I added such thing in code, so it only applies for the copied mkinitcpio.conf.
I could make it configurable with manjaro-tools.conf what kernel modules to add and load.
Created by: udeved
@ppatpat
Could you please test, if the boot image is right if you add the kernel modules to your profile's mkinitcpio.conf? Add the modules to "MODULES=" array.
I created an image this way, and it contains the e100/1000(e) modules, which will be explicitly loaded.
Created by: ppatpat
@udeved
I do not have a Manjaro box running at the moment but I can test your manjaro.img, can you upload it somewhere? remember the image must be assembled for kernel version "3.16.7.4-1-MANJARO" that is the kver used on the distribution ISO .
Created by: ppatpat
I saw your patch
[conf] add kernel_modules to profile.conf #84
but you are adding there "only"
kernel_modules="e100 e1000 e1000e"
There are many more kernel modules corresponding to other NIC models.
There must be a way to include "all" the network drivers. I've tried to find Arc's conf file and see how they do it but so far I haven't found it...
Best,
Patrick
Created by: Kirek
Maybe this can help
https://projects.archlinux.org/archboot.git/tree/usr/lib/initcpio/install/arch_net
https://projects.archlinux.org/archboot.git/tree/usr/lib/initcpio/install/arch_wireless
https://projects.archlinux.org/archboot.git/tree/usr/lib/initcpio/install/arch_wireless_staging
Created by: ppatpat
@Kirek
I find discrepancies between
https://projects.archlinux.org/archboot.git/tree/usr/lib/initcpio/install/arch_net
and what I see within archiso.img
I think that's not the script that added network capabilities to archiso.img in archlinux-2015.04.01-dual.iso
Created by: udeved
Network drivers, or kernel modules in general are added by mkinitcpio in the MODULES array of mkinitcpio.conf.
One has to specify modules to be loaded explicitly for initrd
Thats why I added a possible solution only, since its basically useless to introduce a kernel_modules var in profile.conf which is only handed over to the MODULES array. Ie, one can use the mkinitcpio.conf directly to achieve a initrd with exlicitly loaded net modules available instantly when ramdisk is loaded before udev kicks in.
A mkinitcpio.conf for the arch-net would be interesting, however, I think mkinitcpio should autoload all modules with PXE hook or something similar.
Created by: udeved
@Kirek
I am wrong to think that our PXE initcpio stuff lacks a module prober so necessary net modules won't be added? add_checked_modules()
Created by: ppatpat
If that is the case; is the mkinitcpio.conf that created manjaro.img available?
As a reference finding the mkinitcpio.conf used for Arch would provide valuable help.
That's not what Arch does; AFAIK there's not any "hook" loading (insmod, etc) kernel modules.
Created by: udeved
No, I have come to the conclusion after looking at the arch install hooks, that our PXE initcpio install hook lacks a module loader.
My approach yesterday would only be a workaround, we need to fix the initcpio stuff for PXE.
Thats the function we need to use I think:
Created by: ppatpat
if you are talking about the missing hooks corresponding to archiso_pxe_common and archiso_pxe_http in Arch that's a different thing. Those hooks are the ones that will net "retrieve" the squashed images later. For those hooks to work the underlying kernel network engine must be already up and working.
Lets get focused in getting a manjaro.img with network drivers working; we move to fix/add the corresponding hooks later.
Created by: udeved
I talk about the arch-net hook kirek pointed out, and our missing usage of the modules loading function.
We talk different things, you want image, I want to fix image creation.
Created by: ppatpat
ok, we are talking about different things. Just keep trying to get the image done and let me know when you have news.
Best,
Patrick
Created by: udeved
I believe I have fixed to non present drivers.
I noticed the net hook in the initcpio install, which is exactly what I was missing in PXE hook.
So I added the net hook to the iso-profiles, and I will get you an image to test.
Created by: udeved
@ppatpat
http://sourceforge.net/projects/manjaro-lxqt-openrc/files/manjaro-0.8.11/
Please test manjaro-openrc.img
Created by: aadityabagga
Hi,
The files rebuilt with udeved's changes from my side:
https://sourceforge.net/projects/mefiles/files/Manjaro/pxe/
This is the original ISO (without the changes):
http://sourceforge.net/projects/manjaro-openrc/files/upcoming/net/manjaro-net-0.8.13-openrc-rc1-x86_64.iso
Kernel in the ISO:
Created by: udeved
We should think about if we provide a specific pxe net boot image.
The additional modules made it necessary to increase the fat partition size for uefi to work.
Ie, two routines setting up the uefi partition and a separate mkinitcpio.conf
Created by: aadityabagga
I also got the error stated above, I thought I had ran out of space on my root partition.
Created by: udeved
I made the uefi partition size configurable in profile.conf, just needs to be added to the profile.conf in the profiles repo.
Created by: aadityabagga
Adding the net hook with udeved's seems to have increased the size of the ISO by ~40mb..
The files and their changes seem to be as follows:
Edit-
Here is the stuff that seems to have been added:
Created by: udeved
Yep, it is why it eventually made sense to offer a net boot image separately?
Created by: Kirek
To build special ISO for net boot will be a waste of time, just make it bigger and be done with it, 40mb is not that much.
Created by: udeved
Ok, I am fine with it also, since uefi size can be set now.
Created by: ppatpat
Please do not do that. It is just a bad approach.
Comparatively archiso.img (almost PXE ready) is just only 21 MB; we should get the same figures with manjaro
I've tested:
http://sourceforge.net/projects/manjaro-lxqt-openrc/files/manjaro-0.8.11/
it cannot be booted as it was made for kernel version "3.19.6-1-MANJARO"
https://sourceforge.net/projects/mefiles/files/Manjaro/pxe/
it cannot be booted as it was made for kernel version "3.18.12-1-MANJARO"
As I've said before the kernel version must be the one available at the distribution ISO manjaro-xfce-0.8.12-x86_64 (3.16.7.4-1-MANJARO). We should avoid moving targets when testing (i.e. RC, pre-releases, etc.). Remember when testing, the new manjaro.img will be loaded by the kernel available at the distribution ISO, then we should build the testing image based on a system "exactly" as it comes when just installed from Manjaro's current distribution ISO. If a different version is used I'd surely get a kernel panic as soon as the img is loaded.
Created by: philmmanjaro
@ppatpat: please use the net-edition, then it should work:
http://sourceforge.net/projects/manjaro-openrc/files/upcoming/net/manjaro-net-0.8.13-openrc-rc1-x86_64.iso
We didn't do an image based on lxqt yet.
I agree with you, but lxqt is a community edition. For now we should check if the new approach works with the given install media here. If so you can recreate a new image on your own using what ever profile you want.
Also our initramfs images are totally different created than within archiso. We are using aufs and other modules which makes it bigger already.
Created by: philmmanjaro
Quick looking at https://projects.archlinux.org/archiso.git/tree/archiso/initcpio/install/archiso_pxe_common gave me the impression that Arch is loading the needed modules on runtime but not provide them already inside the kernel .img file. Did somebody mounted that file to check what is inside of it? I see no modules in their mkinitcpio.conf.
Created by: udeved
@philmmanjaro
The important part is add_checked_modules().
We don't have a hook in our config which calls this function, hence I added for testing the net hook which calls it.
Created by: ppatpat
@philmmanjaro
I was just testing the images kindly created by udeved and aadityabagga but both were based on different than 3.16.7.4-1-MANJARO kernels. To solve this problem I recommend sticking to manjaro-xfce-0.8.12-x86_64.iso and creating the testing images exclusively from that distro.
No, it won't. Despite its name Manjaro net-edition also has a manjaro.img that paradoxically does not have net support. That means you "cannot" boot Manjaro net-edition from a boot server; you need a CD or Pendrive !
I can hear you asking then, how does "Manjaro net-edition" do when needs to net retrieve components from the net? well "Manjaro net-edition" has net support "only" after mounting its root-image.sqfs and in a PXE environment you first need a working net to retrieve root-image.sqfs !
The goal here is producing Manjaro ISO distributions with a content that can be directly copied to, and booted from a regular PXE server like SUSE, EHEL, FEDORA, UBUNTU, LXLE, MINT, MAGEIA, etc do.
I'm fine with that; I just mentioned Arch as a reference; the point I'm trying to make is that adding net support to manjaro.img should only require a marginal amount of extra room compared to the distribution ISO size.
Sure I did; Arch Linux does include net support in its /arch/boot/x86_64/archiso.img. You can open that file and see modules like e100.ko, e1000.ko e1000e.ko, etc, which are network drivers; Net support is there and Arch can be PXE booted w/o any question.
Created by: ppatpat
@udeved
Yours and aadityabagga's images even when they were created for the wrong kernel seem to have net support; the net drivers are there. I encourage you and aadityabagga to re-create the images on a host based on manjaro-xfce-0.8.12-x86_64.iso. I think we are getting closer.
Created by: udeved
@ppatpat
I don't have much time for this currently.
The kernel you asked for in initial post in not in repos anymore.
Remember, its rolling release, but yes, with netk hook, iso images should be net bootable.
Created by: ppatpat
@udeved
The kernel I'm mentioning is the kernel used in manjaro-xfce-0.8.12-x86_64.iso which is one of the current release distributions of Manjaro.
This kind of fixing/testing cannot be done in the roller-coaster of a rolling release schema; we should all agree on working on a particular release version.
I understand you have no much time for this now then when you get a minute please write here the exact steps for creating the image. Thanks.
Best,
Patrick
Created by: ppatpat
Creating manjaro.img for PXE testing:
https://github.com/manjaro/manjaro-tools/tree/master/initcpio/inst
to:
/usr/lib/initcpio/install
https://github.com/manjaro/manjaro-tools/tree/master/initcpio/hooks
to:
/usr/lib/initcpio/hooks
HOOKS="base udev net miso miso_pxe_nbd miso_loop_mnt miso_kms modconf block pcmcia filesystems keyboard keymap plymouth"
COMPRESSION="xz"
I got an img of 29 MB with net support.
I'll start testing the the rest of things.
Best,
Patrick
Created by: philmmanjaro
@ppatpat: we are rolling release and we are using snapshots. Please use the given images we extra upload for these testing cases. We can't recreate any older images like 0.8.12. That one is gone. Either create a new install media based on our scripts or try out those we provide. So for the last time:
linux318 3.18.12-1:
PXE files
Fitting test image
When my 0.8.13-dev builds support net-boot I'll mention it here.
Created by: ppatpat
@philmmanjaro
It is not necessary to "recreate" anything it's just a matter of agreeing to work based on one particular release, version, snapshot or whatever you want to call it.
I'll keep working based on manjaro-xfce-0.8.12-x86_64.iso
So far the image I have created has network support "working"; now I'm facing fixing the HTTP hooks. If any of you guys want to fix ipconfig it is a 2 lines patch.
Best,
Patrick
Created by: philmmanjaro
@ppatpat: sure. Well we are working on the scripts and generate our images on the fly. It actually doesn't matter which snapshot you use. Our install medias change day by day. They have mostly the same name but get re-uploaded with new packages until we have a working one we release to the wider public.
Sure you can use an older snapshot for your own testings, but don't expect that we will work on those. It is simply like this: We run our scripts, test our changes first locally and then maybe upload them to a server. Most test cases will be done locally on personal PCs. Regarding ipconfig, please send me the patch and I'll modify the ipconfig package in our unstable branch. It will be automatically added to our testbuilds later.
I might generate a new install media later today. It will be based on linux40 4.0.2 as I will do another 0.9.0-dev build.
Created by: ppatpat
@philmmanjaro
Please, it definitely does. I create an image today, that image "must" match the rest of the distro (I need same kernel version) then tomorrow someone changes the kernel version then I need to update my whole sandbox with a new image "and" the whole new matching distro to see if it PXE loads correctly??? That's nuts my friend.
For this kind of issues we should take the latest stable release, the whole group works with the same version on a testing branch and then when the fix is finished we merge the changes back to trunk; that is how it has to be done.
No problem; at least just fix ipconfig ;-) That fix is not related to kernel version then you can easily do it your way w/o caring about kver integrity..
the patch is as simple as
.../ipconfig/dhcp_proto.c
If your build is "stable" I could upgrade for my testings but if it is not, I do not want other unknown issues masking my own problems (that's why on this kind of patching I always take the last stable release as the starting point).
Best,
Patrick
Created by: udeved
@ppatpat
Please understand that we can't do isos with old kernels not in repo anymore.
I definitely will not do such thing and get myself from somewhere an earlier kernel build.
I don't get your troubles with "complete iso release".
What you need for your tests is a mkinitcpio boot image of any kernel.
I don't see how you need an entire iso release to test your case, apart from the ipconfig patch.
Manjaro-tools are currently set for kernel 3.19.x
Created by: ppatpat
we have some misunderstanding here...
I never requested that. I just said let's agree on ONE stable version to work with. That version if we are testing PXE ISO boot it must be an stable ISO.
you need a complete ISO release because the idea of PXE boot is to take one distribution ISO, copy its content to the PXE server, and PXE boot a client; that's it. Then the testing "manjaro.img" MUST match the rest of the ISO if not you get a kernel panic. Trust me; I've done it million times with the top 20 distros from distrowatch. I'm not wrong on this.
but can't you see that the booting kernel MUST match the kernel drivers WITHIN the manjaro.img
and at the same time MUST also match the kernel drivers at root-image.sqfs.??
Imagine I boot kernel version N but majaro.img contains drivers for kernel version N+1 ; this is bad right? now you can say well just boot kernel version N+1 and next you load manjaro.img built for kernel N+1, OK and what do you think is going to happen when down the boot process you chroot to root-image.sqfs which is built based on kernel version N ?? can you see now why I need an entire ISO?
Created by: udeved
Can't you see that you request a kernel version which has already new kernel release builds? Ie, your iso will not work with a newer kernel, no matter you ask for eg 3.16.
The minor vesion numbers are also important for modules to work, and we all use higher kernel versions for iso building, because that will be the next kernel version we will release on iso.
I am willing to help with PXE, but I am not willing to manually build a mkinitcpio image with an old kernel.
We create these images with manjaro-tools during iso build process, so adding a net hook to mkinitcpio.conf seems to solve missing network drivers in the initrd. We can very likely close this issue.
I still don't get why you don't get yourself a newer iso for testing, say one of phil's 0.9.0 builds?
Created by: ppatpat
What? man I need a Manjaro ISO to test and define the changes that have to be made in future releases of Manjaro in order to support PXE boot natively. I can use for that an old version, at the end I'll just produce a set of patches applied to Manjaro building chain and that's it; all the new versions coming out of the kitchen will support PXE. Then no one wants to mix the old version with the new one, the old version only temporarily provides the common scenario for people working on the fix and the stable point needed to build those patches.
as I've said before, give me a new "stable" ISO and I could make my tests with a more updated version, but do you really think that the new version will impact the kind of patching we have to do here? I do not think so.
Please let's move on
Created by: udeved
"Give me" is not the right approach here.
You asked for a boot image you want to replace to make it boot an iso for that kernel version, but you refuse to get yourself a matching iso. Phil offered you one, and he will likely add the net hook in his build.
There is nothing I can do to help you then.
Created by: udeved
I may add, to me, your issue is solved by adding a net hook.
Created by: ppatpat
Please man , "give me" is just an expression; I'm not demanding anything here.
Yesterday I created the image myself based on manjaro-xfce-0.8.12-x86_64.iso and I will continue doing the way I think is the best. Surely I failed making you to understand why; sorry for that.
Please read; I did not refuse, I said I need an stable ISO though.
Yes you can help i.e. taking the time for reading this thread correctly
No it is not; the net hook is just one part of this fix please read; read the first post.
Best,
Patrick
Created by: Kirek
This weekend philm will probably generate a new test iso, with net hook and and patched ipconfig, with that we can continue testing.
Created by: udeved
Let me get this straight in a last attempt of being friendly.
I read your stuff, I know what you try to do, phil offered you an iso to test.
If you exclusively demand a "stable" iso based on a rolling release, then I can't help you until you tested any new iso with net hook set and further inlcuding a patched ipconfig version. Until then, I will not put anymore effort into PXE, until its tested and we get feedback.
Created by: ppatpat
@udeved
man you are not being really friendly; you do things "your way" w/o paying much attention to other people's reasons. I think you do not correctly read my post, so far I really do not know if you understood why I do things the way I do.
You wrong again, you do not read again;
I have already done that, I created a manjaro.img based on manjaro-xfce-0.8.12-x86_64.iso with the net hook, it is up and running with an Ubuntu based patched ipconfig. I'm working on that now...
You ask for feedback? please read the thread I have said "several times" the image I've created is up and running; "the network works". I understand this topic is getting into your nerves probably it would be good for all if you stop helping us until you calm down a bit.
@Kirek
Thanks.
Best,
Patrick
Created by: udeved
@ppatpat
Seriously, change your tone, your entire approach of coming here demanding "give me stable" pisses me off, while neglecting any explanation you got.
I shall stop helping you, so somebody else can try in the next year, whenever.
I don't tolerate people on my project's bugtracker, who insult the developers whoare supposed to help.
"You are nuts" and similar is the stuff you should not say to people you ask for help.
Created by: ppatpat
@udeved
It only takes reading this thread to see that I never demanded anything, I never said "You are nuts", I never insulted anyone.
Somehow I might've hurt your ego, then you tried closing this thread, later you removed the milestone, and now your only goal is me losing my temper and ending up insulting you or thing like that; well that's no going to happen.
I take your offer, I courteously ask you; please stop helping us on this topic.
Best,
Patrick
Created by: udeved
You did not hurt my ego at all, you simply have no manners, and that's the problem, that you come across as somewhat demanding without the willingness to do your part.
I just point out, if you talk about "we", I have no clue who you talk about. I am the person who spends most time on manjaro-tools. Perhaps kirek or phil will pick up, as said, I won't until you change your attitude. Look at the commits to realize who does the main work on the manjaro-tools, and that you are counterproductive to solving your problem, which eventually generates income for you if solved.
Created by: ppatpat
@philmmanjaro & @Kirek
OK it's done;
I just PXE booted manjaro-xfce-0.8.12-x86_64.iso with the custom made manjaro.img, plus patched ipconfig and added curl.
How to add PXE support to Manjaro builds:
this directory
https://github.com/ppatpat/manjaro-tools/tree/master/initcpio/hooks
contains the patched/new hooks that should be installed in the PC that runs mkinitcpio at
/usr/lib/initcpio/hooks
this directory
https://github.com/ppatpat/manjaro-tools/tree/master/initcpio/inst
contains the patched/new hooks that should be installed in the PC that runs mkinitcpio at
/usr/lib/initcpio/install
we need curl then
3.1) the file
/usr/lib/initcpio/install/net
should add the line
3.2) copy /bin/curl to /usr/lib/initcpio/
NOTE_1: I just added the hooks to the forked repository then some change might be still necessary.
NOTE_2: remeber ipconfig has to be patched.
If philmmanjaro will work this WE on a new build probably he could add the described mods.
Best,
Patrick
Created by: philmmanjaro
@ppatpat: maybe we have here a big misunderstanding. I thought you want to have our install medias able to boot via PXE. manjaro-tools should be able to add this feature to our ISO images. These initcpio hooks we have on our install medias will never make it into our installed systems. They are part of our livecd overlay. We don't plan to make a installed Manjaro bootable by PXE. We don't care about install medias we released. These are just snapshots we think they are stable. You can use any given ISO image to install Manjaro as long as we provide them. I'll check your manjaro-tools fork and try to understand what you're trying to do here.
Created by: philmmanjaro
also I don't understand why ipconfig needs to be patched at all ...
Created by: ppatpat
@philmmanjaro
That's right...
not completely; if you need to alter the "net" hook in order to add i.e. "curl" you have to do it somewhere else as the net hook AFAIK is not part of manjaro-tools.
OK then manjaro-tools is not used to build i.e. manjaro-xfce-0.8.12-x86_64.iso? then please send me the initcpio hooks used in your distribution ISOs (i.e. manjaro-xfce-0.8.12-x86_64.iso) and I'll sent them back patched. BTW do you have some sort of tool for building those ISOs?
I really do not know why you are mentioning installed systems. I think this is all about "distribution ISOs"...
I'm not following you; i.e. do you want to make the next version of "manjaro-xfce-0.8.12-x86_64.iso" PXE bootable?
did you read the patch?
ipconfig does not correctly implement the DHCP protocol and some DHCP servers do not honor ipconfig requets.
BEst,
Patrick
Created by: udeved
Yes, manjaro-tools, where an issue has been opened, and you somehow managed to annoy to main dev you asked not to help.
Created by: philmmanjaro
@ppatpat: I see no patch. What I see is a guideline how to change the file. I never saw anybody creating a patch and nobody applying the patch. I merged some stuff via #89, but I'm still confused why ipconfig needs to be patched. Also I've to check what is needed to be changed in our profiles
Created by: philmmanjaro
Why is the net hook needed at all. ArchISO uses followed mkinitcpio.conf:
Created by: ppatpat
you talking about ipconfig right? ok you do not see a patch file; you do not see a "guideline". you see the lines of code that must be replaced. I cannot give you a patch file because I do not have the sources of your package. The code I gave you is taken from the Ubuntu patch.
Sorry; I really do not know what you mean...
I have written before "ipconfig does not correctly implement the DHCP protocol and some DHCP servers do not honor ipconfig requets." ipconfig fails implementing RFC-1531 correctly and some DHCP servers do not answer ipconfig requests That's why we should patch ipconfig
there you should add the net hook to the shared/mkinitcpio.conf or use the recently created shared/mkinitcpio-net.conf (if I'm right the last one has some module missing)
you are right, there are 2 hooks one under "install" and the other one under "hooks". "Probably" we just need the one under install, the one that adds the network binaries and kernel modules to the image but at boot time the net hook is probably not necessary. I have opened it and it does some NFS stuff that we do not need. We just really need the hook at building time adding the required net stuff
Created by: udeved
hook and install hook are both needed.
One is the build instruction for initrd, the other runs stuff at initrd runtime, eg loading modules based on hardware detection before udev if needed.
Created by: philmmanjaro
@ppatpat: the sources are inside of mkinitcpio-nfs-utils. Those lines don't exist there:
static int dhcp_send(struct netdev *dev, struct iovec *vec) { struct bootp_hdr bootp; char dhcp_hostname[SYS_NMLN+2]; int i = 4; memset(&bootp, 0, sizeof(struct bootp_hdr)); bootp.op = BOOTP_REQUEST; bootp.htype = dev->hwtype; bootp.hlen = dev->hwlen; bootp.xid = dev->bootp.xid; bootp.ciaddr = INADDR_ANY; bootp.yiaddr = dev->ip_addr; bootp.giaddr = INADDR_ANY; bootp.secs = htons(time(NULL) - dev->open_time); memcpy(bootp.chaddr, dev->hwaddr, 16); vec[1].iov_base = &bootp; vec[1].iov_len = sizeof(struct bootp_hdr); dprintf("xid %08x secs %d ", bootp.xid, ntohs(bootp.secs)); if (vendor_class_identifier_len > 2) { vec[i].iov_base = vendor_class_identifier; vec[i].iov_len = vendor_class_identifier_len; i++; dprintf("vendor_class_identifier \"%.*s\" ", vendor_class_identifier_len-2, vendor_class_identifier+2); } if (dev->reqhostname[0] != '\0') { int len = strlen(dev->reqhostname); dhcp_hostname[0] = 12; dhcp_hostname[1] = len; memcpy(dhcp_hostname+2, dev->reqhostname, len); vec[i].iov_base = dhcp_hostname; vec[i].iov_len = len+2; i++; printf("hostname %.*s ", len, dhcp_hostname+2); } vec[i].iov_base = dhcp_end; vec[i].iov_len = sizeof(dhcp_end); return packet_send(dev, vec, i + 1); }So please check the sources first before your talk about changing some lines.
Created by: philmmanjaro
Do we need udev/rules.d/81-dhcpcd.rules ?!?
Created by: philmmanjaro
@ppatpat: please test followed image and give me feedback what might still be missing if pxe is not yet booting:
0.8.12.201-2 (May 08 20:37)
download iso-image
install log
Created by: ppatpat
@philmmanjaro
My friend, even when I didn't see manjaro sources, before I talked to you about changing lines I've thoroughly tested manjaro's ipconfig which presented the characteristic buggy behavior. Then please do not think I was making you waste your time
you didn't read correctly; I have said before:
The bug off-course is there; please see the following already patched code:
Created by: ppatpat
No we don't.
Created by: ppatpat
There's something wrong here
manjaro-xfce-0.8.13-dev-x86_64.iso (708MB)
manjaro-xfce-0.8.12-x86_64.iso (1.37GB)
I've opened the new manjaro-xfce-0.8.13-dev-x86_64.iso and it has not net support.
you can also see that on the building log you posted before; there's not net hook.
that tells me the used mkinitcpio.conf did not include "net" as one its listed hooks.
Best,
Patrick
Created by: ppatpat
@philmmanjaro
The created image is sensible to the hook order in mkinitcpio.conf; I'm working on that, please wait before you make the next build.
udeved is wrong "hook and install hook are NOT both needed".
it is possible to have net support w/o getting the hook present in the distribution
just rename the file /usr/lib/initcpio/hooks/net to /usr/lib/initcpio/hooks/__net in the PC where you build the image. this way you include the building hook w/o including the runt time hook.
I'll be back to you after solving the hook order issue.
Created by: philmmanjaro
@ppatpat: So in short we need this patch:
--- dhcp_proto.c.orig 2012-03-25 23:12:36.000000000 +0200 +++ dhcp_proto.c 2015-05-09 09:32:37.434212660 +0200 @@ -176,7 +176,8 @@ static int dhcp_send(struct netdev *dev, bootp.hlen = dev->hwlen; bootp.xid = dev->bootp.xid; bootp.ciaddr = INADDR_ANY; - bootp.yiaddr = dev->ip_addr; + bootp.yiaddr = INADDR_ANY; + bootp.flags = htons(0x800); bootp.giaddr = INADDR_ANY; bootp.secs = htons(time(NULL) - dev->open_time); memcpy(bootp.chaddr, dev->hwaddr, 16);Created by: philmmanjaro
So the only open question is now regarding the net hook. Needed or not ...
Created by: udeved
We need this net hook.
As already pointed out, we need to call add_checked_modules().
How we call it and in what hook does not matter, but we need those kernel modules added, and I would use net heek, since order of hooks matters
As a general rule, hooks come always in pairs, as stated, a build instruction "install" hook, and a run hook.
Created by: ppatpat
@philmmanjaro
Yes, we do.
The net hook is "mandatory".
A hook script has 2 instances represented by 2 script files at the building machine i.e. for the net hook you have
The first one is sourced by the bash shell during mkinitcpio execution at "building time". it has instructions to add net drivers, net utilities, etc
The second one is sourced by the ash shell during init execution at "run-time". In the net case it does some stuff related to NFS that we do not use. Then we can ERASE or rename /usr/lib/initcpio/hooks/net, (we cannot erase /usr/lib/initcpio/install/net !!! ) this way we avoid the inclusion of the unnecessary "run-time" net hook script within manjaro.img.
If you do this the building process will inform you that there was a "missing run-time net script" (we erased it) but no problem the image is created correctly.
The building log will look like:
Finally the mkinitcpio.conf should be:
the miso_pxe_nbd is broken that's why I removed it from the HOOK list
At the moment I'm PXE booting manjaro with the last manjaro.img I've created; it only needs the injection of the patched ipconfing; the rest works perfectly.
I think now you have all the info to create the new ISO, hopefully we'll nail the think w/o much additional effort.
Best,
Patrick
Created by: ppatpat
If you do not want to get the error at build time after removing
/usr/lib/initcpio/hooks/net you can do the following
removing (or commenting out ) the line that says
this way you include net support, you do not include the runtime net hook script and you do not get an error when building the image.
Best,
Patrick
Created by: philmmanjaro
@ppatpat: please test followed image and give me feedback what might still be missing if pxe is not yet booting:
0.8.12.201-2 (May 10 11:23)
download iso-image
install log
Created by: ppatpat
@philmmanjaro
The net services are correctly injected now !
The net hook scrip is still present within manjaro.img this is really unnecessary and can be easilly avoided by editing at the building PC the file
and removing (or commenting out ) the line that says "add_runscript"
you should add the line
then at the building machine copy the curl as follow
Now you can rebuild the ISO.
Best,
Patrick
Created by: philmmanjaro
Ok, I added curl but I still don't get why we should drop the run-script ...
Created by: philmmanjaro
I now also updated the hook order ...
Created by: philmmanjaro
@ppatpat: please test followed image and give me feedback what might still be missing if pxe is not yet booting:
0.8.12.201-2 (May 10 16:36)
download iso-image
install log
Created by: ppatpat
I see the manjaro.img /config shows
HOOKS="udev net plymouth miso miso_pxe_common miso_pxe_http miso_loop_mnt keymap"
please make mkinitcpio.conf look like:
you are adding the wrong order here
github.com/manjaro/manjaro-tools-iso-profiles@3e89d3563eThere's something weird here because the building Log shows the correct order, then please check
the link of the ISO youu just gave me really correspond to the image with the included mods.
you are doing something different here:
github.com/manjaro/packages-core@51f4fe5b15you can easily check by yourself some characteristics of the ISO; Just open the ISO with 7z and first check that /bin/curl is there,
next you can navigate and open the contained /manjaro/boot/x86_64/manjaro.img;
within this image (you have to click a couple of times until 7z opens the cpio structure) then check that the file /config contains the correct hook order (do not worry is there's some module listed in mkinitcpio.conf that is not included in /config; that's normal)
Best,
Patrick
Created by: ppatpat
because it makes no sense to include a run-time script for an NFS based boot that is as broken as the pxe_nbd script.
Created by: philmmanjaro
@ppatpat: did you actually downloaded the new image or not. If you check my post the order of hooks is correct. See my logs. I'm now checking if curl is there, normally it should.
Created by: philmmanjaro
@ppatpat: please check your ISO and check also the given checksums. I don't see any error here. Please redownload and check the sums:
Created by: philmmanjaro
regarding run-script: I think we should blank it then. Tell me where it is located at ...
Created by: ppatpat
@philmmanjaro
sure I did
I saw that; that's why I started wondering if the link was pointing to your last version or not..
I re-downloaded and the md5 matched (unfortunately I've overwritten the previous version)
Well, congratulations ! you just cooked the first Manjaro distro able to be net booted (PXE) without a single patch. Good job !
Notes:
1)
manjaro-xfce-0.8.12-dev-x86_64.iso had the kernel at
/manjaro/boot/x86_64/manjaroiso
now
manjaro-xfce-0.8.13-dev-x86_64.iso changed its name
/manjaro/boot/x86_64/manjaro
Is this an intended change? if it is why don't you guys call the kernel "vmlinuz" like most of the distros?
and comment out the line that says "add_runscript"
For the ones that want to try the PXE boot capabilities you guys can do it with Serva using the following ServaAsset.inf (or just using any other PXE server editing correspondingly the following "kernel" and "append" variables.
ServaAsset.inf
Best,
Patrick
Created by: udeved
We had changed the mechanics how the entire boot loaders and their configs get on iso, and have set new defaults for some variables dealing with iso label, branding and bootloader stuff.
manjaroiso target sounds like leftover from old manjaroiso script.
Created by: philmmanjaro
@ppatpat: I can't just remove the run-script. It will fail and stop the creation of the image as manjaro-tools is more sensible to error messages. So if any script is broken we have to fix it or make it just echo something as workaround. As the run-script should be part of mkinitcpio-nfs-utils we need to edit it as needed. So please take another look also on our hooks and tell me what is still needed to close this one here ...
Please also keep attention at our build-log. It will provide you the information like disk label and such. Your script is still using an outdated one ...
Also it would be great if you start learning the basics like creating patches or be more specific. I'm for more than 10 years now working with a lot of projects and still see some beginner mistakes in your work. We need to many takes to get to the goal here. Regarding ndb: Seems during portation from manjaroiso this hook got broken and we never adopted to archiso changes. I'll simply will drop it now ...
Created by: ppatpat
@udeved
ok here is the point to consider; as you see on my previous post PXE servers rely on scripts that point to the ISO distribution booting kernel and initrd image. Clever distributions keep these filenames and their paths "as stable as possible" in order to avoid being forced to change the PXE booting scripts with every new release or among different flavors and/or architectures of a particular distro.
@philmmanjaro
mmhh you will not get any building error when running mkinitcpio (isn't that enough?). The procedure I gave you before instructs the "building time" net script not to include the corresponding "run time" net script in manjaro.iso but there's not any error at building time.
Good catch, I'll try to see if I can eliminate that variable.
Really?? you want me to give you a patch file or being "more specific" for a "2 lines /20 chars" fix ?
I apologize, I'm just a beginner, sorry for making you waste your time.
I apologize again. next time you lead and I'll make the ISOs
It is not that, I'm tempted to explain you what's really going on with the broken nbd scrip but I'm afraid my beginner mistakes might trigger your discomfort again..
Best,
Patrick
Created by: philmmanjaro
@ppatpat: I just took a look at archiso and their scripts. Now I know why Arch doesn't need the net hook at all. Did you ever took a look at those?
Created by: ppatpat
@philmmanjaro
you are loosely talking about the "net hook" when at the moment you should know the net hook has 2 components; "building time" and "run time" script, then: what do you mean by "net hook" ?
I really do not know how Arch particularly gets the net support into its archiso.img
I'm not an Arch maintainer. it was udeved who realized that adding the net hook in manjaro the net supports makes it to the image. After considering the extra room was no much we decided to go with that approach.
Now; what are you doubts? please be sharp, concise and specific.
Created by: philmmanjaro
@ppatpat: I'm actually talking about your superb work. I really like it. If you would have more quality this thread wouldn't be so long my friend. I fixed your pxe hooks so the net hook, which is outdated and dropped by Arch long time ago, is not needed anymore. I'm currently uploading a new build you can test. It is 13 MB smaller in manjaro.img:
0.8.12.201-2 (May 11 00:30)
download iso-image
install log
It gives you a smaller image with basic network support and whould have been much easier if you actually would have paid more attention to upstream code. So if you would be so kind and test the current image the last time, we might close this long thread for good.
I thank you for making my life so easy and pleasant ...
Created by: philmmanjaro
And before somebody asks: Please check also the checksum ...
Created by: ppatpat
@philmmanjaro
thanks.
"if I would've had more quality" you say... well I'm going to take into account you're a young ESL and I will not consider your words as an offense. Also I'm not going to consider the number of times I was forced to repeat myself explaining to you guys what I consider simple stuff.
OK lets talk properly; you did not fix my hooks; you now use the initcpio/inst/miso_pxe_common to add net support which is just an alternative approach to the one used so far.
Congratulations again ! you nailed it !;
this image also works.
Again I'm going to consider you are an ESL. Let me tell you I consider myself pretty knowledgeable with networking/PXE/Linux stuff ammong other things but I'm not a Manjaro mantainer. if you read my first post you'll see I was offering PXE knowledge to Manjaro mantainers. Please do not blame me for not knowing Manjaro's code; that my friend IS YOUR JOB not mine ;-)
Best,
Patrick
Created by: philmmanjaro
@ppatpat: all right my friend. Thx for your support and given us PXE. We can now finally close this. All the best to you too. Your work is really great !!!
Created by: ppatpat
you're welcome;
Manjaro's community will surely be thankful for your effort (and udeved's) adding PXE support.
Good luck guys ;-)
Best,
Patrick
PS: Please let me know when this mod makes it to the official release distributions then we'll add Manjaro's PXE boot script (ServaAsset.inf) in our Serva page:
http://vercot.com/~serva/an/NonWindowsPXE3.html
Created by: skimj
I was unable to PXE boot the "direct" way like this:
KERNEL http://<IP address>/manjaro/boot/x86_64/manjaroINITRD http://<IP address>/manjaro/boot/intel_ucode.img,http://<IP address>/manjaro/boot/x86_64/manjaro.imgAPPEND ip=:::::eth0:dhcp archiso_http_srv=http://<IP address>/ archiso_basedir=x86_64SYSAPPEND 3The kernel and initrd images load but then I get these errors:
Looking for boot deviceERROR: Couldn't mount /dev/sr0ERROR: Couldn't mount /dev/sd[a-z][0-9]*My real goal is to PXE boot the net-edition. It seems that the work above was applied to the XFCE edition. I tried both with the same result.
To check my method, I was able to boot Arch this way.
As a work around, I was able to boot the .iso image with memdisk, thanks to this: #177
Created by: ppatpat
Manjaro PXE is broken "again"...
Your work around is not good; memdisk is not a good solution, it does not support UEFI, it is slow, it does not always work, etc etc
It would be a good idea really fixing Manjaro PXE boot.
Best,
Patrick
Created by: philmmanjaro
What is the current working approach then? Merge requests are welcome.
Created by: udeved
Isn't this just an issue of appending the IP to the kernel args of uefi or syslinux?
Created by: ppatpat
No it's not. I'll try to take a deeper look later.
Best,
Patrick
Created by: he-loop
I successfully booted manjaro-xfce-16.06-x86_64 via PXE with these settings :
KERNEL manjaro/boot/x86_64/manjaro
APPEND initrd=manjaro/boot/x86_64/manjaro.img archiso_http_srv=http://x.x.x.x/ archiso_basedir=manjaro/x86_64
Both kernel and initrd from the iso.
Thank you for your work,
Alan