api mode #105
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Created by: udeved
@fhdk
Can you make it so that in api mode setbranch auto sync can be deactivated so it only set branch in conf and mirrorlist? Perhaps additional arg -N?
Created by: fhdk
4.2.0-dev
man pacman-mirrors4.1.4
Created by: fhdk
@philmmanjaro
Have you had the time to check on pacman-mirrors-dev?
I think it is ready for release.
The lastest commits is only finetuning of the new man page
Created by: udeved
So -y also works in api mode?
My last test to use it in chroot-run code produced a sync by default.
Created by: fhdk
@philmmanjaro
I will check on what udeved has in mind and if possible adopt it into 4.2.0-dev
Created by: fhdk
@udeved
-y and -n are mutually exclusive and not limited to api-mode. The reasoning behind is that if you don't want to generate a mirrorlist then syncing pacman databases makes no sense.
The --no-mirrorlist argument causes pacman-mirrors to exit after downloading a set of files mirrors.json and status.json
The --set-branch command does not touch the mirrorlist - only conf. To have the --set-branch reflected in the mirrorlist - the mirrorlist must be generarated with eg. -g, -f1
You can think of the api-mode as an addon - the api-commands are executed before anything else but only one makes pacman-mirrors exit and that is --get-branch.
Can you give me samples on you are combining the arguments?
And maybe a liner on what you want to achieve?
Created by: fhdk
@philmmanjaro
I have just stumpled on a regression or maybe something I have not tested - there is a bug in 4.2.0-dev
Created by: udeved
@fhdk
I would want to replace these two shell functions with the more native pacman-mirrors calls in api mode.
I haven't committed the code changes yet, I just tested.
https://github.com/manjaro/manjaro-tools/blob/devel/lib/util.sh#L143
https://github.com/manjaro/manjaro-tools/blob/devel/lib/util.sh#L149
https://github.com/manjaro/manjaro-tools/blob/devel/bin/chroot-run.in#L100
The mechanism is without pacman-mirrors calls, I got an empty mountpoint, the pacman-mirrors.conf gets copied to mount point, along with basic folders created to run pacman. The mirror is in build cases always statically set to netzspielplatz usually.
The base group for example gets pulled in chroot, after the branch is set in mirrorlist. The "problem" is setting the branch in mirrorlist, not the variable in pacman-mirrors.conf.
If it must sync with --setbranch, then all build related calls would sync twice repo dbs, one by pacman-mirrors the other by chrooted pacman calls to pull in packages.
The same mechanism is used by calamares modules.
I could change the chrooted pacman calls to use only
pacman -S $pkglistinstead ofpacman -Sy $pkglistto work around that, but it would be very unsafe, and the sync would rely on pacman-mirrors.Ideally, pacman-mirrors could set branch variable and adapt branch in mirrorlist without sync too.
Created by: fhdk
Commit
github.com/manjaro/pacman-mirrors@a655719a77I have made some changes to achieve what you describe - so a command like this
Explained
The api commands are prioritized
All commands hereafter will reflect the prefix supplied or defaults if not
Created by: fhdk
@philmmanjaro
When you have some time to spare I would appreciate a review to spot if I missed something.
Created by: udeved
@fhdk
Is the commit you posted packaged yet?
Created by: fhdk
No it is not Current dev version is from the 18.
Created by: udeved
Ok, I'll rebuild the package, I need it in repos to test in chroot.
Created by: philmmanjaro
I've updated the package to the current changes made.
Created by: udeved
@fhdk
How do I stop it from generating the mirrorlist?
Would be totally sufficient if the code stops here optionally:
.: Info Branch in config is changedPerhaps, if a mirrorlist exists, eg static list with one build mirror, then execute Rebranch
Created by: udeved
Or, if I was able to supply additionally a specific mirror, I could use fasttrack -f 1 and I would not even have to write it to the empty folder for the mkchroot.
Created by: fhdk
If you actually looked at the sample a couple of post up 😸
You forgot -n argument which will exit with no mirrorlist
Created by: udeved
It rebranches, yes?
Created by: fhdk
Among other things the pacman-mirrors installation creates the path
as a part of the installation process.
snip from setup.py in the pacman-mirrors project
Created by: fhdk
At least it should - Your output is wrong - you're missing the info
Created by: fhdk
I need to revisit the code hang on
Created by: udeved
Thing is, I was playing with the idea to only use pacman-mirrors.
Currently, in order to set up a chroot, it will copy a default pacman-mirrors.conf from manjaro-tools data dir to an emtpty chroot destination, and sed the branch to the desired branch.
Once the pacman-mirrors.conf is present in empty chroot, a mirrorlist is written, with only one build mirror and its corresponding branch.
After this was done, packages can be installed in the empty chroot with pacman.
Created by: fhdk
I am deeply sorry - somehow the actual rebranch-call has slipped without me noticing.
I have fixed it with commit
github.com/manjaro/pacman-mirrors@47d979df0bCreated by: udeved
Would it be possible in api mode to have an additional mirror arg?
Ie I can supply pacman-mirrors with a specific mirror and -nR still works?
Created by: fhdk
It is - the cogs are turning - but I am done for today - it was phone constant telling me mails popped in an I thought it was you 🐎
I will work out a -M --mirror arg tomorrow.
Created by: udeved
Sounds nice, greatly appreciated, because it provides indeed an api that solves a task you have to code in every little project that involves creating chroots.
Created by: udeved
Put it on halt @fhdk for some days , I am currently making a decision if I want to further contribute to manjaro.
Created by: fhdk
That was an overnight change 😞
So will others benefit from the changes made
Configuration flow of pacman-mirrors
At launch an internal default configuration is setup, file configuration is applied and then the commandline is parsed and applied.
API arguments
These arguments modifies key elements of pacman-mirrors configuration according to the packagers needs.
The actions performed by the API are in strict order and performed before any other actions.
If --prefix
If --set-branch
If --url $URL must end with
/If --re-branch
If --get-branch
When done pacman-mirrors checks the internet connection and download the latest datafiles for creating the mirrorlist. At this point it is possible to interupt further processing.
If the -n / no-mirrorlist argument is present pacman-mirrors will now exit.
Created by: philmmanjaro
Current code changes are released as 4.2.0dev.20170601-1
Created by: fhdk
@philmmanjaro
Sorry - and thanks - just that I was disturbed and had not committed the changes
Created by: udeved
@fhdk
Its definitely useful for anyone who want to create a chroot, its not lost effort.
Anyway, if my contributions are not appreciated, and only tolerated, kind of evil step child, I think I need to reconsider if I want to work on manjaro further.
Created by: fhdk
Don't sell yourself short - I think you're more appreciated than you think - mankind simply is not in a habit of expressing appreciation - generally speaking of course.
As an example - your input has been highly valued in the evolution of pacman-mirrors.
Created by: udeved
This is better discussed non publicly, but I don't have any problem with you.
You did a fine job in my view with pacman-mirrors. ;)
Created by: fhdk
pacman-mirrors does as expected
Kind of misleading
.: Info The mirrors has not changedsince in fact it hasWith this addition you can skip the -R and have the same result
Created by: fhdk
@philmmanjaro
I would like to request a package on latest commit
Created by: philmmanjaro
Latest code released as 4.2.0dev.20170601-2.
Created by: udeved
@fhdk
Little problem... but almost there with relying completely on pacman-mirrors. Very nice so far.
Could the json also be downloaded or skipped? I start with no packages installed for chroot, ie no pacman-mirrors package installed yet. I think skipping it would be best, because if pacman installs afterwards pacman-mirrors package, it will throw a filesystem conflict, since mirrors.json already exists in chroot.
The -U option doesn't seem to sanitize path, I need to add a slash(/) to the url?
Created by: udeved
If I download the json and delete it in chroot after pacman-mirrors ran, it looks like this:
But, this doesn't work for buildiso.
Created by: fhdk
I can see it. Let me look at it tomorrow. we will find a solution.
What is notvworking for build iso?
Created by: udeved
Its ok for buildiso too, best way would be to skip the json thingy on -U switch
Created by: fhdk
I have made some changes to the code and I will describe them:
As described above the supplied branch has been applied to the internal configuration before applying the url argument.
So when using -U the
-Ris obsolete since -U will apply $BRANCH from the config and then $repo/$arch and with the current refactor pacman-mirrors will exit when -U is used so the-nargument is obsolete too.Your args will then be
The changes as been commited
github.com/manjaro/pacman-mirrors@08b1b6bd58but I need some mentoring using boxit to upload the changes.Created by: udeved
A pretty cool addition would be similar to pacman a -C $config option, where pacman-mirrors.conf can be located any place to avoid copying the file to empty root.
Created by: fhdk
I don't understand - how would that be of use?
At the point where you execute pacman-mirrors outside chroot - supplying --api and a prefix, a branch and a mirror - pacman-mirrors don't care about pacman-mirrors.conf
github.com/manjaro/pacman-mirrors@0fce51cd8cAll you have to do next is exec pacman in your chroot. You don't have to preload anything onto the chroot just
which - on an empty chroot - will result in
with content
ping @udeved
Created by: fhdk
I found that the branch thingy is not working correct. The -S does not use the correct branch - I am looking into it right now
Created by: fhdk
Correction - it does - I have made another minor adjustment.
When using -S in combination with -U, branch is only used to create the mirrorlist and is not written to config - thus eliminating the need for chroot/etc/pacman-mirrors.conf.
Created by: fhdk
And I have also uploaded an updated pacman-mirrors-dev package 🎉
creates a file
with content
Created by: udeved
@fhdk
The key is, pacman is not executed in the chroot, because it don't exist yet there.
Pacman is executed on the host system, and it has --root set, and --config, ie pacman.conf.
Being able to specify a pacman-mirrors.conf would make copying pacman-mirrors.conf to empty chroot obsolete. Afaik, pacman-mirrors.conf is only eeded to set the branch in chroot persistently, and pacman-mirrors package would not install default pacman-mirrors.conf.
man pacman--config <file> Specify an alternate configuration file.I'll test the changes. Another issue to solve is that upon installing pacman-mirrors, the package generates a mirrorlist, which is unwanted in chroot, as it overwrite the set build mirror.
Created by: fhdk
Sorry - I must be retarded - I still don't understand.
Why the need for telling pacman-mirrors where to find its conf file when your need is to create a mirrorlist with a specific branch and a specfic mirror in an empty chroot.
That is exactly what pm can do now.
In this specific api case pacman-mirrors does not care about the pacman-mirrors.conf and I can't see what info from pacman-mirrors.conf could possibly be of any value in that regard.
When it comes to - when pacman-mirrors has been installed - you can still use a 'remote' pacman-mirrors to modify the installed defaults by skipping -U argument and adding --no-mirrorlist
Created by: udeved
It didn't work yesterday, I still needed pacman-mirroors.conf in chroot to write mirrorlist with the buildmirror but I haven't tried latest changes yet. Looks like it may run now without a pm-conf present.
The new feature just make it new approach to creating the chroot, its kind of fine tuning now, but I like it, since this will come handy in calamares chrootcfg module too.
Created by: fhdk
I have made some changes since yesterday - fine tuning pacman-mirrors
Created by: fhdk
try using the latest dev - and check the man page or the docs/index.md page in the repo
Created by: udeved
The api stuff replaces spaghetti code in couple of scripts. :)
I just need to make sure, the chroot also has the pm.conf set to the branch, after pm package was installed. But this should not be a problem now with api mode.
Created by: udeved
We can now detect, get and set branch at package level too, so we coould generate a proper mirrorlist for the right branch the system is set. Previously, we either deactivated it, or it wrote default stable branch.
Created by: udeved
@fhdk
Do you think we probably try to trigger the mirrorlist update at package level with a json update trigger?
Created by: fhdk
your primary concerns are:
creating a mirrorlist on an empty chroot - Check
changing the default branch - Check
getting the branch
Created by: udeved
The are package level concerns, how we handle mirrorlist on install or update. This needs fine tuning. ;)
Created by: fhdk
Which file are you referring to
Created by: udeved
/usr/share/pacman-mirrors/mirrors.json
This one.
We could trigger mirrorlist generation if this file gets updated.
File based alpm hook.
Created by: udeved
I need to prevent the package from generating a mirrorlist in chroot, which it does currently, and it overwrites the custom mirrorlist with the buildmirror.
Created by: fhdk
pm must not generate a new mirrorlist then use -n
this will download a new mirror file but not create a new mirror list
Created by: udeved
Yes, I mean our dev package, not the code itself.
We need to fine tune the package with the new api options available to make it work in all cases.
Created by: fhdk
you will you pacman-mirrors-dev package and then in the dev package it will never generate a mirrorlist and overwrite the existing.
Created by: fhdk
So we should have in unstable a pm-pkg package solely for using in building iso's?
Created by: udeved
No, its not really realted specifically to buildiso. Its more general to properly address the shortcomings we had before api mode with the package.
We start out with no mirrorlist, and thats kind of problem, because our package needs to write one on first install.
Created by: fhdk
If the buildiso thingy creates the mirror list in the above described manner
The mirrorlist is now existing.
Subsequently the buildiso installs pacman-mirrors but since the mirrorlist is now existing there is no need to have pacman-mirrors generating the mirrorlist at first run.
A mirror file will be delivered with the package and installed in
/usr/share/pacman-mirrors.That mirror file (mirrors.json) will at some point be outdated but calling
upon succesful installation will fetch the latest available mirrors.json file from Github and the latest available status.json from repo.manjaro.org.
I have just pushed a new pacman-mirrors-dev pkg with some modified hooks and scripts which only calls pacman-mirrors with -n which will download up-to-date data files but not touch the mirror list.
Created by: fhdk
Actually I find that this approach should be expanded to the the regular install as well - so an update to pacman-mirrors does not trigger a regeneration of the users mirror list since that regeneration is the mother of all kinds of possible disasters regarding the users updates.
Created by: udeved
Yes, our dev package is just test bed, I started with hooks there.
I had to alter pkglists for build stuff to pull in dev package. Hence I'd love to see better branching with packages.
Created by: udeved
@fhdk
Cann you push the pkgbuild changes?
Created by: fhdk
Yes - I have been granted boxit access
Created by: fhdk
I cant change the github repo
Created by: fhdk
And likewise Jonathon and Matti
Created by: udeved
I see, too bad, can youpost your changes here?
I can add them.
Created by: udeved
I don't have admin control to add you on github. You need to ask @philmmanjaro
Created by: fhdk
That is not an issue - if this works as you expect I will make PR on the core repo
Created by: philmmanjaro
@fhdk: I added you to the group. Please accept the invite. I like were this all goes.
Created by: udeved
@fhdk
Nice, but one little problem...
All fine.
But...
We need to figure out a way to prevent the package from writing a mirrorlist, if it already exists.
Created by: udeved
Once the base-devel group is installed in chroot/root, this chroot will be cloned to chroot/$USER.
While the original chroot/root still has proper mirrorlist set, the user copy chroot has a ranndom mirrorlist, sinnce the package writes a new random list before the rsync chroot copy.
chroot/root
Created by: udeved
For buildiso, it looks different.
There we might want a random list generated.
Created by: fhdk
I have committed some changes to the install scripts for pacman-mirrrors-dev
github.com/manjaro/packages-core@e1ca56ef52Have look at them - I am not experienced bash scripter - but I could figure the script checked for the presence of a pacsave and that check I adopted to check for a mirror list and act accordingly.
I build a pacman-mirrors-dev pkg from the pkgbuild
Created by: udeved
Look good, but, it doesn't work.
Something seems to happen earlier.
Is the random list generation triggered if no network connection?
What I haven't found out yet, how I can set the branch corresponding to the mirrorlist in api mode.
Or is the Branch var in pm.conf not used at all any longer?
Created by: udeved
.: Info Mirror list generated and saved to: /var/build/manjaro-tools/buildpkg/unstable/x86_64/root//etc/pacman.d/mirrorlistNote the root//etc. Perhaps best to sanitize it to avoid path problems?
Created by: fhdk
It is - but you are not using the package I build to today for pacman-mirrors-dev
pacman-mirrors.install - post_install() section
pacman-mirrors-install.script
This should trigger the --no-mirrorlist if a mirror list exist - right?
pacman-mirrors-upgrade.script
I will have a look but I have made some sanitizing in todays package
Created by: udeved
I think this might be better.
Created by: fhdk
And the not operator shouldn't it be outside the []
And why is this better than if exist only update mirror file
--
The sanitizer - I got it backwards :) - in 5-10 a new package is available
Created by: fhdk
oh no - I messed that sanitizing - I was using the same one in two places - have to split them :)
Created by: udeved
The not operator is also accepted outside, but it give a different logic afaik.
sometime, bash is strange thing.
https://www.tutorialspoint.com/unix/unix-basic-operators.htm
http://tldp.org/LDP/abs/html/testconstructs.html#DBLBRACKETS
http://tldp.org/LDP/abs/html/fto.html
;)
What about pm.conf? Is it used any longer to store the branch, or is branch only stored in mirrorlist?
Ideally, I can set also the branch in pm.conf, ad if it doesn't exist, it will be created. The package would later honour the backup array, ie it would create a pm.conf.pacnew
Created by: fhdk
It depends on your use of the api.
If you do not use -U then pacman-mirrors.conf is used to store the branch.
This is the only situation where the pacman-mirrors.conf is not used is with -U.
Created by: udeved
I see, I use almost only -U, but it would be nice to reuse the chroot, if the branch was set in pm.conf. It may be unreliable with mirrorlist only?
On buildiso, I would need the branch set in pm.conf. No problem to do it in scripts, but its a nice to have pm feature I think.
Created by: fhdk
It is not unreliable with the mirror list - as long as the pm installation does not touch an existing mirror list.
With the changes to the install script and hook scripts this shouldn't be an issue looking forward.
I have been around long enough to realize that the pm installation shouldn't touch the mirror list at all unless it does not exist. Seen to many forum issues relating to exactly that which is also why I have added the -y argument to the situations where pm should create the mirror list.
Created by: udeved
Hmm, I could call pm twice, one time before packages get installed to set branch and build mirror, and second time after all packaged have been installed, to set branch permanently annd to set new mirrorlist.
This would work around not having the branch set in pm.conf for the iso.
Created by: udeved
Perhaps something like this would be nice?
$ sudo pacman-mirrors -a -p /mnt/work -U http://mirror.netzspielplatz.de/manjaro/packages/ -S unstable -W-W -- write branch to pm conf
Created by: fhdk
I don't see why - the only program caring about pm.conf is pm and pm is not installed in the chroot.
So why bother writing a pm.conf with a branch for an app that is not installed?
The other approach where you call pm a second time seem more logical as the installation of pm not touches an existing mirror list.
When you're just about ready to close and create the iso
which will set the final branch and create a random mirror list with all mirrors
Created by: udeved
Well, we distribute the chroots for iso. So I need to set the branch in pm.conf in the iso chroot.
Otherwise, as soon as user runs pm on the new install, it will set branch to stable.
Created by: fhdk
When you know that pacman-mirrors is installed you can manipulate that pacman-mirrors.conf almost anyway you like.
It makes no sense to install a configuration file for an app not installed - lets say if pm could install a conf then this conf would have to be maintained two places in code for the api and ex code for the installer. That is prone to go wrong at some point and it makes no sense.
Created by: udeved
Yes, true, I can sed the file, but, we would rely on the installer to set the branch permanently, the instllaer would have to call pm. Thats a cool thing, but it should not be a depend, eg to select the branch for the install, currently not possible is on todo.
The installer basically accesses the same chroot again, similar procedure in general to to create the iso and connfigure some chroot stuff.
Created by: fhdk
I thought that all releases with exception of the testing stuff was preset with stable branch.
Are you telling me your'e are trying to build an option for the end user to select which branch the user wants?
Created by: fhdk
So you just want a pm.conf to put on the iso where the only content is a chosen branch?
pacman-mirrors.conf
Created by: udeved
Yes, I have to make it permanent on iso. Not so important in pkg chroot, but iso chroot I need to set it.
Would be totally sufficient it this happened on -U too, and it writes a pm.conf with branch.
Created by: fhdk
I can do that :)
But this specific has the potential to screw a users config and it cannot be copied to the final installation as the only content will be the branch.
I will need to check for the existence of a conf at the given target and if target file exists refuse to write the branch.
After thought - maybe a little rewrite and reordering.
It is a little confusing working with you - first you don't want to copy a pm.conf then you want pm to write one.
Created by: udeved
Your work is already great help, the build scripts already only rely on pm, and ugly code got removed. :)
Created by: fhdk
Thanks
The pm.conf we are talking about - it will not be copied to the end users install?
Created by: udeved
What got me thinking, I know this is not quickly done.
What if we only relied on the pm.conf branch, and mirrorlist would automatically append $branch/$repo/$arch ?
I am saying, perhaps we should consider eliminating the cluttering of branch setting?
Created by: udeved
I try to explain.
I need to cover the offline install, where the squashed iso chroots get simply extracted in the target install root. So the target install has the stuff set that was set and created at build time. Anything elese to manipulate branch or mirrorlist would have to be donne by the installer.
It looks different for netinstalls, there, we could make the branch selectable in calamares with a combobox. This would be another great use case for pm api mode.
Created by: fhdk
The 'new' pacman-mirrors uses a mirror file which is in json format. This file does pacman-mirrors not change - except on occasion when updating the existing with a new one from Github.
Pacman-mirrors does not include any mirror list like the 'old' pacman-mirrors did. The mirror list is always generated on the fly. So technically when pacman-mirrors gets installed for the first time - there is no /etc/pacman.d/mirrorlist it is generated at the end of the pacman-mirrors installation.
Created by: udeved
I know, the mirrorlist is fine as is now, but the branch would need to be written to pm.conf optioally to avoid user troubles to stick to proper branch the iso was build from. We need to prevent that a user will switch branch without knowing after the install, so a blank pm.conf would set branch to stable from unstable eventually..
Created by: fhdk
So a complete pm.conf must exist on the iso - so the end users pm install is complete.
I think this would be the best way of achieving your goal would be as the final task for the installer
so the end user has a mirror list with up-2-date mirrors and the pacman db's in a synced state
Created by: udeved
Hmm, we don't want the sync db on iso, it is not unnecessary. The repo db will be synced on iso boot.
BUt I am very open to make mirror handling i live session more robust than it is now.
This is the snippet of code from basestrap.
It is executed before pacman pulls in packages in the empty chroot. Ideally, It would also set the branch in pm.conf to avoid calling pm after package install again with different args
Created by: fhdk
The sync db is not for the iso it is for the last task inside the end user install.
You install pm in the chroot used for buildiso - am I right?
At some point after that install you run
As it is - right now - from your previous requests - pm creates a mirror list and exits. Originally pm didn't exit but wrote the branch to pm.conf - that behavior can be restored.
As I have understood you so far a complete pm.conf must exist on the iso for the end user install to be complete.
Created by: fhdk
Your only issue seems to be that you do not have a mirror list in the chroot - pacman does not care about pacman-mirrors.conf.
But why do you copy a pm.conf for an app which does not exist?
I can see the logic if you excute pacman-mirrors in the chroot - but do you?
Created by: udeved
No, I can explain better, its not how it works.
Example buildiso:
buildiso calls for the chroot creation mkchroot which calls basestrap. Basestrap is the lowest level script, it only configures the host pacman to install to a different root dir, using host pkg cache.
So basestrap calls pacman-mirrors before it calls pacman. Before, I had to take a different route and this was done in yet another script, chroot-run
I don't copy the pm.conf any longer to chroot, I use consequently your enhancements.
But, if basestrap runs pm, it only sets the mirrorlist in the right brannch, but the package installs a default pm.conf. If a pm.conf was alrewady present, the default pm.conf from the package was not used, but installed as *.pacnew file, thus the branch remains set in pm.conf.
Created by: fhdk
I see.
What I am thinking is
Instead of having pm write its own config we could let
pacmanpacman-mirrors pull the configuration from the machine where it is run and modfied according to the api arguments.This would solve the issue of maintaining the conf in code - which I must add is a very bad idea.
We would have to add a parameter telling pacman-mirrors to copy its own conf to the $prefix destination
Created by: fhdk
My head cannot hold any longer - but I will look at the possibility tomorrow
Created by: udeved
However, we could take a different approach, we could make the package detect the branch from mirrorlist and make it write it to pm.conf.
The trouble I see with pm in general for some time, the translation between Branch and mirrorlist branch is kind of troublesome. I think Branch = should be in pacman.conf
Do you mean pacman-mirrors? I am thinking since a long time, that lots of pm stuff should go in a pacman patch. Sorry you need to do same gymnastics I did with scripting it, but its highly useful to have an api to code against instead of replicating the pm thing for each and every chroot creation project.
Created by: udeved
@fhdk
What about a pm.conf generator on the fly?
Generate on first pm run, ie if no pm.conf is present?
Could live in /run/pacman-mirrors/pm.conf, and it would take the branch from mirrorlist?
Created by: fhdk
I am sorry - I just realize I got my self all wrong.
Created by: fhdk
I have made the following changes
When run against an empty chroot with api like
$prefix/etc/pacman-mirrors.conf# Protocols =# OnlyCountry =# SSLVerify = False$prefix/etc/pacman.dServer = $mirror/$branch/$repo/$archCreated by: philmmanjaro
@fhdk and @udeved: seems this issue is taking to much loops already.
Case 1: Creation of an ISO
manjaroiso -b testingfor example. The resulting ISO will add testing to its ISO name and therefore pacman-mirror.conf was set to testing branch permanentlySo with this on installation/creation of the chroot the config needs to be set to the wanted branch set with the -b switch thru manjaroiso
Case 2: Net-Install thru Calamares
So in this case the installer defines the branch.
Case 3: Updating an existing installation
This case is a little more complex as it is a two-part option. During an update you can renew the mirror.json list but not touch any config or mirrorlist. When the user however wants to do it, it should be done in a simple way, but not in combination of an update-process.
I hope this might clear it up. How to solve it is up to you guys. The functions might be already there, only documentation might be missed ...
Created by: philmmanjaro
The normal user might not use the API at all. That is designed for tools. In this case the result and the way pacman-mirrors acts might be total different as to the normal user cases 😅
Created by: udeved
I think, most api stuff was ideally at home in pacman source.
It was all much more easy if pacman could be supplied with a branch arg too.
Setting branch needs to do two things, set branch in pm.conf(or ideally in pacman,conf) and adapt the mirrorlist.
In case of basestrap, it calls pm before pacman installs packages, but only sets the mirrorlist branch.
This script is called by either buildpkg or buildiso through mkchroot.
If I was runig basestrap standalone, it should also make branch permanent.
So, don't care too much about iso case.
Created by: fhdk
Thanks for expressing my thoughts so clearly.
With the changes made to the install scripts the mirror list is only created if it does not exist.
To accomodate for the specifics of @udeved iso building pacman-mirrors can ensure a sane enviroment exists. The -U argument does not touch an existing enviroment - it only creates a sane environment for building iso.
The end user have some sane options to change his config through the api.
Created by: udeved
Or, @fhdk we make the api a standalone project, eg pacman-branch?
https://github.com/manjaro/manjaro-tools/blob/devel/bin/basestrap.in#L111
Created by: philmmanjaro
You guys are almost there. You only have to differ from user cases to toolbased cases. That is all. Define all possibilities in a sane way and we can accomplish it also in one application, since they are similar anyway.
Created by: udeved
@philmmanjaro
I have removed syncfirst pacman and defaulted to manjaro-system-ng in unstable.
First confused users included, typical troubles, pacnew file handling. :)
Created by: fhdk
Using that idiom
user cases is
tool cases is
Created by: fhdk
Created by: philmmanjaro
@udeved: is it already ready or do we need a new branch, staging for example? As of now we use unstable for normal building packages to be later snapped over to testing to be reviewed by a wider audience to be then finally dropped to stable. However we don't have yet the option to hold a set of packages back, as this is still missing in boxit as function. It is always good to look forward, but adding new functions so close to the end of a development cycle for our next stable ISO is a little to far fetched. Anyway, let me review the changes made.
@fhdk: seems we are on the right track now. Sometimes the discussion takes longer since we all understand them differently. So if a discussion takes to many loops it is better to stop, take a breath and talk to the other person why there might be a misinterpretation to clear that up first. Then we can tackle along ...
Created by: udeved
@philmmanjaro
I pushed the set of packages in unstable last night before bed time.
This would a case for having extra repos, but I wanted it done finally because other stuff depends on it, so I pushed it.
Created by: philmmanjaro
@udeved: it is OK, however we should rethink on how we use the branches as of now in development with given tools. It might work as is already, but it might also take longer to polish. Adding a fourth layer is no problem. It might solve it for now. Let me think on how we may find a better working place with the options we have now not to have to modify to much tools.
Created by: udeved
I would be actually more for either keeping 3 branches, or reducing to two branches, but we add repos.
Not sure a 4th branch helps us to keep overview. I usually know only whats going on in unstable branch, its where all our packages go, and why I don't push them to testing too.
Created by: fhdk
For the case idiom - those two which has practical use for the end user - I think it would make sense to make them independent of the --api argument.
My reasoning is the less we can draw attention to the --api argument - the lesser chance of a user making an unintended change to the system. Removing their dependency of the --api argument does not change their function when --api is present - they just work both places.
Nah - better keep it the way it is.
Created by: udeved
I would see the benefit of a separate little tool that it may be easier to merge it in pacman source with a patch. In my view, the cool api features replicate stuff pacman can do, so pm is able to do the stuff pacman is missing with the branch.
Created by: fhdk
Is pacman patched already?
I have a weird thing when I build a package with buildpkg
Created by: udeved
You are on unstable, right?
sudo mv -v /etc/pacman.conf.pacnew /etc/pacman.confCreated by: philmmanjaro
OUT-TOPIC
@udeved
We have several issues with this as it is a little against the current concept:
Now we have some stuff which collides with that:
We simply have to figure out on how we merge those two different efforts. Arch simply don't like to maintain a second init system like OpenRC. New tools are included in the development process of the whole distro.
As of now we are a small team which takes a snapshot of Arch, compiles a few custom packages to that set and tries to stabilize it. No 3rd-party projects added, which doesn't exist on Arch itself.
Since we do that time to time we may collide. Also user will post issues to those extra packages only we create in those given branches.
Adding new repos might only give you the option to place the packages above a repo like core. This might be useful for a init-system. However, that need to be tested first. I don't say it is not possible, but we need a new testbed, which is outside of the given development cycle.
So if I create a new branch called staging, we may clone the given unstable to it and even create a new repo for your OpenRC stuff, to see if that ease something on your end.
As always: any change needed to the given tool-chain needs a in-deep discussion first with some members of the team, as it may affect the process flow we currently use for developing Manjaro.
Created by: udeved
@philmmanjaro
But we have our own tools, pacman-mirrors, mt, and other bigger packages in community.
The issue I pointed out is not really related to openrc, its related to our manjaro packages.
The trouble is with tools packages much more than openrc, would be simply better to have the nonsystemd stuff in a different repo. Would open up new possibilities to keep systemd and nonsystemd out of their way respectively. We could simply activate or deactivate the repo, idiot proof so to speak.
Created by: philmmanjaro
@udeved: We have our own tools. Yes. However those don't collide with the process how the distro works. Packages for another init-system as systemd however do. They may collide in some situation and currently hard to handle.
We currently have a working process in regard compiling packages against those we get from Arch. Since we added a new init-system, which doesn't exist upstream, some things don't work as they should. However we managed them in a way so far.
Adding new repos might be an option as pacman is designed for that. Maintaining a 3rd-party project on top of an given fast changing project is hard. I've done that with KDE-Mod in the past. Your Open-RC set is similar to that. For that to work you need to be doing it on your own speed. That is why we decided to use snapshots of Arch rather than using the repos directly. So we have time.
With the current small team for Open-RC it simply is a 3rd-party project as of now, not used by other Manjaro developers. This might be a problem for the user in quality of the given packages. However it is also a good thing to give people the choice to have a second init-system available and supported. Take it as an example.
Lets talk about the tools. Those are also designed for the purpose to compile packages against the snapshots we take from Arch as our upstream project. Yes, we are our own distro but are somehow dictated to follow the changes made by Arch. We are still far away on managing our own distro completely for given reasons.
Non the less we have to talk within the team on changes made to core packages. The simple task to get the API function to pacman-mirrors is currently done the right way. We use the development package to test it out. Therefore we don't collide with given packages and also don't change the way our distro works. However changing the way manjaro-system package works, breaks that rule. That has to be done with pacman-dev until it can be merged to pacman.
You always wondered why we have so many development packages. It is simple. To have a test-bed outside the current stable development.
We all work with the given workarounds as we didn't find the time to add some features to boxit.
So, if changes are needed to the current development process, we are all ears, but changing stuff cos it is just needed to give you the freedom to continue the current development cycle you're currently in is simply a no-go for the given project.
There are reasons why nobody uses the development branch of manjaro-tools as of now. Also why everybody is discussing every small change made to the packages. We are currently on the end of a development cycle and therefore we need stabilization. When that is accomplish we are all free to review your fantastic work and might have time to discuss on how we may include it to the current project and therefore start the development of v17.1 as changes are made to the distro is made.
Please start to see the bigger picture and why some developers act as they do. We all want the same in the end. Provide a nice distro as Manjaro with cool features not every distro has.
Changes can be made always, however they should not collide with current process flows when they are not widely tested.
Let me think on a proper solution and how we may continue to have all the options and functions we plan or currently have for Manjaro.
Created by: udeved
A practical example is system packages that are named *-dev, instead of having simply higher version in unstable branch. This little issue triggers a chain of little changes to adapt static package lists for example use in build processes.
Further trouble is sometimes iso-profiles being not in sync with tools within the branch. So I am also fine with an extra branch, but I don't think its the best solution, it just moves the problem until blacklisting works in boxit.
Created by: philmmanjaro
We use -dev packages as a workaround due given facts. Adding another branch might do the same. Having a higher version in the packages might be the solution, sure. However, who has currently time to tackle boxit? Do you?
So we either have to change the process for now and make testing to unstable and use unstable as a staging platform or we create another branch and do it there until we have a proper solution. Anyway. Changes made to packages which may affect the flow our distro works need a discussion within the team. That is why some developers are sometimes on the edge with changes made not discussed properly. They may be useful but might have simply a wrong timing as the focus is somewhere else ...
Created by: udeved
I would take the time, but I would take the time for boxit to realize what is described in the forum thread.
We would either end up with one boxit server instance running, or two instances, but I'd prefer one, which has two sync urls where it gets repos from.
I would simply let boxit do where its good at, branching, snapshots, but uplodading manjaro specific packages would be done by a new client directly to sync url.
Created by: fhdk
I realize I am the new comer and as that I have to adjust to the way things are done.
From other aspects of my life I have learned
It is always the best policy for a new comer to learn how things are done and in that process I might have a lot of thoughts and ideas on how to improve this or that, I always tells my self to keep quiet and work with what I got. If I am asked an opinion then I won't be afraid of spelling it out because I know it is wanted.
I know it might sound old-fashioned but it actually helped me a lot from getting frustrated and saved me a lot of trouble with my collegues.
Which is why I cannot comment on the OT issue here - only offer some hard learned advice.
Created by: philmmanjaro
That is fine. However we had not yet the time to document boxit as we should have. Also the core developer is not part of our distro anymore. From the develop process we sync Arch packages to unstable, compile our packages if needed and snap the new package set to testing. Therefore testing is kinda frozen and only needed packages to fix issues should been put there. When the test was successful we snap the fixed set to stable, which is also always frozen. Only security updates or needed fixes should go directly to stable.
With our devtools we were able to compile for each branch if needed. We had a dedicated security team of packages who only compiled packages for testing or stable.
manjaro-tools then replaced devtools, but kept the process flow we had designed.
After adding another init-system or other stuff the current process and options we have might be not enough anymore and new features are needed. We managed to avoid them or used complicated workarounds to temporally get to the next step.
Since we are now a larger team, more talking and rules are needed to keep the process going. Changes might take longer to be accepted by all. So the planning is also more in focus now and our development cycles take longer now.
So we have to document more, so everybody might get to know all processes we have. If only one person is doing it, why document it. That person already knows what he is doing. So even with the API function in pacman-mirrors it took a while to match all needs there given to accomplish the goal which satisfies everybody.
In the end we will get there where we want to go.
Created by: philmmanjaro
@fhdk: exactly. That is why pacman-mirrors works so good. Take what is given and improve it. Add new features if they make sense and test them before they get merged to the release package. That is why we have -dev or -git packages. Having just one application as a project is much easier than maintaining the whole distro and keep everything on track. Together as team we may even achieve that.
Created by: udeved
Its actually what drives me as human being, I always want to improve stuff with better solutions.
I kind of dislike workarounds, and I have come to this exact conclusion after some years fiddling with manjaro. We should take it to next level and eliminate the many, many workarounds to make stuff fit, because we don't build all packages for example.
I think @jonathonf summed it up, we need to decide if we stay hobby project organization, or little bit more organized and professional. We gain many new users, and I think the hobby approach will soon reach its limits and we can't handle all the stuff any longer.
Created by: philmmanjaro
@udeved: exactly. As of now it is still a hobby project which demands a lot of time from us all. Every improvement is welcome. I'm also sick of all those workarounds and explaining everyone why I get a little "angry" about things pushed or added at a wrong time-frame.
Since we identified a few of them, we should make a plan on how we solve them together. What is even needed we didn't had thought of. All changes and options should be discussed. This might need a meeting with most of our team to brainstorm. Also it would be good to know who put how much time into Manjaro already or is willing to do so when things get more organized.
We should take it step by step as always to find the proper way to manage the bigger goal.
So I recommend to prioritize boxit a little higher now and see if we may optimize its functions for the better for us all. Also a full documentation and guideline for new packagers might be useful to have, plus a better user management of boxit itself. The bare minimum might be not enough anymore.
And if our repository structure needs to be changed, we should discuss it also. Don't get me wrong. I'm open for new things, but I won't throw working stuff we had accomplished over the years out of the window so fast.
Created by: fhdk
I participate in a Linux usergroup a couple of times a month I have been doing for a couple of years now. They thought I was a little out cause I didn't use Ubuntu like they did. But lately they have been watching Manjaro gaining more and more attention and some has begun to ask if its worth changing.
So the interest for Manjaro is growing and so the user base is growing - which can also is reflected at the forum.
How this should affect organization or structure I cannot say but polishing and adapting the tools is needed but it is bad if it is done at the same time as a release - that would either cause the release to be buggy or postpone it until necessarry testing has been done.
Created by: fhdk
Documentation is one of my priorities - mostly because even I was the only developer - a few weeks let it months gone and I couldn't always remember why a thing was done a specific way.
So I am a documentation writer for good and bad :)
All those questions from @udeved regarding pacman-mirrors has resulted in better understanding of my own code and thereby improved documentation.
A great man once said "If can't explain it in a simple manner, you don't understand it well enough"
Created by: udeved
@philmmanjaro
I see several weaknesses with boxit:
server side
client side
I propose we use the client only for snapshots, and only a few people can actually do snapshots.
Maintainers would upload with a new client to the sync url, where the origial repos exists. The client would only create symlinks and update the repo db. From there, we let boxit crew do their thing.
The new client could be easily connected or replaced by automated build system.
Lets say the new client was a simple deploypkg script part of mt.
Created by: udeved
So my idea is, we simply configure boxit server to read either from second url, or we manage from one sync url, ie we create the sync structure for boxit server ourselves, and focus on separating branching necessary for the server, from uploading packages to a repo pool. It would be more fiddling with urls syncing arch and manjaro repo pools for boxit server.
Result should be, minimal changes to boxit-server code, and we have an additional client for the simple upload where the server syncs from.
I would think, we we take the perhaps justified workaround, to sync arch repos to a manjaro url, where it also hosts the manjaro specific repos, annd we let boxit-server syc from this url, it saves some code changes.
Created by: fhdk
@philmmanjaro
I am trying to put together a documentation like document for boxit.
I will use the knowledge I have gained while setting up a boxit server local and will incorporate those from the forum post. I will send you a draft on your email when I am through it.
Created by: philmmanjaro
@fhdk: thx mate. More people will use boxit after all. Its first concept was already good and we taught each other on how to use it. However, nobody took the time to document it.
@udeved: as always. New functions should be tested first and then integrated properly to the current project. If a droplet for testing this is needed, let me know. We never should affect the current process by testing or introducing new stuff. Only after common approval we should make the change. Do the needed changes in a fork or new git branch and present the results for discussion. Maybe concentrate on core repo first and don't start with a full working distro to begin with.
Created by: udeved
Now you are getting on board.
Absolutely, plan was first to have a manjaro specific core repo additionally, which can be slowly filled.
Shouldn't be a problem then to enhance the filer for arch upstream as soon as we build a given package ourselves. Bash comes to mind, solving desktop settings bash customization.
Its about that we realize the necessary foundation to have this option at hand at all.
I currently don't see an option with current structure. Yeah, a testing instance of a boxit sever, a available sync url would do it. Perhaps even a SF repo will do to sync the boxit server instance?
Created by: philmmanjaro
@udeved: again you're all over the place. Take it step by step. I'm not against changes, but against unplanned pushing of things no one else uses currently. If a new approach might be better, it should be tested totally separately and as long as needed, presented to the team and discussed before it replaces or get merged to the current project. And please no one-man shows anymore. If infrastructure is needed for these test-cases we can accomplish. I simply don't like ideas not thought fully thru.
Since we are now far from the original topic with this? Does @fhdk still need to do some changes to the current API or is it fine as is right now?
Created by: udeved
@philmmanjaro
I don't want to fiddle with the running boxit server. Who is saying that?
I would like to have test boxit server instance, that is perhaps accessible by more than just one member, ie not a home test environment. This is where you or someone with the power to set this up would come into play. Let the rest be not your concern, you would be informed if something workable comes out, because you could try yourself.
I don't get your hostility here.
Created by: philmmanjaro
@udeved: each project I had mentioned here should be taken as an example. It simply doesn't matter which part we change of Manjaro. As long as it changes the work flow in some kind of way we had communicated to the team it should be tested in a testbed which doesn't affect the current established processes. That is all. It is also clear for me that it should be a separate instance of boxit to test those changes or ideas. Again, if therefore some new infrastructure is needed, lets talk. Else prepare the changes and present them to the the team as always. Nothing more nothing less. See it as a normal PR we will discuss as needed.
Created by: udeved
I can't present changes without beig able to configure a boxit server with a different sync url.
So we agree, lets have a test evironment, where several people cann work on, which goes separate from default workflow. Once we have something that works, wre ca discuss to make it default. All I am saying.
I would like a cooperative effort, but precondition is simply a necessary url, rights granted, test instance to make it cooperative,something that should be possible with the server privileges of those able to set this up.
Created by: fhdk
I have the details for configuring and setting up a boxit server - I have one working since yesterday.
Created by: fhdk
@udeved I hope you don't mind - I do think you are putting to much into @philmmanjaro s opinion - I do not sense hostility - au contraire - I sense a will to help accomodate your wishes.
Created by: philmmanjaro
@udeved: again. If a new infrastructure is needed lets talk. If @fhdk found a way to do it separately it is also OK. What I only see is. You jump from one task to the next and then back and forth without having the complete project in picture. This you can also see on this issue report, which went out-topic several posts ago.
This discussion would be better placed in our mailing list. I'm working for a lot of big projects also at my work. And structure is the key in all.
So the last time: Does @fhdk need to change anything on the API of pacman-mirrors or is the original request accomplished by now?
Created by: udeved
@philmmanjaro
I see a bigger picture, like you demanded.
Created by: philmmanjaro
@udeved: good, but I still don't see a clear answer on the our original request for pacman-mirrors to add a proper API so other tools may use that function. Is it accomplished with the given changes made or not? If so, we can close this request and might release v4.2 of pacman-mirrors as given.
@fhdk: any other functions or things prevent us from releasing the next series of pacman-mirrors?
Created by: udeved
@philmmanjaro
.
Regarding pacman mirrors, I 'd say almost there, if it set also branch in pm.conf in api mode, I think mission accomplished.
Created by: fhdk
Try with the latest package. The actions and the order is described above.
It sets a pacman-mirrors.conf if none is found otherwise not.
-U forces a mirrorlist with $url since that is the purpose.
Please test and let me know if you're satisfied.
https://github.com/manjaro/pacman-mirrors/issues/105#issuecomment-306407470
https://github.com/manjaro/pacman-mirrors/issues/105#issuecomment-306401753
Created by: philmmanjaro
@udeved: OK, if that is the last missing step to have a proper API @fhdk might add that and give the changes for review. This also means, development for pacman-mirrors works. Hope we can accomplish that also for other projects we have. So far I like the changes made to pacman-mirrors over the last releases.
Created by: fhdk
@philmmanjaro
The goal for this issue has been reached 🎉
Created by: philmmanjaro
That is excellent news @fhdk. Lets test it for regressions and consider to release v4.2 when ready.