add feature to publish iso files on server

philm commented

2015-07-07 19:52:25 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Currently I'm using some personal scripts to maintain my install medias on sourceforge. To create check-sums we already implemented:

md5sum *x8*iso > manjaro-xfce-minimal-0.9.0-dev-x86_64-md5sum.txt
sha1sum *x8*iso > manjaro-xfce-minimal-0.9.0-dev-x86_64-sha1sum.txt
sha256sum *x8*iso > manjaro-xfce-minimal-0.9.0-dev-x86_64-sha256sum.txt
sha256sum *i686*iso > manjaro-xfce-minimal-0.9.0-dev-i686-sha256sum.txt
sha1sum *i686*iso > manjaro-xfce-minimal-0.9.0-dev-i686-sha1sum.txt
md5sum *i686*iso > manjaro-xfce-minimal-0.9.0-dev-i686-md5sum.txt

On the other hand I'm using rsync to upload my images to sourceforge:

#!/bin/sh
user=[sourceforge user]
project=[sourceforge project]
path=0.9.0/xfce-minimal/0.9.0-dev
files=*0.9.0*
pwfile=`pwd`/passwd
sshpass -f $pwfile rsync -vP  --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path
#sshpass -f $pwfile rsync -vP  --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path
#sshpass -f $pwfile rsync -vP  --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path
#sshpass -f $pwfile rsync -vP  --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path
#
#shutdown -h now

As you can see I'm using also sshpass to provide the needed password in a separate file called passwd. Optionally I shutdown my system when the images are larger.

As the connection might break I call the rsync cmds several times.

6.34.181.57: Broken pipe

rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.1]

Using rsync will continue to upload your images. If you have minimal changes you upload in seconds to get the new image there. All you have to use is the same filename.

./up.sh
manjaro-xfce-minimal-0.9.0-dev-i686-build.log
        132,441 100%  125.64MB/s    0:00:00 (xfr#1, to-chk=13/14)

It might be good if we would have a proper solution within our manjaro-tools.

either config file
or switches to adjust the needed settings
option to pass the needed password ad-hoc

*Created by: philmmanjaro* Currently I'm using some personal scripts to maintain my install medias on sourceforge. To create check-sums we already implemented: <pre>md5sum *x8*iso > manjaro-xfce-minimal-0.9.0-dev-x86_64-md5sum.txt sha1sum *x8*iso > manjaro-xfce-minimal-0.9.0-dev-x86_64-sha1sum.txt sha256sum *x8*iso > manjaro-xfce-minimal-0.9.0-dev-x86_64-sha256sum.txt sha256sum *i686*iso > manjaro-xfce-minimal-0.9.0-dev-i686-sha256sum.txt sha1sum *i686*iso > manjaro-xfce-minimal-0.9.0-dev-i686-sha1sum.txt md5sum *i686*iso > manjaro-xfce-minimal-0.9.0-dev-i686-md5sum.txt</pre> On the other hand I'm using rsync to upload my images to sourceforge: <pre>#!/bin/sh user=[sourceforge user] project=[sourceforge project] path=0.9.0/xfce-minimal/0.9.0-dev files=*0.9.0* pwfile=`pwd`/passwd sshpass -f $pwfile rsync -vP --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path #sshpass -f $pwfile rsync -vP --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path #sshpass -f $pwfile rsync -vP --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path #sshpass -f $pwfile rsync -vP --progress -e ssh $files $user,$project@frs.sourceforge.net:/home/frs/project/m/ma/$project/$path # #shutdown -h now</pre> As you can see I'm using also sshpass to provide the needed password in a separate file called **passwd**. Optionally I shutdown my system when the images are larger. As the connection might break I call the rsync cmds several times. ``` 6.34.181.57: Broken pipe rsync: [sender] write error: Broken pipe (32) rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.1] ``` Using rsync will continue to upload your images. If you have minimal changes you upload in seconds to get the new image there. All you have to use is the same filename. ``` ./up.sh manjaro-xfce-minimal-0.9.0-dev-i686-build.log 132,441 100% 125.64MB/s 0:00:00 (xfr#1, to-chk=13/14) ``` It might be good if we would have a proper solution within our **manjaro-tools**. - either config file - or switches to adjust the needed settings - option to pass the needed password ad-hoc

philm commented

2015-07-07 20:14:11 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Lets say we have a publishiso script.
How do we handle passwords?
Plain text password files is no good I think.

*Created by: udeved* Lets say we have a publishiso script. How do we handle passwords? Plain text password files is no good I think.

philm commented

2015-07-07 20:48:26 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Please check sshpass for this. It either needs a file with plain text or the password added with -p switch in plain text. Especially read the Security Considerations section.

*Created by: philmmanjaro* Please check [**sshpass**](http://linux.die.net/man/1/sshpass) for this. It either needs a file with plain text or the password added with -p switch in plain text. Especially read the **Security Considerations** section.

philm commented

2015-07-09 16:32:57 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Yes, I know, so we don't store a ssh password locally, we always use password switch.
Or do we also want to store an encrypted password locally?

*Created by: udeved* Yes, I know, so we don't store a ssh password locally, we always use password switch. Or do we also want to store an encrypted password locally?

philm commented

2015-07-09 21:25:26 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ryanvade

I don't think storing of the password is a good idea. Too many problems..even if it is encrypted. Then you have to worry about the encryption.

*Created by: ryanvade* I don't think storing of the password is a good idea. Too many problems..even if it is encrypted. Then you have to worry about the encryption.

philm commented

2015-07-10 05:12:31 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Well, somehow you need to "store" it as your upload might get broken and with a re-try you need it again. If you upload several images you do it over night without yourself present ;)

*Created by: philmmanjaro* Well, somehow you need to "store" it as your upload might get broken and with a re-try you need it again. If you upload several images you do it over night without yourself present ;)

philm commented

2015-07-10 18:05:23 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ryanvade

Perhaps this tool should only work with ssh key login? No passwords to worry about.

*Created by: ryanvade* Perhaps this tool should only work with ssh key login? No passwords to worry about.

philm commented

2015-07-10 20:25:21 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Well, maybe you didn't get it yet how sourceforge sftp upload works with rsync. If the pipe breaks a new cmd needs the password again. As we want it to be automated the password has to be stored somewhere. Maybe in RAM ...

*Created by: philmmanjaro* Well, maybe you didn't get it yet how sourceforge sftp upload works with rsync. If the pipe breaks a new cmd needs the password again. As we want it to be automated the password has to be stored somewhere. Maybe in RAM ...

philm commented

2015-07-12 22:02:38 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ryanvade

I don't use sourceforge so no, I don't know how they deal with sftp.

*Created by: ryanvade* I don't use sourceforge so no, I don't know how they deal with sftp.

philm commented

2015-11-11 10:04:37 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

I will try to make it generic, so not tied to SF.

*Created by: udeved* I will try to make it generic, so not tied to SF.

philm commented

2015-11-11 13:44:00 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Ok, some ideas I want to share before implementing something.

Apart from password question, we use a centralized location to store isos,
We can use it, but how do we want to handle what iso gets uploaded?

We could make the cache_dir_iso to have subdirs, named according to iso, and we specify what iso to upload.
We could also have all isos uploaded that are in the cache_dir_iso, and perhaps have a blacklist to exclude some iso.
Or, we have eg /etc/manjaro-tools/deploy.d/{xfce, net, kde} , created when iso is built, and we use then a target arg to select upload, or else all are uploaded.

Please share your thoughts, perhaps last idea makes most sense?

*Created by: udeved* Ok, some ideas I want to share before implementing something. Apart from password question, we use a centralized location to store isos, We can use it, but how do we want to handle what iso gets uploaded? We could make the cache_dir_iso to have subdirs, named according to iso, and we specify what iso to upload. We could also have all isos uploaded that are in the cache_dir_iso, and perhaps have a blacklist to exclude some iso. Or, we have eg /etc/manjaro-tools/deploy.d/{xfce, net, kde} , created when iso is built, and we use then a target arg to select upload, or else all are uploaded. Please share your thoughts, perhaps last idea makes most sense?

philm commented

2015-11-12 00:14:05 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

$ deployiso -q
==> manjaro-tools
  -> version: r1653.59e9e19
  -> user_config: /home/devtux/.config/manjaro-tools.conf
==> HOST:
  -> cache_dir_iso: /build/manjaro-tools/iso
  -> dist_release: 15.09
==> REMOTE:
  -> remote_url: frs.sourceforge.net
  -> remote_project: manjaro-lxqt-openrc
  -> remote_target: /home/frs/project/m/ma
  -> remote_user: udeved
  -> remote_pwd: Please set your password!
  -> url: udeved,manjaro-lxqt-openrc@frs.sourceforge.net:/home/frs/project/m/ma/manjaro-lxqt-openrc

Now we have to think about password storage and what mechanism we want for iso selection.

*Created by: udeved* ``` $ deployiso -q ==> manjaro-tools -> version: r1653.59e9e19 -> user_config: /home/devtux/.config/manjaro-tools.conf ==> HOST: -> cache_dir_iso: /build/manjaro-tools/iso -> dist_release: 15.09 ==> REMOTE: -> remote_url: frs.sourceforge.net -> remote_project: manjaro-lxqt-openrc -> remote_target: /home/frs/project/m/ma -> remote_user: udeved -> remote_pwd: Please set your password! -> url: udeved,manjaro-lxqt-openrc@frs.sourceforge.net:/home/frs/project/m/ma/manjaro-lxqt-openrc ``` Now we have to think about password storage and what mechanism we want for iso selection.

philm commented

2015-11-12 11:34:13 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

It would be easier, if all iso were uploded to the same location.
Reason, we can come up with a server structure that gets created before upload with ssh login.
But, if we want to create remote folders, it would be good to have a common structure.

eg

official releases
|
|-----xfce
|-----kde

community releases
|
|----cinnamon 
|----lxqt

How should openrc iso integrate?
We use openrc appended to iso file name.
Probably best was to simply have subfolders in above structure.

*Created by: udeved* It would be easier, if all iso were uploded to the same location. Reason, we can come up with a server structure that gets created before upload with ssh login. But, if we want to create remote folders, it would be good to have a common structure. eg ``` official releases | |-----xfce |-----kde community releases | |----cinnamon |----lxqt ``` How should openrc iso integrate? We use openrc appended to iso file name. Probably best was to simply have subfolders in above structure.

philm commented

2015-11-12 17:42:51 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Great effort so far. Yes, we have to think about how we structure the server for this. Also more people need then access to upload to the manjarolinux project. However we can start with the manjarotest project first. Most have only access to manjarodev.

*Created by: philmmanjaro* Great effort so far. Yes, we have to think about how we structure the server for this. Also more people need then access to upload to the **manjarolinux** project. However we can start with the **manjarotest** project first. Most have only access to **manjarodev**.

philm commented

2015-11-12 19:55:30 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

I have created a sandbox repo, we we can play around with structure.
Phil has admin access.
https://sourceforge.net/projects/manjaro-testing/

*Created by: udeved* I have created a sandbox repo, we we can play around with structure. Phil has admin access. https://sourceforge.net/projects/manjaro-testing/

philm commented

2015-11-13 11:42:05 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Quick overview.

I implemented subfolders for the cache_dir_iso in buildiso, so the cache_dir basically holds server tree.
deployiso can be called with profile name to select the iso folder to upload, else it will take cache_dir as default(not implemented yet).
Before uploading, deployiso has to be run with -c arg, to create a release folder on server.
Perhaps we avoid shell session and somehow use rsync to create the folders in separate call?

We now have a version generator for the dist_release var.
The iso will be categorized automatically, if official or community edition, based on iso sets definitions.

The downside with SF, before any release is created, the shell service Vserver needs to be manually started atm, or else the server will not be found when trying to connect. The SF shell session lasts 4 hours.

Once everything is sorted, we can add a switch to buildiso, to immediately publish after build.

official
|
15.09 (stable)
    |-----xfce
    |-----kde
|
15.09.1 (unstable)
    |-----xfce
    |-----kde
|
15.09.2 (unstable)
    |-----xfce
    |-----kde
|
15.12 (stable)
    |-----xfce
    |-----kde

community
|
15.09 (stable)
    |----cinnamon 
    |----lxqt
|
15.09.1 (unstable)
    |----cinnamon 
    |----lxqt
|
15.09.2 (unstable)
    |----cinnamon 
    |----lxqt
|
15.12 (stable)
    |----cinnamon 
    |----lxqt

*Created by: udeved* Quick overview. I implemented subfolders for the cache_dir_iso in buildiso, so the cache_dir basically holds server tree. deployiso can be called with profile name to select the iso folder to upload, else it will take cache_dir as default(not implemented yet). Before uploading, deployiso has to be run with -c arg, to create a release folder on server. Perhaps we avoid shell session and somehow use rsync to create the folders in separate call? We now have a version generator for the dist_release var. The iso will be categorized automatically, if official or community edition, based on iso sets definitions. The downside with SF, before any release is created, the shell service Vserver needs to be manually started atm, or else the server will not be found when trying to connect. The SF shell session lasts 4 hours. Once everything is sorted, we can add a switch to buildiso, to immediately publish after build. ``` official | 15.09 (stable) |-----xfce |-----kde | 15.09.1 (unstable) |-----xfce |-----kde | 15.09.2 (unstable) |-----xfce |-----kde | 15.12 (stable) |-----xfce |-----kde community | 15.09 (stable) |----cinnamon |----lxqt | 15.09.1 (unstable) |----cinnamon |----lxqt | 15.09.2 (unstable) |----cinnamon |----lxqt | 15.12 (stable) |----cinnamon |----lxqt ```

philm commented

2015-11-13 18:12:58 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ghost

personal: a user should readout the iso what is a preview or rc or fixed iso if a month later a iso rebuild cause of repo as example, people will find the iso and use it as stable iso's but since its on unstable.. is bit mixed up..
even manjaro got history of minor version, and all most fixed stable version. i understand the version would be te same.. only got issues with it , if you can read from the iso file if its a preview rc series...given only to minor is not enough..

*Created by: ghost* personal: a user should readout the iso what is a preview or rc or fixed iso if a month later a iso rebuild cause of repo as example, people will find the iso and use it as stable iso's but since its on unstable.. is bit mixed up.. even manjaro got history of minor version, and all most fixed stable version. i understand the version would be te same.. only got issues with it , if you can read from the iso file if its a preview rc series...given only to minor is not enough..

philm commented

2015-11-13 18:14:26 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

We should do away with preview vetsions and such things.

*Created by: udeved* We should do away with preview vetsions and such things.

philm commented

2015-11-13 18:17:41 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ghost

people would read 15.12.1 as a fixed version.

*Created by: ghost* people would read 15.12.1 as a fixed version.

philm commented

2015-11-13 18:19:43 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Nope, ringo, seriously, its a good scheme, we need one, except you come up with something alternative and viable. We can't afford to pollute server with dozens of differently named files and folders, we need order there.
who the hell is supposed to code something without a proper structure? We get spghetti code without structure, that handles tons of special cases.

*Created by: udeved* Nope, ringo, seriously, its a good scheme, we need one, except you come up with something alternative and viable. We can't afford to pollute server with dozens of differently named files and folders, we need order there. who the hell is supposed to code something without a proper structure? We get spghetti code without structure, that handles tons of special cases.

philm commented

2015-11-13 19:01:17 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ghost

only i would not know to understand , if peopel download iso A but some days after iso A got fixed and tells isue about iso A how do you explain it, download the iso again that is fixed?

*Created by: ghost* only i would not know to understand , if peopel download iso A but some days after iso A got fixed and tells isue about iso A how do you explain it, download the iso again that is fixed?

philm commented

2015-11-13 19:03:37 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Sure, I don't understand your beef with this, because if some iso is borked, the user has to download anyway new one.

*Created by: udeved* Sure, I don't understand your beef with this, because if some iso is borked, the user has to download anyway new one.

philm commented

2015-11-14 06:24:13 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

well, we can name them also -pre1 and so on. Then it would be clearer. But let us roll with this one on the test-server first to see if some needs still to be changed.

*Created by: philmmanjaro* well, we can name them also **-pre1** and so on. Then it would be clearer. But let us roll with this one on the test-server first to see if some needs still to be changed.

philm commented

2015-11-14 10:18:57 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ghost

also if a build had a borked function lets say. iso A but the fix to name it also iso A not iso A.1 as example, there are users that saves the iso and also mayby weeks after it comes with that iso. how to tell he got that iso. would easier to tell the iso got fixed and a fixed release is up. eventually from te community can tell it. but on first point community wont understand the issue on first place because he had iso a and iso a is current release.. even ubuntu as example 14.04 release there fixed release would be 14.04.1

*Created by: ghost* also if a build had a borked function lets say. iso A but the fix to name it also iso A not iso A.1 as example, there are users that saves the iso and also mayby weeks after it comes with that iso. how to tell he got that iso. would easier to tell the iso got fixed and a fixed release is up. eventually from te community can tell it. but on first point community wont understand the issue on first place because he had iso a and iso a is current release.. even ubuntu as example 14.04 release there fixed release would be 14.04.1

philm commented

2015-11-14 11:51:31 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Ringo, quite frankly, let us just establish a structure without worrying what community may or may not understand. Manjaro-tools are development tools, targeting manjaro devs primarily to make life easier.

@philmmanjaro

Do you think its a viable version scheme for the minor releases? It is basically the same what kde-apps do.
I reverted this scheme back due to the big fuzz it produced.

Open question to me, if we do say 15.11, what codename does it have and what distrib_release version does it have?

*Created by: udeved* Ringo, quite frankly, let us just establish a structure without worrying what community may or may not understand. Manjaro-tools are development tools, targeting manjaro devs primarily to make life easier. @philmmanjaro Do you think its a viable version scheme for the minor releases? It is basically the same what kde-apps do. I reverted this scheme back due to the big fuzz it produced. Open question to me, if we do say 15.11, what codename does it have and what distrib_release version does it have?

philm commented

2015-11-14 12:09:34 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ghost

i think codenames you can pull it of as current development this case 15.11 can have bellatrix

*Created by: ghost* i think codenames you can pull it of as current development this case 15.11 can have bellatrix

philm commented

2015-11-14 12:11:07 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Good you say that, because Bellatrix is 15.09, hence we should use 15.09.2, the thing you guys disliked so much but did not answer how to handle this otherwise.

*Created by: udeved* Good you say that, because Bellatrix is 15.09, hence we should use 15.09.2, the thing you guys disliked so much but did not answer how to handle this otherwise.

philm commented

2015-11-14 12:15:45 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: ghost

Would be ruff if you have every month a different codename Also it can differce from versioning....it can go purely go to Development stage does not mean you must follow 15.09.1 does not tell about that iso is build and developed... codename would have more milestone then a iso build date

*Created by: ghost* Would be ruff if you have every month a different codename Also it can differce from versioning....it can go purely go to Development stage does not mean you must follow 15.09.1 does not tell about that iso is build and developed... codename would have more milestone then a iso build date

philm commented

2015-11-14 12:18:14 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

I stop here, we go in circles, which seriously pisses me off.
You don't need a dev version, if iso has release every month on a rolling release distro.
You would use minor versions to be dev versions, and codenamed version release as major release.
Is it so difficult to understand?

*Created by: udeved* I stop here, we go in circles, which seriously pisses me off. You don't need a dev version, if iso has release every month on a rolling release distro. You would use minor versions to be dev versions, and codenamed version release as major release. Is it so difficult to understand?

philm commented

2015-11-17 18:04:26 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Does anybody know how to start a shell session on SF without login?
Their manual says this to login to remote shell, but I only want to create the 4h lasting session.

ssh -t USER,PROJECT@shell.sourceforge.net create

So, if we want to use ssh, eg to create folders, we need to start a 4h session first, ugly.

I currently use rsync and an empty dir to create more than one level deep folders.

*Created by: udeved* Does anybody know how to start a shell session on SF without login? Their manual says this to login to remote shell, but I only want to create the 4h lasting session. ``` ssh -t USER,PROJECT@shell.sourceforge.net create ``` So, if we want to use ssh, eg to create folders, we need to start a 4h session first, ugly. I currently use rsync and an empty dir to create more than one level deep folders.

philm commented

2015-11-17 18:09:16 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Seem to work without -t arg, but it is painfully slow.

*Created by: udeved* Seem to work without -t arg, but it is painfully slow.

Rows
Columns

add feature to publish iso files on server #121