bad order if ftp error ? #169

Closed
opened 2021-07-25 18:19:01 +00:00 by papajoke · 19 comments
papajoke commented 2021-07-25 18:19:01 +00:00 (Migrated from gitlab2.manjaro.org)

if we have one ftp error, all result list is not sorted ?

https://forum.manjaro.org/t/pacman-mirrors-fasttrack-adding-slow-or-invalid-entries-to-mirrorlist/75348/8

pacman-mirrors -c Bulgaria --api --protocol all
::INFO Querying mirrors - This may take some time
  0.686 Bulgaria       : https://mirrors.netix.net/manjaro/
  1.078 Bulgaria       : https://mirror.telepoint.bg/manjaro/
  0.880 Bulgaria       : http://mirror.telepoint.bg/manjaro/
  ..... Bulgaria       : ftps://mirror.telepoint.bg/manjaro/

::ERROR unknown url type: ftps
  0.003
  1.016 Bulgaria       : https://manjaro.ipacct.com/manjaro/
  0.557 Bulgaria       : http://manjaro.telecoms.bg/
::INFO Writing mirror list
::Bulgaria        : https://mirror.telepoint.bg/manjaro/stable/$repo/$arch
::Bulgaria        : http://manjaro.telecoms.bg/stable/$repo/$arch
::Bulgaria        : https://mirrors.netix.net/manjaro/stable/$repo/$arch
::Bulgaria        : https://manjaro.ipacct.com/manjaro/stable/$repo/$arch

first at 0.5576 is number 2 🙃
second at 0.686 is number 3 🙃

if we have one ftp error, all result list is not sorted ? https://forum.manjaro.org/t/pacman-mirrors-fasttrack-adding-slow-or-invalid-entries-to-mirrorlist/75348/8 ``` pacman-mirrors -c Bulgaria --api --protocol all ::INFO Querying mirrors - This may take some time 0.686 Bulgaria : https://mirrors.netix.net/manjaro/ 1.078 Bulgaria : https://mirror.telepoint.bg/manjaro/ 0.880 Bulgaria : http://mirror.telepoint.bg/manjaro/ ..... Bulgaria : ftps://mirror.telepoint.bg/manjaro/ ::ERROR unknown url type: ftps 0.003 1.016 Bulgaria : https://manjaro.ipacct.com/manjaro/ 0.557 Bulgaria : http://manjaro.telecoms.bg/ ::INFO Writing mirror list ::Bulgaria : https://mirror.telepoint.bg/manjaro/stable/$repo/$arch ::Bulgaria : http://manjaro.telecoms.bg/stable/$repo/$arch ::Bulgaria : https://mirrors.netix.net/manjaro/stable/$repo/$arch ::Bulgaria : https://manjaro.ipacct.com/manjaro/stable/$repo/$arch ``` first at `0.5576` is number 2 🙃 second at `0.686` is number 3 🙃
papajoke commented 2021-07-25 18:26:36 +00:00 (Migrated from gitlab2.manjaro.org)

changed the description

changed the description
papajoke commented 2021-07-25 18:26:47 +00:00 (Migrated from gitlab2.manjaro.org)

changed the description

changed the description
papajoke commented 2021-07-25 18:28:35 +00:00 (Migrated from gitlab2.manjaro.org)

changed the description

changed the description
papajoke commented 2021-07-25 18:29:47 +00:00 (Migrated from gitlab2.manjaro.org)

changed the description

changed the description
ghost commented 2021-07-26 06:14:56 +00:00 (Migrated from gitlab2.manjaro.org)

The reason it is spotted is a symptom caused by an earlier issue.

Some time ago - there was an issue where mirror response was not sorted correctly because the values used was text based and those values couldn't be used in calculations.

The return value from the probing was then changed to float but the sorting function's field argument was forgotten and still defined as string - which will wreck the sorting of numbers.

Hmmm - No - that is not it ...

The reason it is spotted is a symptom caused by an earlier issue. Some time ago - there was an issue where mirror response was not sorted correctly because the values used was text based and those values couldn't be used in calculations. The return value from the probing was then changed to `float` but the `sorting` function's field argument was forgotten and still defined as string - which will wreck the sorting of numbers. Hmmm - No - that is not it ...
ghost commented 2021-07-26 07:29:06 +00:00 (Migrated from gitlab2.manjaro.org)

I found out why.

The Python sorted function does not change the list - but creates a new list.

My mistake was to assume that when using return sorted() a new list would be returned.

Assigning a the result of sorted to a variable then return the variable did the trick.

I found out why. The Python sorted function does not change the list - but creates a new list. My mistake was to assume that when using return sorted() a new list would be returned. Assigning a the result of sorted to a variable then return the variable did the trick.
ghost commented 2021-07-26 07:55:56 +00:00 (Migrated from gitlab2.manjaro.org)

v4.21.5 is pushed to unstable

v4.21.5 is pushed to unstable
papajoke commented 2021-07-26 10:32:17 +00:00 (Migrated from gitlab2.manjaro.org)

no change with 4.21.5

same "no sort" if Bulgaria is in input list (same command as post #1)

no change with 4.21.5 same "no sort" if `Bulgaria` is in input list (same command as post `#1`)
papajoke commented 2021-07-26 10:38:12 +00:00 (Migrated from gitlab2.manjaro.org)

changed the description

changed the description
ghost commented 2021-07-26 15:37:58 +00:00 (Migrated from gitlab2.manjaro.org)

It works at my end

➜  ~ sudo pacman-mirrors -c bg -aSstable -P all       
::INFO Branch in config is changed
::INFO Protocols in config is changed
::INFO Downloading mirrors from Manjaro
::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
::INFO => Mirror status: http://repo.manjaro.org/status.json
::INFO User generated mirror list
::------------------------------------------------------------
::INFO Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.392 Bulgaria       : https://manjaro.ipacct.com/manjaro/
  0.528 Bulgaria       : http://manjaro.telecoms.bg/
::INFO Writing mirror list
::Bulgaria        : https://manjaro.ipacct.com/manjaro/stable/$repo/$arch
::Bulgaria        : http://manjaro.telecoms.bg/stable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id'
::INFO To remove custom config run  'sudo pacman-mirrors -c all'
➜  ~ pacman-mirrors
Pacman-mirrors version 4.21.5
Local mirror status for stable branch
Mirror #1   OK  00:54   Bulgaria  https://manjaro.ipacct.com/manjaro/
Mirror #2   OK  03:14   Bulgaria  http://manjaro.telecoms.bg/
It works at my end ``` ➜ ~ sudo pacman-mirrors -c bg -aSstable -P all ::INFO Branch in config is changed ::INFO Protocols in config is changed ::INFO Downloading mirrors from Manjaro ::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json ::INFO => Mirror status: http://repo.manjaro.org/status.json ::INFO User generated mirror list ::------------------------------------------------------------ ::INFO Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json ::INFO Using custom mirror file ::INFO Querying mirrors - This may take some time 0.392 Bulgaria : https://manjaro.ipacct.com/manjaro/ 0.528 Bulgaria : http://manjaro.telecoms.bg/ ::INFO Writing mirror list ::Bulgaria : https://manjaro.ipacct.com/manjaro/stable/$repo/$arch ::Bulgaria : http://manjaro.telecoms.bg/stable/$repo/$arch ::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist ::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id' ::INFO To remove custom config run 'sudo pacman-mirrors -c all' ``` ``` ➜ ~ pacman-mirrors Pacman-mirrors version 4.21.5 Local mirror status for stable branch Mirror #1 OK 00:54 Bulgaria https://manjaro.ipacct.com/manjaro/ Mirror #2 OK 03:14 Bulgaria http://manjaro.telecoms.bg/ ```
papajoke commented 2021-07-26 20:19:08 +00:00 (Migrated from gitlab2.manjaro.org)

yes, work for you because list is now update : ftp Bulgaria with error is removed from list ;)
no error == no problem ;)

but we have another ftp error in Ukraine, so same "no order" if user select europe

pacman-mirrors -c Ukraine,Sweden --api --protocol all
  0.528 Sweden         : https://ftp.lysator.liu.se/pub/manjaro/
  0.876 Sweden         : ftp://ftp.lysator.liu.se/pub/manjaro/
  0.702 Sweden         : https://mirror.zetup.net/manjaro/
  0.565 Ukraine        : http://mirrors.colocall.net/manjaro/
  ..... Ukraine        : ftp://mirrors.colocall.net/manjaro/

::ERROR ftp error: error_perm('550 Failed to change directory.')

  1.358 Ukraine        : https://fastmirror.pp.ua/manjaro/
  1.056 Ukraine        : http://fastmirror.pp.ua/manjaro/
::INFO Writing mirror list
::Sweden          : https://mirror.zetup.net/manjaro/stable/$repo/$arch
::Sweden          : https://ftp.lysator.liu.se/pub/manjaro/stable/$repo/$arch
::Ukraine         : https://fastmirror.pp.ua/manjaro/stable/$repo/$arch

here "zetup" is first with only 0.7 ...

europe: truncated output and re-ordered lines for easy compare

pacman-mirrors --continent --api --protocol all
...
 ..... Iceland        : ftp://mirrors.opensource.is/manjaro/
::ERROR ftp error: error_perm('550 Failed to change directory.')
  ..... Ukraine        : ftp://mirrors.colocall.net/manjaro/
::ERROR ftp error: error_perm('550 Failed to change directory.')
 0.425 Belgium        : https://mirror.futureweb.be/manjaro/
 0.359 United_Kingdom : http://manjaro.mirrors.uk2.net/
 1.568 Germany        : https://mirror.23media.com/manjaro/
...

::INFO Writing mirror list
::Belgium         : https://mirror.futureweb.be/manjaro/stable
::United_Kingdom  : http://manjaro.mirrors.uk2.net/stable
::Germany         : https://mirror.23media.com/manjaro/stable
...

also bad order (23media.com at pos 3 !! , i have 15 mirrors before)

yes, work for you because list is now update : ftp `Bulgaria` with **error** is removed from list ;) no error == no problem ;) but we have another ftp error in Ukraine, so same "no order" if user select europe ``` pacman-mirrors -c Ukraine,Sweden --api --protocol all 0.528 Sweden : https://ftp.lysator.liu.se/pub/manjaro/ 0.876 Sweden : ftp://ftp.lysator.liu.se/pub/manjaro/ 0.702 Sweden : https://mirror.zetup.net/manjaro/ 0.565 Ukraine : http://mirrors.colocall.net/manjaro/ ..... Ukraine : ftp://mirrors.colocall.net/manjaro/ ::ERROR ftp error: error_perm('550 Failed to change directory.') 1.358 Ukraine : https://fastmirror.pp.ua/manjaro/ 1.056 Ukraine : http://fastmirror.pp.ua/manjaro/ ::INFO Writing mirror list ::Sweden : https://mirror.zetup.net/manjaro/stable/$repo/$arch ::Sweden : https://ftp.lysator.liu.se/pub/manjaro/stable/$repo/$arch ::Ukraine : https://fastmirror.pp.ua/manjaro/stable/$repo/$arch ``` here "zetup" is first with only `0.7` ... europe: truncated output and re-ordered lines for easy compare ``` pacman-mirrors --continent --api --protocol all ... ..... Iceland : ftp://mirrors.opensource.is/manjaro/ ::ERROR ftp error: error_perm('550 Failed to change directory.') ..... Ukraine : ftp://mirrors.colocall.net/manjaro/ ::ERROR ftp error: error_perm('550 Failed to change directory.') 0.425 Belgium : https://mirror.futureweb.be/manjaro/ 0.359 United_Kingdom : http://manjaro.mirrors.uk2.net/ 1.568 Germany : https://mirror.23media.com/manjaro/ ... ::INFO Writing mirror list ::Belgium : https://mirror.futureweb.be/manjaro/stable ::United_Kingdom : http://manjaro.mirrors.uk2.net/stable ::Germany : https://mirror.23media.com/manjaro/stable ... ``` also bad order (23media.com at pos 3 !! , i have 15 mirrors before)
ghost commented 2021-07-27 14:00:52 +00:00 (Migrated from gitlab2.manjaro.org)

The --api --protocol all arguments has no relation to the sorting results - it only indicates that all protocols are accepted and the config is written accordingly.

I am wondering why you get those results - because my result is expected - the sort parameter is the https result because default https is prioritized and only one protocol is used from each mirror. It makes no sense to include both http and https for the same mirror.

An error does not change the sort order as the protocol/mirror combination is not included in the final list.

➜  ~ sudo pacman-mirrors -c ua,se   
[sudo] password for fh: 
::INFO Downloading mirrors from Manjaro
::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
::INFO => Mirror status: http://repo.manjaro.org/status.json
::INFO User generated mirror list
::------------------------------------------------------------
::INFO Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.190 Sweden         : https://ftp.lysator.liu.se/pub/manjaro/
  0.195 Sweden         : ftp://ftp.lysator.liu.se/pub/manjaro/
  0.229 Sweden         : https://mirror.zetup.net/manjaro/
  0.184 Sweden         : https://ftpmirror1.infania.net/mirror/manjaro/
  0.097 Sweden         : http://ftpmirror1.infania.net/mirror/manjaro/
  0.216 Sweden         : ftp://ftpmirror1.infania.net/mirror/manjaro/
  0.296 Ukraine        : http://mirrors.colocall.net/manjaro/
  ..... Ukraine        : ftp://mirrors.colocall.net/manjaro/

::ERROR ftp error: error_perm('550 Failed to change directory.')

  0.711 Ukraine        : https://fastmirror.pp.ua/manjaro/
  0.424 Ukraine        : http://fastmirror.pp.ua/manjaro/
::INFO Writing mirror list
::Sweden          : https://ftp.lysator.liu.se/pub/manjaro/unstable/$repo/$arch
::Sweden          : https://ftpmirror1.infania.net/mirror/manjaro/unstable/$repo/$arch
::Sweden          : https://mirror.zetup.net/manjaro/unstable/$repo/$arch
::Ukraine         : https://fastmirror.pp.ua/manjaro/unstable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id'
::INFO To remove custom config run  'sudo pacman-mirrors -c all'
The `--api --protocol all` arguments has no relation to the sorting results - it only indicates that all protocols are accepted and the config is written accordingly. I am wondering why you get those results - because my result is expected - the sort parameter is the https result because default https is prioritized and only one protocol is used from each mirror. It makes no sense to include both http and https for the same mirror. An error does not change the sort order as the protocol/mirror combination is not included in the final list. ``` ➜ ~ sudo pacman-mirrors -c ua,se [sudo] password for fh: ::INFO Downloading mirrors from Manjaro ::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json ::INFO => Mirror status: http://repo.manjaro.org/status.json ::INFO User generated mirror list ::------------------------------------------------------------ ::INFO Custom mirror file saved: /var/lib/pacman-mirrors/custom-mirrors.json ::INFO Using custom mirror file ::INFO Querying mirrors - This may take some time 0.190 Sweden : https://ftp.lysator.liu.se/pub/manjaro/ 0.195 Sweden : ftp://ftp.lysator.liu.se/pub/manjaro/ 0.229 Sweden : https://mirror.zetup.net/manjaro/ 0.184 Sweden : https://ftpmirror1.infania.net/mirror/manjaro/ 0.097 Sweden : http://ftpmirror1.infania.net/mirror/manjaro/ 0.216 Sweden : ftp://ftpmirror1.infania.net/mirror/manjaro/ 0.296 Ukraine : http://mirrors.colocall.net/manjaro/ ..... Ukraine : ftp://mirrors.colocall.net/manjaro/ ::ERROR ftp error: error_perm('550 Failed to change directory.') 0.711 Ukraine : https://fastmirror.pp.ua/manjaro/ 0.424 Ukraine : http://fastmirror.pp.ua/manjaro/ ::INFO Writing mirror list ::Sweden : https://ftp.lysator.liu.se/pub/manjaro/unstable/$repo/$arch ::Sweden : https://ftpmirror1.infania.net/mirror/manjaro/unstable/$repo/$arch ::Sweden : https://mirror.zetup.net/manjaro/unstable/$repo/$arch ::Ukraine : https://fastmirror.pp.ua/manjaro/unstable/$repo/$arch ::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist ::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id' ::INFO To remove custom config run 'sudo pacman-mirrors -c all' ```
papajoke commented 2021-07-28 21:30:28 +00:00 (Migrated from gitlab2.manjaro.org)

OK, found why order is false :: is not with ftp errors (fixed by last commit) !!
ps: not possible to send you mp in forum for this long text 🥲

bad order is since beginning 😬

I had add "time info" in output filefn.py#L118 when app write list

message = f'{server["country"]:<15} {server["resp_time"]}: {msg_url}'

result :

# test
  0.472 Belgium        : https://mirror.futureweb.be/manjaro/
  0.384 Belgium        : http://mirror.futureweb.be/manjaro/
  0.526 Netherlands    : https://manjaro.mirrors.lavatech.top/
  0.398 Netherlands    : http://manjaro.mirrors.lavatech.top/
  0.509 Switzerland    : https://manjaro.osmir.ch/
  0.436 Switzerland    : http://manjaro.osmir.ch/

# write
::Belgium         0.384: https://mirror.futureweb.be/manjaro/stable
::United_Kingdom  0.392: http://manjaro.mirrors.uk2.net/stable
::Netherlands     0.398: https://manjaro.mirrors.lavatech.top/stable
...
::Switzerland     0.436: https://manjaro.osmir.ch/stable

App use best time but always protocol[0] ! result
"osmir.ch" is in pos 7 (https:0.509) when "lavatech.top" (https:0.526) is at pos 2 🤪


Why test http if we have before https in protocol ? it is a big waste of time, we can skeep if we use always https


http is always most faster than https ;) if user want best speed why not chose best speed and not always first protocol ?


When mirror have only http protocol, it still has an advantage over https (in time) but is not necessarily faster in transfer (if other have only https)

Some mirrors have only http, result:
they are always very disadvantaged against those who have http and https, too bad they could sometimes be in first place because very fast if we compare only the https

exists aur package httpstat-go for understand diff https/http and what pamac-mirrors mesure

httpstat -L -k "https://mirror.futureweb.be/manjaro/stable/core/x86_64/core.db"
  DNS Lookup   TCP Connection   TLS Handshake   Server Processing   Content Transfer
[     32ms  |          36ms  |        159ms  |             99ms  |           274ms  ]
            |                |               |                   |                  |
   namelookup:32ms           |               |                   |                  |
                       connect:69ms          |                   |                  |
                                   pretransfer:228ms             |                  |
                                                     starttransfer:328ms            |
                                                                                total:603ms

httpstat -L "http://mirror.futureweb.be/manjaro/stable/core/x86_64/core.db"
   DNS Lookup   TCP Connection   Server Processing   Content Transfer
[      34ms  |          32ms  |             93ms  |           270ms  ]
             |                |                   |                  |
    namelookup:34ms           |                   |                  |
                        connect:67ms              |                  |
                                      starttransfer:161ms            |
                                                                 total:431ms  

transfert http/https is the same (no reason to test 2 times...) - but time total diff is big ;)


note: with ftp we write on disk so we can't compare to http(s) ...

OK, found why order is false :: is not with ftp errors (fixed by last commit) !! ps: not possible to send you mp in forum for this long text 🥲 bad order is since beginning 😬 I had add "time info" in output filefn.py#L118 when app write list ``` message = f'{server["country"]:<15} {server["resp_time"]}: {msg_url}' ``` result : ``` # test 0.472 Belgium : https://mirror.futureweb.be/manjaro/ 0.384 Belgium : http://mirror.futureweb.be/manjaro/ 0.526 Netherlands : https://manjaro.mirrors.lavatech.top/ 0.398 Netherlands : http://manjaro.mirrors.lavatech.top/ 0.509 Switzerland : https://manjaro.osmir.ch/ 0.436 Switzerland : http://manjaro.osmir.ch/ # write ::Belgium 0.384: https://mirror.futureweb.be/manjaro/stable ::United_Kingdom 0.392: http://manjaro.mirrors.uk2.net/stable ::Netherlands 0.398: https://manjaro.mirrors.lavatech.top/stable ... ::Switzerland 0.436: https://manjaro.osmir.ch/stable ``` App use best time but always protocol[0] ! result "osmir.ch" is in pos 7 (https:0.509) when "lavatech.top" (https:0.526) is at pos 2 🤪 --- Why test http if we have before https in protocol ? it is a big waste of time, we can skeep if we use always https --- http is always most faster than https ;) if user want best speed why not chose best speed and not always first protocol ? --- When mirror have only http protocol, it still has an advantage over https (in time) but is not necessarily faster in transfer (if other have only https) Some mirrors have [only](https://gitlab.manjaro.org/webpage/manjaro-web-repo/-/blob/master/mirrors.json#L103) http, result: they are always very disadvantaged against those who have http and https, too bad they could sometimes be in first place because very fast if we compare only the https exists aur package `httpstat-go` for understand diff https/http and what pamac-mirrors mesure ``` httpstat -L -k "https://mirror.futureweb.be/manjaro/stable/core/x86_64/core.db" DNS Lookup TCP Connection TLS Handshake Server Processing Content Transfer [ 32ms | 36ms | 159ms | 99ms | 274ms ] | | | | | namelookup:32ms | | | | connect:69ms | | | pretransfer:228ms | | starttransfer:328ms | total:603ms httpstat -L "http://mirror.futureweb.be/manjaro/stable/core/x86_64/core.db" DNS Lookup TCP Connection Server Processing Content Transfer [ 34ms | 32ms | 93ms | 270ms ] | | | | namelookup:34ms | | | connect:67ms | | starttransfer:161ms | total:431ms ``` transfert http/https is the same (no reason to test 2 times...) - but time total diff is big ;) --- note: with ftp we write on disk so we can't compare to http(s) ...
ghost commented 2021-12-02 11:19:17 +00:00 (Migrated from gitlab2.manjaro.org)

mentioned in commit 5a313d78e8

mentioned in commit 5a313d78e85ad7592bcee033e7218be9b3e8029e
ghost commented 2021-12-02 11:19:18 +00:00 (Migrated from gitlab2.manjaro.org)

mentioned in commit 5d867a90e6

mentioned in commit 5d867a90e6172e6aa7938454b4a9c57c63a4d491
codesardine commented 2023-06-26 14:08:18 +00:00 (Migrated from gitlab2.manjaro.org)

is this solved?

is this solved?
codesardine commented 2023-06-26 14:51:38 +00:00 (Migrated from gitlab2.manjaro.org)

mentioned in merge request !336

mentioned in merge request !336
codesardine commented 2023-06-26 14:51:54 +00:00 (Migrated from gitlab2.manjaro.org)

mentioned in commit af61fd271e1a1cfefa94243a8403a8be037b072f

mentioned in commit af61fd271e1a1cfefa94243a8403a8be037b072f
codesardine commented 2023-09-25 14:04:49 +00:00 (Migrated from gitlab2.manjaro.org)

No response closing this one

No response closing this one
codesardine (Migrated from gitlab2.manjaro.org) closed this issue 2023-09-25 14:04:49 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
applications/pacman-mirrors#169
No description provided.