[buildiso] generate log

philm commented

2015-01-27 15:59:24 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Redirect build log to log if it is either not run in terminal, or copy terminal output to log file.

*Created by: udeved* Redirect build log to log if it is either not run in terminal, or copy terminal output to log file.

philm commented

2015-02-17 19:34:19 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

If anybody know a good approach to log the build process, please say so.

I have added some time ago a logging function in util-iso and experimented with it, but I didn't like the results so far.

*Created by: udeved* If anybody know a good approach to log the build process, please say so. I have added some time ago a logging function in util-iso and experimented with it, but I didn't like the results so far.

philm commented

2015-03-02 10:26:33 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

I have enabled logging github.com/manjaro/manjaro-tools@b57b45ed2e

Its still experimental

*Created by: udeved* I have enabled logging https://github.com/manjaro/manjaro-tools/commit/b57b45ed2ef5b48d94041963f0acd0be19d1525e Its still experimental

philm commented

2015-03-02 18:14:10 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Ok, will test it out.

*Created by: philmmanjaro* Ok, will test it out.

philm commented

2015-03-02 18:39:43 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

I haven't figured out yet how to keep the fd in terminal while logging.

*Created by: udeved* I haven't figured out yet how to keep the fd in terminal while logging.

philm commented

2015-03-08 13:20:23 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

what do you mean by fd?

*Created by: philmmanjaro* what do you mean by fd?

philm commented

2015-03-08 13:46:57 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

fd=file descriptor, like stdout or stderr

*Created by: udeved* fd=file descriptor, like stdout or stderr

philm commented

2015-03-08 15:35:44 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

ah, ok. did you check how makepkg does it? They do logging and terminal output also ...

*Created by: philmmanjaro* ah, ok. did you check how makepkg does it? They do logging and terminal output also ...

philm commented

2015-03-08 15:39:11 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Does this help in any way?

*Created by: philmmanjaro* Does [this](http://unix.stackexchange.com/questions/13724/file-descriptors-shell-scripting) help in any way?

philm commented

2015-03-08 16:47:22 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Well, I basically used the logger part of makepkg.
When I implemented it in buildiso, it produced strange results in the log file as well as missing stderr in terminal.
I now moved it to mkiso, but I haven't tested it yet.

*Created by: udeved* Well, I basically used the logger part of makepkg. When I implemented it in buildiso, it produced strange results in the log file as well as missing stderr in terminal. I now moved it to mkiso, but I haven't tested it yet.

philm commented

2015-03-08 17:40:27 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Well, we can push it to 0.9.7.x and try to do it later or so ... Atm, I've no clue how to help here ...

*Created by: philmmanjaro* Well, we can push it to 0.9.7.x and try to do it later or so ... Atm, I've no clue how to help here ...

philm commented

2015-03-08 18:52:51 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

I agree, its why I already disabled the log switch again.

To test, the mkiso args would have to add a -Z arg to the args array.

*Created by: udeved* I agree, its why I already disabled the log switch again. To test, the mkiso args would have to add a -Z arg to the args array.

philm commented

2016-02-14 00:21:47 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

The only ugly thing with the logger enabled is ugly lines in the log file.
Perhaps, the entire msg calls need to be checked and perhaps %s to be used.

*Created by: udeved* The only ugly thing with the logger enabled is ugly lines in the log file. Perhaps, the entire msg calls need to be checked and perhaps %s to be used.

philm commented

2016-02-14 08:27:59 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

What do you mean with ugly lines? There are tools and one-liners to clean up these. Here are some examples:

Skript gestartet auf So 14 Feb 2016 09:21:14 CET
]0;phil@manjaro:/home/phil[01;32m[phil@manjaro[01;37m ~[01;32m]$[00m ls dir
ls: Zugriff auf 'dir' nicht möglich: Datei oder Verzeichnis nicht gefunden
]0;phil@manjaro:/home/phil[01;32m[phil@manjaro[01;37m ~[01;32m]$[00m ls *

To clean that up you can use this:

cat test.log  | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > test-cleaned.log

Result is then:

Skript gestartet auf So 14 Feb 2016 09:21:14 CET
[phil@manjaro ~]$ ls dir
ls: Zugriff auf 'dir' nicht möglich: Datei oder Verzeichnis nicht gefunden
[phil@manjaro ~]$ ls *

*Created by: philmmanjaro* What do you mean with ugly lines? There are tools and one-liners to clean up these. Here are some examples: ``` Skript gestartet auf So 14 Feb 2016 09:21:14 CET ]0;phil@manjaro:/home/phil[01;32m[phil@manjaro[01;37m ~[01;32m]$[00m ls dir ls: Zugriff auf 'dir' nicht möglich: Datei oder Verzeichnis nicht gefunden ]0;phil@manjaro:/home/phil[01;32m[phil@manjaro[01;37m ~[01;32m]$[00m ls * ``` To clean that up you can use this: ``` cat test.log | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > test-cleaned.log ``` Result is then: ``` Skript gestartet auf So 14 Feb 2016 09:21:14 CET [phil@manjaro ~]$ ls dir ls: Zugriff auf 'dir' nicht möglich: Datei oder Verzeichnis nicht gefunden [phil@manjaro ~]$ ls * ```

philm commented

2016-02-14 08:37:11 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Also there is an application for it if needed: typescript2txt

*Created by: philmmanjaro* Also there is an application for it if needed: [typescript2txt](https://github.com/RadixSeven/typescript2txt)

philm commented

2016-02-14 10:17:20 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

The log has screwed up the arrows that are printed.
It prints them as escape sequences, and I don't think we should "fix" this by filtering the stuff.
The log should be written correctly already.

*Created by: udeved* The log has screwed up the arrows that are printed. It prints them as escape sequences, and I don't think we should "fix" this by filtering the stuff. The log should be written correctly already.

philm commented

2016-02-14 10:34:49 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Well, it always depends which interpreter you're using. Gimme an example so I can take a look at it.

*Created by: philmmanjaro* Well, it always depends which interpreter you're using. Gimme an example so I can take a look at it.

philm commented

2016-02-14 10:40:08 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

My guess is that its caused by wrong encoding.

*Created by: udeved* My guess is that its caused by wrong encoding.

philm commented

2016-02-14 11:27:36 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Well, we pass color codes and such. Try to use col to see if it will be better using this interpreter. You can also try my filters to see if that helps.

*Created by: philmmanjaro* Well, we pass color codes and such. Try to use **col** to see if it will be better using this interpreter. You can also try my filters to see if that helps.

philm commented

2016-02-23 19:21:50 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

It should work without any filters, it does for makepkg.

I investigated, its only an issue if colors are enabled, if disabled, the logfiles are fine and readable.
Needs further digging ...

*Created by: udeved* It should work without any filters, it does for makepkg. I investigated, its only an issue if colors are enabled, if disabled, the logfiles are fine and readable. Needs further digging ...

philm commented

2016-02-23 19:24:25 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

It is the color code which most likely messes it up. The filters deal with them ...

*Created by: philmmanjaro* It is the color code which most likely messes it up. The filters deal with them ...

philm commented

2016-02-23 19:25:59 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Things is, I would like to understand what's going on there.
As fr as I see, the makepkg code does nothing special to filter.

*Created by: udeved* Things is, I would like to understand what's going on there. As fr as I see, the makepkg code does nothing special to filter.

philm commented

2016-02-23 19:34:07 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Also what I notice are the progress bars for example in download procedure. Ask @Kirek for how he solved it in manjaro-system-settings:

:: Retrieving packages ...
downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz...
downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz...
downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz...
downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz...
downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz...
downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz...
downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz...
downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz...
downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz...
downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz...
downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz...
downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz...
downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz...
downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz...
downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz...
downloading udiskie-1.4.8-1-any.pkg.tar.xz...
downloading udiskie-1.4.8-1-any.pkg.tar.xz...
downloading udiskie-1.4.8-1-any.pkg.tar.xz...
downloading udiskie-1.4.8-1-any.pkg.tar.xz...
downloading udiskie-1.4.8-1-any.pkg.tar.xz...
checking keyring...
checking package integrity...

*Created by: philmmanjaro* Also what I notice are the progress bars for example in download procedure. Ask @Kirek for how he solved it in **manjaro-system-settings**: ``` :: Retrieving packages ... downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz... downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz... downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz... downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz... downloading sonar-mate-settings-20160222-1-any.pkg.tar.xz... downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz... downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz... downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz... downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz... downloading packagekit-1.1.0-1.1-x86_64.pkg.tar.xz... downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz... downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz... downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz... downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz... downloading remmina-1:1.2.0rcgit.8-1-x86_64.pkg.tar.xz... downloading udiskie-1.4.8-1-any.pkg.tar.xz... downloading udiskie-1.4.8-1-any.pkg.tar.xz... downloading udiskie-1.4.8-1-any.pkg.tar.xz... downloading udiskie-1.4.8-1-any.pkg.tar.xz... downloading udiskie-1.4.8-1-any.pkg.tar.xz... checking keyring... checking package integrity... ```

philm commented

2016-02-23 19:39:06 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

The "missing" stuff is explainable with redirection of file descriptor.
I think its ok that the log file does not contain progress bars.

But I don't understand why the color stuff screws up the log output.

*Created by: udeved* The "missing" stuff is explainable with redirection of file descriptor. I think its ok that the log file does not contain progress bars. But I don't understand why the color stuff screws up the log output.

philm commented

2016-02-23 19:45:44 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Another issue with this logger is that if we have a dbus workaround pkill it fails and complains that something went wrong. However this happens also without the logging. What changed?

(253/253) reinstalling zfs-utils                   [######################] 100%
 --> Killing chroot process: dbus-launch (28243)
 --> Killing chroot process: dbus-daemon (28244)
Generating locales...
Generation complete.
  -> Copying mate-overlay ...
==> Configuring [mate-image]
  -> Setting plymouth sonar-elegant ....
  -> Configuring Displaymanager ...
 --> Loading [/usr/share/manjaro-tools/desktop.map] ...
  -> Trying to detect desktop environment ...
  -> Detected: mate
  -> Configuring AccountsService ...
  -> Configured: gdm
 --> Configuring [systemd] ....
  -> Setting bluetooth ...
  -> Setting cronie ...
  -> Setting ModemManager ...
==> ERROR: A failure occurred in /var/lib/manjaro-tools/buildiso/mate-sonar/x86_64/mate-image().
    Aborting...

*Created by: philmmanjaro* Another issue with this logger is that if we have a **dbus workaround pkill** it fails and complains that something went wrong. However this happens also without the logging. What changed? ``` (253/253) reinstalling zfs-utils [######################] 100% --> Killing chroot process: dbus-launch (28243) --> Killing chroot process: dbus-daemon (28244) Generating locales... Generation complete. -> Copying mate-overlay ... ==> Configuring [mate-image] -> Setting plymouth sonar-elegant .... -> Configuring Displaymanager ... --> Loading [/usr/share/manjaro-tools/desktop.map] ... -> Trying to detect desktop environment ... -> Detected: mate -> Configuring AccountsService ... -> Configured: gdm --> Configuring [systemd] .... -> Setting bluetooth ... -> Setting cronie ... -> Setting ModemManager ... ==> ERROR: A failure occurred in /var/lib/manjaro-tools/buildiso/mate-sonar/x86_64/mate-image(). Aborting... ```

philm commented

2016-02-23 19:57:27 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Hmm, seems not to be related with pkill. KDE-minimal profile stopps for me also. Will test an older snapshot now ...

*Created by: philmmanjaro* Hmm, seems not to be related with pkill. KDE-minimal profile stopps for me also. Will test an older snapshot now ...

philm commented

2016-02-23 20:03:03 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Its to do with the wrapper function, I think.
Introduced with logger, "run_safe" in util-iso-log

*Created by: udeved* Its to do with the wrapper function, I think. Introduced with logger, "run_safe" in util-iso-log

philm commented

2016-02-24 19:20:17 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

[1;1m[1;32m==>[1;0m[1;1m To import the data required by pacman for package verification run:[1;0m
[1;1m[1;32m==>[1;0m[1;1m `pacman-key --init; pacman-key --populate archlinux manjaro`[1;0m
[1;1m[1;32m==>[1;0m[1;1m See: https://www.archlinux.org/news/having-pacman-verify-packages[1;0m

installing mhwd...
Using default
Default lib32 support: true
libGl and libglx symlinks are invalid!
Falling back to default...
libGl: '/usr/lib/mesa/libGL.so.1.2.0'
libGLESv1: '/usr/lib/mesa/libGLESv1_CM.so.1.1.0'
libGLESv2: '/usr/lib/mesa/libGLESv2.so.2.0.0'
libEGL: '/usr/lib/mesa/libEGL.so.1.0.0'
warning: could not find '/usr/lib/xorg/modules/extensions/libglx.xorg'!
[1;1m[1;32m==>[1;0m[1;1m use mhwd-gpu to set catalyst as default: 'mhwd-gpu --setgl catalyst'[1;0m
[1;1m[1;32m==>[1;0m[1;1m use mhwd-gpu to set mesa as default: 'mhwd-gpu --setgl mesa'[1;0m

Same for colored pacman .install scriptlets.

*Created by: udeved* ``` [1;1m[1;32m==>[1;0m[1;1m To import the data required by pacman for package verification run:[1;0m [1;1m[1;32m==>[1;0m[1;1m `pacman-key --init; pacman-key --populate archlinux manjaro`[1;0m [1;1m[1;32m==>[1;0m[1;1m See: https://www.archlinux.org/news/having-pacman-verify-packages[1;0m ``` ``` installing mhwd... Using default Default lib32 support: true libGl and libglx symlinks are invalid! Falling back to default... libGl: '/usr/lib/mesa/libGL.so.1.2.0' libGLESv1: '/usr/lib/mesa/libGLESv1_CM.so.1.1.0' libGLESv2: '/usr/lib/mesa/libGLESv2.so.2.0.0' libEGL: '/usr/lib/mesa/libEGL.so.1.0.0' warning: could not find '/usr/lib/xorg/modules/extensions/libglx.xorg'! [1;1m[1;32m==>[1;0m[1;1m use mhwd-gpu to set catalyst as default: 'mhwd-gpu --setgl catalyst'[1;0m [1;1m[1;32m==>[1;0m[1;1m use mhwd-gpu to set mesa as default: 'mhwd-gpu --setgl mesa'[1;0m ``` Same for colored pacman .install scriptlets.

philm commented

2016-02-24 20:05:37 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Simply use the code I posted to filter it out: perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b

*Created by: philmmanjaro* Simply use the code I posted to filter it out: `perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b`

philm commented

2016-02-24 20:09:06 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: philmmanjaro

Check this out: http://ubuntuforums.org/showthread.php?t=2231058

*Created by: philmmanjaro* Check this out: http://ubuntuforums.org/showthread.php?t=2231058

philm commented

2016-02-24 20:09:37 +00:00

(Migrated from gitlab2.manjaro.org)

Created by: udeved

Won't filter any ansi sequence.
I tried.

Its connected with terminal and fd2 where msg get redirected.
Inn case of the PKGBUILD install files, I think it can be solved with...

[[ -t 2 ]] && do_something

we could also source /usr/share/makepkg/util/messages,sh

Already been to ubuntu and everywhere.

*Created by: udeved* Won't filter any ansi sequence. I tried. Its connected with terminal and fd2 where msg get redirected. Inn case of the PKGBUILD install files, I think it can be solved with... ``` [[ -t 2 ]] && do_something ``` we could also source /usr/share/makepkg/util/messages,sh Already been to ubuntu and everywhere.

Rows
Columns

[buildiso] generate log #45