capture stdout and stderr

[Thomas Danckaert]

Hi Guilers,

I want to run an external process, and capture both its stdout and
stderr output streams. I didn't find an easy way to get a port for
an external process' stderr, other than wrapping the call to the
process in "with-error-to-port".

My questions are:
0) Did I miss something obvious? :)
1) why doesn't the second approach in the code below work (the
output-string is always empty)?
2) are there any other (better) solutions?
3) I suppose my approach with (pipe) might block if the process
writes too much data to stderr before I read from the pipe. Should I
use "select" to interleave reads from stdout and stderr?

thank you!

Thomas

(use-modules (ice-9 format)
(ice-9 popen)
(ice-9 textual-ports))

(define (run-command)
(let ((process (open-input-pipe "./stdoutstderr.sh")))
(format #t "stdout: '~a'~%" (get-string-all process))
(close-pipe process)))

;; Works:
(let ((err-pipe (pipe)))
(with-error-to-port (cdr err-pipe) run-command)

(close-port (cdr err-pipe))
(format #t "stderr: '~a' ~%" (get-string-all (car err-pipe))))

;; Doesn't work: (get-output-string) returns empty string.
(let ((err-port (open-output-string)))
(with-error-to-port err-port run-command)

(format #t "stderr: '~a'~%" (get-output-string err-port))
(close-output-port err-port))

[Catonano]

Post by Thomas Danckaert
Hi Guilers,
I want to run an external process, and capture both its stdout and stderr
output streams. I didn't find an easy way to get a port for an external
process' stderr, other than wrapping the call to the process in
"with-error-to-port".
0) Did I miss something obvious? :)
1) why doesn't the second approach in the code below work (the
output-string is always empty)?
2) are there any other (better) solutions?
3) I suppose my approach with (pipe) might block if the process writes
too much data to stderr before I read from the pipe. Should I use "select"
to interleave reads from stdout and stderr?
thank you!
Thomas
(use-modules (ice-9 format)
(ice-9 popen)
(ice-9 textual-ports))
(define (run-command)
(let ((process (open-input-pipe "./stdoutstderr.sh")))
(format #t "stdout: '~a'~%" (get-string-all process))
(close-pipe process)))
(let ((err-pipe (pipe)))
(with-error-to-port (cdr err-pipe) run-command)
(close-port (cdr err-pipe))
(format #t "stderr: '~a' ~%" (get-string-all (car err-pipe))))
;; Doesn't work: (get-output-string) returns empty string.
(let ((err-port (open-output-string)))
(with-error-to-port err-port run-command)
(format #t "stderr: '~a'~%" (get-output-string err-port))
(close-output-port err-port))

Hi Thomas

I'm the most incompetent guile user on earth and yet I want to venture into
offering a couple of observations

The first observation is that in the first case, you close the port
cotaining the error _prior_ to attempting reading from the port

In the second case, you attempt reading from the port AND THEN you close
the port

I undertsand that the writing on ports is mediated by some form of
buffering, or something, and closing the port also "flushes" the port,
meaning that if there are some remnants to be written yet, they get written

I don't even know of with-error-to-port automatically closes the port so
that manually closing it shouldn't be necessary

If with-error-to-port closes the port, then my observations doesn't apply

I run into this misundertsanding not a long time ago and I wrote about it
on Mastodon, here
https://mastodon.social/@catonano/99979214792830581

The second observation is that you could try to see forr yourself by
expanding your calls and visually inspect the expanded versions

They should differ in that you should be able to see the closing of the
port (I never checed this out)

Ok I rambled enough

Hope this helps !

Ciao

[Thomas Danckaert]

From: Catonano <***@gmail.com>
Subject: Re: capture stdout and stderr
Date: Sat, 26 May 2018 12:24:11 +0200

Post by Catonano
The first observation is that in the first case, you close the port
cotaining
the error _prior_ to attempting reading from the port
In the second case, you attempt reading from the port AND THEN you
close the
port

Yes, I also found that a bit surprising when I wrote that. However,
with a string-port, you have to call get-output-string *before*
closing the port, so that's why it's written like that
(https://www.gnu.org/software/guile/manual/html_node/String-Ports.html).

Post by Catonano
I undertsand that the writing on ports is mediated by some form of
buffering,
or something, and closing the port also "flushes" the port, meaning
that if
there are some remnants to be written yet, they get written
I don't even know of with-error-to-port automatically closes the
port so that
manually closing it shouldn't be necessary

My reason to close the port was in order to have an EOF at the end of
the error output, so I can simply use get-string-all to retrieve it.
But yes, otherwise buffering might also come into play. I think I
checked that with-error-to-port didn't close the port for me. I'm
now leaving on a short vacation, so don't have time to check again in
more detail, but it's a good suggestion.

Thanks!

Thomas

[Thomas Danckaert]

From: Stefan Israelsson Tampe <***@gmail.com>
Subject: Re: capture stdout and stderr
Date: Thu, 24 May 2018 15:25:51 +0200

I think that your code whould work but doesn't because string ports
are not file ports and system commands pipes out to file ports.
'open-input-pipe' uses 'scm_open_process' that returns only a
read-port and a write-port, the write port is then taped the output
string in your run-command.
and it works.
ALso when you create a err-port with pipe, you create a file port
nad this will be used translated as an error port for the system
comand and it all works.
However the failing program uses a string port which is not a file
port and it does not end up capturing the error output in the
system command.
One would wish that guile if the error port is not a file port make
a file port, as done with 'pipe' and create a thread that pipes the
data from the file port to the non file port and your code
would work. Also similarly behavior for read and write ports
towards the shell.
the relevant code is in posix.c, scm_open_process

Hi! Thanks for the message. Indeed, that makes sense. In fact I had
some vague intuition that some magic would be needed to channel the
stderr output into such a string port. I think the manual could be
more explicit about that, though? (e.g. in the section on default
ports
https://www.gnu.org/software/guile/manual/html_node/Default-Ports.html#Default-Ports
). Maybe I should propose a patch to the docs, stating that those
"with-x-to-port" methods only work on file ports.

Thomas

[Catonano]

Post by Thomas Danckaert
Subject: Re: capture stdout and stderr
Date: Thu, 24 May 2018 15:25:51 +0200
I think that your code whould work but doesn't because string ports are

not file ports and system commands pipes out to file ports.
'open-input-pipe' uses 'scm_open_process' that returns only a read-port
and a write-port, the write port is then taped the output string in your
run-command.
and it works.
ALso when you create a err-port with pipe, you create a file port nad
this will be used translated as an error port for the system comand and it
all works.
However the failing program uses a string port which is not a file port
and it does not end up capturing the error output in the system command.
One would wish that guile if the error port is not a file port make a
file port, as done with 'pipe' and create a thread that pipes the data from
the file port to the non file port and your code
would work. Also similarly behavior for read and write ports towards the
shell.
the relevant code is in posix.c, scm_open_process

Hi! Thanks for the message. Indeed, that makes sense. In fact I had some
vague intuition that some magic would be needed to channel the stderr
output into such a string port. I think the manual could be more explicit
about that, though? (e.g. in the section on default ports
https://www.gnu.org/software/guile/manual/html_node/Default-
Ports.html#Default-Ports ). Maybe I should propose a patch to the docs,
stating that those "with-x-to-port" methods only work on file ports.
Thomas

Thomas

thank you for sharing the solution