Realtime scp output with ruby

September 07, 2012

I have had an annoying problem recently with a shell script where it would not show the output of an scp command in realtime. Initially I was fighting with different methods of capturing output in realtime from a ruby script.

The solution to this which I went for is using IO.popen


def stdout_redirect(command)
    f = IO.popen(command)
    while line = f.gets
      puts line

However, this was only part of the solution. It wasnt working so I even tried implementing a solution in python.

import subprocess

  def myrun(cmd):

      print "Running " + cmd

      p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
      stdout = []
      while True:
          line = p.stdout.readline()
          print line,
          if line == '' and p.poll() != None:
      return ''.join(stdout)

  myrun('scp -r "remote_server:/directory/test_copy" ~/test_files/')

To my surprise, all was quiet so had a check in the terminal and it would output fine. Hmmm.

file1.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file2.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file3.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file4.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file5.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00    
  file6.txt                                                                                                                                                                         100%    0     0.0KB/s   00:00

After some time with mr google, I managed to track down the issue with scp itself. It runs the isatty() function to check and see if the command is being run in a shell. If it is not, then all is quiet! This means ruby, python et al do not get scp output.

The solution thanks to stack exchange was to send all output to a shell!


scp -r "remote_server:/directory/test_copy" ~/test_files/ > /dev/tty

Hope this helps you if you are having a similar issue!