How to simulate shell pipes with the subprocess module

• Posted 13 January 2024
• Tagged with python

I had a shell command featuring a pipe that I wanted to replicate with subprocess:

youtube-dl --get-id "$PLAYLIST_URL" \
  | xargs -I '{}' -P 5 youtube-dl 'https://youtube.com/watch?v={}'

I could try to create this command as a string, pass it to subprocess.call(…, shell=True), and I hope I’ve used shlex.quote() correctly – but that’s dangerous and error-prone.

I found a better approach in a Stack Overflow answer by Taymon: use subprocess.PIPE to pass stdout/stdin between processes.

get_ids_proc = subprocess.Popen(
    ["youtube-dl", "--get-id", youtube_url],
    stdout=subprocess.PIPE
)

subprocess.check_call(
    ["xargs", "-I", "{}", "-P", "5", "youtube-dl", "https://youtube.com/watch?v={}"],
    stdin=get_ids_proc.stdout,
)

get_ids_proc.wait()