Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle EPIPE error in sample.py #27

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Handle EPIPE error in sample.py
Previously, the program would raise an exception if piped to a program that closes the pipe before all the data is written. For example:

    cat data.txt | sample.py 1/10 | head -n 1
maxhawkins committed Sep 13, 2015
commit 049147829cb4e95e0c2244c7b1c31858abaae8c0
9 changes: 8 additions & 1 deletion data_hacks/sample.py
Original file line number Diff line number Diff line change
@@ -20,6 +20,7 @@
https://github.com/bitly/data_hacks
"""

import errno
import sys
import random
from optparse import OptionParser
@@ -29,7 +30,13 @@ def run(sample_rate):
input_stream = sys.stdin
for line in input_stream:
if random.randint(1,100) <= sample_rate:
sys.stdout.write(line)
try:
sys.stdout.write(line)
except IOError, e:
if e.errno == errno.EPIPE:
return
else:
raise

def get_sample_rate(rate_string):
""" return a rate as a percentage"""