Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bears/python: Add ShebangBear #1937

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

bears/python: Add ShebangBear #1937

wants to merge 1 commit into from

Conversation

AMR-KELEG
Copy link
Contributor

@AMR-KELEG AMR-KELEG commented Jul 20, 2017

Closes #1932

For short term contributors: we understand that getting your commits well
defined like we require is a hard task and takes some learning. If you
look to help without wanting to contribute long term there's no need
for you to learn this. Just drop us a message and we'll take care of brushing
up your stuff for merge!

Checklist

  • I read the commit guidelines and I've followed
    them.
  • I ran coala over my code locally. (All commits have to pass
    individually.
    It is not sufficient to have "fixup commits" on your PR,
    our bot will still report the issues for the previous commit.) You will
    likely receive a lot of bot comments and build failures if coala does not
    pass on every single commit!

After you submit your pull request, DO NOT click the 'Update Branch' button.
When asked for a rebase, consult coala.io/rebase
instead.

Please consider helping us by reviewing other peoples pull requests as well:

The more you review, the more your score will grow at coala.io and we will
review your PRs faster!


def run(self, filename, file):
"""
Ensure that the file begins with the Shebang operator.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove indentation here, should be

"""
Ensure that the ...
"""

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok 👍

diff = Diff(file)
mes = ''

if '#!' in file[0]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just check for #!/usr/bin/env

Copy link
Member

@AsnelChristian AsnelChristian Jul 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

having #! in the first line don't necessarily measn that it was intended to be the shebang operator

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you just add the shebangs if #!/usr/bin/env it is not found

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or what do you think?

Copy link
Contributor Author

@AMR-KELEG AMR-KELEG Jul 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The referenced issue requires that coala enforces using the #!usr/bin/env (#1932).
Additionally, I am not familiar with the unix flags so i don't know whether #! can be used for purposes other than the shebang operator.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A shebang basically specifies the shell for the command to call the interpreter (like python, ruby, php, or another UNIX shell). We want to enforce the usage of using /usr/bin/env to get the proper path of the interpreter.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yukiisbored so first of all, the bear should be moved from the python bears directory.
Now, let's discuss the case where the user uses something like #!/usr/bin/python , how can we check that the user doesn't use the general form of the shebang operator and then deduce that script is written in python(or any other language) ?
Do you think that the current check can be improved somehow?
Thanks 😃

Copy link
Member

@yukiisbored yukiisbored Jul 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AMR-KELEG You can just grab the binary name (without the path) and add that at the end of /usr/bin/env .

ex.

  1. #!/usr/bin/python -> #!/usr/bin/env python
  2. #!/usr/bin/python3 -> #!/usr/bin/env python3
  3. #!/usr/bin/asuka --intepreter-mode --no-warnings -> #!/usr/bin/env asuka --intepreter-mode --no-warnings

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I think the fitting category is between shell and general (depends on how you look at it ;))

self,
mes,
diff.affected_code(filename),
diffs={filename: diff})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and tests for this bear

mes = ''

if '#!' in file[0]:
diff.modify_line(1, '#!/usr/bin/env')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't want #!/usr/bin/env as your shebang without any binary right? That's just a directory

Copy link
Member

@yukiisbored yukiisbored Jul 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, afaik env is used to run a program inside a modified environment (from env(1) GNU coreutils).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sils Yes, you're right.
The modified line should include the binary.
#!/usr/bin/env isn't enough.
The modified line should look like: #!/usr/bin/env python.
@yukiisbored What do you think?

Copy link
Member

@yukiisbored yukiisbored Jul 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AMR-KELEG It's ok. Please don't wait for our response just do the changes needed. We're pretty busy with a lot of stuff and we don't want to have this PR delayed just because of us~

return

diff = Diff(file)
mes = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is message, use message or msg or something that is more clear as a message

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

if '#!' in file[0]:
binary = file[0].split('bin/', 1)[1]
diff.modify_line(1, '#!/usr/bin/env {}'.format(binary))
mes = 'Use the /usr/bin/env.'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use 'Use /usr/bin/env {}'.format(binary) instead ?

message = None

if '#!' in file[0]:
binary = file[0].split('bin/', 1)[1].strip('\n')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please use os.path.basename instead of this? Since, it causes problems if it's /usr/sbin/, /sbin/, /usr/local/sbin, etc.

"""
Ensure that the file uses the generic Shebang operator.
"""
if len(file) == 0 or '#!/usr/bin/env' in file[0]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also noticed an empty list is False and I think it's better to do it that way.
https://stackoverflow.com/questions/53513/best-way-to-check-if-a-list-is-empty#53522

Copy link
Member

@yukiisbored yukiisbored Jul 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think .startswith('#!/usr/bin/env') is better...

binary = os.path.basename(file[0]).strip('\n')
new_line = '#!/usr/bin/env {}'.format(binary)
diff.modify_line(1, new_line)
message = 'Use the {}.'.format(repr(new_line))
Copy link
Member

@yukiisbored yukiisbored Jul 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this message to be unclear especially for users who don't know what is /usr/bin/env. Any ideas? I'm not sure how to phrase it as well.

Copy link
Member

@yukiisbored yukiisbored Jul 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Use /usr/bin/env to call the program" and then a more info "/usr/bin/env is preferable to be used because blah blah blah ..." is better I think...

diff = Diff(file)
message = None

if '#!' in file[0]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return

diff = Diff(file)
message = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find both of the above to be tricky when reading it. I think it's better to put it inside the if statement below and just yield it inside it as well.

Copy link
Member

@yukiisbored yukiisbored Jul 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically,

if "check if it starts with a shebang":
    diff = Diff(file)
    ...
    message = "message goes here"
    ...
    yield ...

@yukiisbored
Copy link
Member

Also, Don't forget to squash your commits. Since this is only one "change".

@yash-nisar
Copy link
Member

Please address @yukiisbored 's comments after which we can have another iteration.

@AMR-KELEG
Copy link
Contributor Author

@yukiisbored @yash-nisar I know that my last commit was (a long long) time ago but could you please review the pull request again?

@sils
Copy link
Member

sils commented Dec 7, 2017 via email

@gitmate-bot
Copy link
Collaborator

Hey! This pull request is being rebased automatically. Please DO NOT push while rebase is in progress or your changes would be lost!

@gitmate-bot
Copy link
Collaborator

Automated rebase was successful!

@AMR-KELEG
Copy link
Contributor Author

@Makman2 Here we go 😄

Ensure that the file uses the generic Shebang operator.
"""
if not file or file[0].startswith('#!/usr/bin/env'):
return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should also return a result if the file is empty, saying there's no shebang?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe without a patch, but at least saying it's missing ;)
(oh would be so cool if we would have custom result actions, then we could write one where the user can input immediately the name of the binary... but we don't have it yet 😢)

Ensure that the file uses the generic Shebang operator.
"""
if not file or file[0].startswith('#!/usr/bin/env'):
return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also add a newline here? I think it improves readability 👍

diff = Diff(file)
message = None
binary = os.path.basename(file[0]).strip('\n')
new_line = '#!/usr/bin/env {}'.format(binary)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are using #!/usr/bin/env quite often, I think it's worth to put it inside a variable as a constant 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is only written twice here..
Putting it inside a variable would be only beneficial if we will need to change it later..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's also for consistency 👍
Everything that's used more than once shouldn't be inlined ;)

if file[0].startswith('#!'):
diff = Diff(file)
message = None
binary = os.path.basename(file[0]).strip('\n')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't you have to strip off the #! part in file[0]?
So in the end you have file[0][2:]?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if i understood your comment but the basename extracts the binary file out of the path.
https://docs.python.org/2/library/os.path.html#os.path.basename
Why would i strip the #! if i don't care what the first part of the path is?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but is os.path.basename able to handle it? As the path starts after #!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/python/cpython/blob/3.6/Lib/posixpath.py#L142
The method just extracts all the characters after the last separator.
IMO, we shouldn't strip off the #!.
What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those characters don't belong to the path, that's why I'm asking for it, as conceptually it's not "completely right". I would do file[0][2:] for the sake of comprehensibility ;)

diff.modify_line(1, new_line)
message_desc = ('This eliminates the limitations caused by systems'
' that have non-standard file system layout')
message = 'Use the {}.\n{}.'.format(repr(new_line), message_desc)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I'm not that convinced of the message, but can't think of a better one right now... we'll discuss it again ;)

diff.modify_line(1, new_line)
message_desc = ('This eliminates the limitations caused by systems'
' that have non-standard file system layout')
message = 'Use the {}.\n{}.'.format(repr(new_line), message_desc)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But please inline message_desc, I think it's just an unnecessary extra variable ;) Then you don't have to use two placeholders ;)


def run(self, filename, file):
"""
Ensure that the file uses the generic Shebang operator.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can be precise and describe the idea of using /usr/bin/env ;)

message_desc),
line=1,
severity=RESULT_SEVERITY.NORMAL,
file=get_testfile_path(file_name))],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also assert for the diff? :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Makman2 could you please give me a hint?
Thanks 😄

Copy link
Member

@Makman2 Makman2 Dec 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just use the diff param in the result here 👍

self.check_validity(self.uut,
['#!/usr/bin/env python \n print hello world'])
self.check_validity(self.uut,
[''])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think an empty file should be written as an empty list, that's how python handles it 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you should put those into test files too? Up to you :)

@Makman2
Copy link
Member

Makman2 commented Dec 16, 2017

The __init__.py inside the test files directory is not necessary, as you don't place python modules there for coala use, they are just test files ;)

AUTHORS = {'The coala developers'}
AUTHORS_EMAILS = {'[email protected]'}
LICENSE = 'AGPL-3.0'
CAN_FIX = {'Syntax'}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm not sure if it's syntax... do we have a better fitting keyword?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting?
I think syntax is the better choice.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You aren't correcting syntax errors... hmm... Could be that you have to create a new keyword, maybe Shebang itself works?
You could maybe get some ideas from the community ;)

@@ -0,0 +1,33 @@
import os
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a newline here to separate system imports from others ;)

@Makman2
Copy link
Member

Makman2 commented Dec 16, 2017

unack 90a7bd6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Bear Proposal: ShebangBear
10 participants