Add smbclient for an easy high level API to manage SMB shares (jborean93#17)

* Added higher level client API

* Fixes after recent pull rebase

* remove utils mention now we use conftest

* Add default timeout to connection

* try and reduce deadlocks in the msg queue processor

* Fix deadlock issue and ensure tests are passing locally

* Try and smooth out transport error handling

* Fix up transport test after change

* Dropped support for Python 3.4

* Added the ability to register and close a session in the pool

* Fix up test and typos

* More typos and minor nit changes

* Set Impersonation as the default Impersonation level

* Fix up listdir and read for pipes

* Add various shutil methods (jborean93#16)

* Add various shutil methods

* reworked rmtree, review comments addressed

* renamed copy2 -> copy and copytree -> copytree_copy, copying stats removed

* service side copy moved to _os

* fix pep8 indentation

* review comments in shutil addressed

* Fix up exceptions that can have unicode chars

* Use open_file for copyfile

* Move copyfile to top of file and add more tests

* Use set size for SRV_REQUEST_RESUMT_KEY response output

* Expand shutil and added more tests

* Attempt to get Appveyor working again

* Hopefully the last Appveyor fixes to go in

* Should be last fix for appveyor

* Last Appveyor fixes

* Simplify stat and add ads tests

* Remove ordereddict requirement

Author: jborean93

.travis.yml

@@ -4,16 +4,12 @@ language: python
 
 dist: bionic
 
-matrix:
-  include:
-  - python: 2.7
-  - python: 3.4
-    dist: trusty
-  - python: 3.5
-    dist: trusty
-  - python: 3.6
-  - python: 3.7
-  - python: 3.8
+python:
+- 2.7
+- 3.5
+- 3.6
+- 3.7
+- 3.8
 
 services:
 - docker
@@ -36,7 +32,7 @@ install:
 - python ./build-scripts/check_samba.py
 
 script:
-- py.test -v --pep8 --cov smbprotocol --cov-report term-missing
+- py.test -v --pep8 --cov smbclient --cov smbprotocol --cov-report term-missing
 
 after_success:
 - coveralls

CHANGELOG.md

@@ -1,12 +1,15 @@
 # Changelog
 
-## 0.3.0 - TBD
+## 1.0.0 - TBD
 
-* Drop support for Python 2.6
+* Dropped support for Python 2.6 and Python 3.4
+* Added the `smbclient` package that provides a higher level API for interactive with SMB servers
+* Deprecated `smbprotocol.query_info` in favour of `smbprotocol.file_info`, `query_info` will be removed in the next major release
+* Add automatic symlink resolver when a symlink is in the path being opened
 * Fix issue when trying to connect to host with IPv6 address
 * Fix response parsing for SMB2 Create Response Lease V1 and V2
 * Added the ability to set the Oplock level when opening a file
-* Revamped the socket listener and message processor to run in a separate thread for better concurrency
+* Revamped the socket listener and message processor to run in a separate thread for faster message resolving
 * Added the `FileSystemWatcher` in `change_notify.py` to provider a way to watch for changes on the SMB filesystem
 * Added the `.cancel()` method onto a Request to cancel an SMB request on the server
 

README.md

@@ -120,13 +120,60 @@ you to set a minimum dialect version if required.
 
 ## Examples
 
-Currently the existing classes expose a very low level interface to the SMB
-protocol which can make things quite complex for people starting to use this
-package. I do plan on making a high-level interface to make things easier for
-users but that's in the backlog.
+There are 2 different APIs you can use with this library.0
 
-For now, the `examples` folder contains some examples of how this package can
-be used.
+* `smbprotocol`: Low level interface that can do whatever you want but quite verbose
+* `smbclient`: Higher level interface that implements the builtin `os` and `os.path` file system functions but for SMB support
+
+The `examples` folder contains some examples of both the high and low level
+interface but for everyday user's it is recommended to use `smbclient` as it
+is a lot simpler.
+
+### smbclient Interface
+
+The higher level interface `smbclient` is designed to make this library easier
+for people to use for simple and common use cases. It is designed to replicate
+the builtin `os` and `os.path` filesystem functions like `os.open()`,
+`os.stat()`, and `os.path.exists()`.
+
+A connection made by `smbclient` is kept in a pool and re-used for future
+requests to the same server until the Python process exists. This makes
+authentication simple and only required for the first call to the server.
+
+You can specify the credentials and other connection parameters on each
+`smbclient` function or register a server with credentials with the following
+kwargs:
+
+* `username`: The username used to connect to the share
+* `password`: The password used to connect to the share
+* `port`: Override the default port (`445`) to connect to
+* `encrypt`: Whether to force encryption on the connection, requires SMBv3 or newer on the remote server (default: `False`)
+* `connection_timeout`: Override the connection timeout in seconds (default: `60`)
+
+If using Kerberos authentication and a Kerberos ticket has already set by
+`kinit` then `smbclient` will automatically use those credentials without
+having to be explicitly set. If no ticket has been retrieved or you wish to use
+different credentials then only the first request for the server in question
+requires the `username` and `password` kwargs.
+
+For example I only need to set the credentials on the first request to create
+the directory and not for the subsequent file creation in that dir.
+
+```
+import smbclient
+
+# Optional - register the credentials with a server
+smbclient.register_session("server", username="user", password="pass")
+
+smbclient.mkdir(r"\\server\share\directory", username="user", password="pass")
+
+with smbclient.open_file(r"\\server\share\directory\file.txt", mode="w") as fd:
+    fd.write(u"file contents")
+```
+
+If you wish to reset the cache you can either start a new Python process or
+call `smbclient.reset_connection_cache()` to close all the connections that
+have been cached by the client.
 
 
 ## Logging
@@ -209,5 +256,4 @@ if you want to implement them yourself;
 
 * Test and support DFS mounts and not just server shares
 * Multiple channel support to speed up large data transfers
-* Create an easier API on top of the `raw` SMB calls that currently exist
 * Lots and lots more...

appveyor.yml

@@ -14,8 +14,6 @@ environment:
   # https://www.appveyor.com/docs/installed-software/#python
   - PYTHON: Python27
   - PYTHON: Python27-x64
-  - PYTHON: Python34
-  - PYTHON: Python34-x64
   - PYTHON: Python35
   - PYTHON: Python35-x64
   - PYTHON: Python36
@@ -56,4 +54,4 @@ install:
 build: off  # Do not run MSBuild, build stuff at install step
 
 test_script:
-- cmd: py.test -v --pep8 --cov smbprotocol --cov-report term-missing
+- cmd: py.test -v --pep8 --cov smbclient --cov smbprotocol --cov-report term-missing

build-scripts/Vagrantfile

@@ -0,0 +1,15 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "centos/7"
+  config.vm.network "forwarded_port", guest: 445, host: 445
+  config.vm.provider "virtualbox" do |vb|
+    vb.gui = false
+    vb.memory = "1024"
+  end
+
+  config.vm.provision "ansible" do |a|
+    a.playbook = "main.yml"
+  end
+end

build-scripts/main.yml

@@ -0,0 +1,73 @@
+---
+- hosts: all
+  gather_facts: no
+  become: yes
+  tasks:
+  - name: install samba
+    package:
+      name: samba
+      state: present
+
+  - name: template out smb.conf file
+    copy:
+      content: |
+        [global]
+        workgroup = WORKGROUP
+        valid users = @smbgroup
+        server signing = mandatory
+        ea support = yes
+        store dos attributes = yes
+        vfs objects = xattr_tdb streams_xattr
+
+        [share]
+        comment = Test Samba Share
+        path = /srv/samba/share
+        browsable = yes
+        guest ok = no
+        read only = no
+        create mask = 0755
+
+        [share-encrypted]
+        command = Test Encrypted Samba Share
+        path = /srv/samba/share-encrypted
+        browsable = yes
+        guest ok = no
+        read only = no
+        create mask = 0755
+        smb encrypt = required
+      dest: /etc/samba/smb.conf
+
+  - name: create smbgroup
+    group:
+      name: smbgroup
+      state: present
+
+  - name: create smbuser and add to smbgroup
+    user:
+      name: smbuser
+      password: "{{ 'smbpass' | password_hash('sha512', 'mysecretsalt') }}"
+      state: present
+      group: smbgroup
+
+  - name: set smbpassword
+    shell: (echo smbpass; echo smbpass) | smbpasswd -s -a smbuser
+
+  - name: create share directories
+    file:
+      path: /srv/samba/{{ item }}
+      state: directory
+      group: smbgroup
+      mode: '755'
+      owner: smbuser
+    loop:
+    - share
+    - share-encrypted
+
+  - name: set selinux permissions for samba dir
+    command: chcon -R -t samba_share_t /srv/samba/
+
+  - name: create and start the samba services
+    systemd:
+      name: smb
+      state: started
+      enabled: yes

build-scripts/setup_samba.sh

@@ -7,6 +7,9 @@ cat > /etc/samba/smb.conf << EOL
 workgroup = WORKGROUP
 valid users = @smbgroup
 server signing = mandatory
+ea support = yes
+store dos attributes = yes
+vfs objects = xattr_tdb streams_xattr
 
 [$SMB_SHARE]
 comment = Test Samba Share

examples/high-level/directory-management.py

@@ -0,0 +1,30 @@
+from smbclient import (
+    listdir,
+    mkdir,
+    register_session,
+    rmdir,
+    scandir,
+)
+
+# Optional - register the server with explicit credentials
+register_session("server", username="admin", password="pass")
+
+# Create a directory (only the first request needs credentials)
+mkdir(r"\\server\share\directory", username="user", password="pass")
+
+# Remove a directory
+rmdir(r"\\server\share\directory")
+
+# List the files/directories inside a dir
+for filename in listdir(r"\\server\share\directory"):
+    print(filename)
+
+# Use scandir as a more efficient directory listing as it already contains info like stat and attributes.
+for file_info in scandir(r"\\server\share\directory"):
+    file_inode = file_info.inode()
+    if file_info.is_file():
+        print("File: %s %d" % (file_info.name, file_inode))
+    elif file_info.is_dir():
+        print("Dir: %s %d" % (file_info.name, file_inode))
+    else:
+        print("Symlink: %s %d" % (file_info.name, file_inode))

examples/high-level/file-management.py

@@ -0,0 +1,39 @@
+from smbclient import (
+    link,
+    open_file,
+    remove,
+    register_session,
+    stat,
+    symlink,
+)
+
+# Optional - register the server with explicit credentials
+register_session("server", username="admin", password="pass")
+
+# Read an existing file as text (credentials only needed for the first request to the server if not registered.)
+with open_file(r"\\server\share\file.txt", username="admin", password="pass") as fd:
+    file_contents = fd.read()
+
+# Read an existing file as bytes
+with open_file(r"\\server\share\file.txt", mode="rb") as fd:
+    file_bytes = fd.read()
+
+# Create a file and write to it
+with open_file(r"\\server\share\file.txt", mode="w") as fd:
+    fd.write(u"content")
+
+# Write data to the end of an existing file
+with open_file(r"\\server\share\file.txt", mode="a") as fd:
+    fd.write(u"\ndata at the end")
+
+# Delete a file
+remove(r"\\server\share\file.txt")
+
+# Get info about a file
+stat(r"\\server\share\file.txt")
+
+# Create a symbolic link
+symlink(r"\\server\share\directory", r"\\server\share\link")
+
+# Create a hard link
+link(r"\\server\share\file.txt", r"\\server\share\hard-link.txt")

examples/directory-management.py examples/low-level/directory-management.py

@@ -4,6 +4,8 @@ universal = 1
 [tool:pytest]
 pep8maxlinelength = 119
 pep8ignore = setup.py E501
+markers =
+    pep8: Satisfy pytest warning about custom markers
 
 [metadata]
-license_file = LICENSE
+license_file = LICENSE

examples/file-management.py examples/low-level/file-management.py

@@ -1,5 +1,7 @@
 #!/usr/bin/env python
-# coding: utf-8
+# -*- coding: utf-8 -*-
+# Copyright: (c) 2019, Jordan Borean (@jborean93) <[email protected]>
+# MIT License (see LICENSE or https://opensource.org/licenses/MIT)
 
 from setuptools import setup
 
@@ -14,26 +16,23 @@
 
 setup(
     name='smbprotocol',
-    version='0.3.0',
-    packages=['smbprotocol'],
+    version='1.0.0',
+    packages=['smbclient', 'smbprotocol'],
     install_requires=[
         'cryptography>=2.0',
         'ntlm-auth>=1.2.0',
         'pyasn1',
         'six',
     ],
     extras_require={
-        ':python_version<"2.7"': [
-            'ordereddict'
-        ],
         'kerberos:sys_platform=="win32"': [
-            'pywin32'
+            'pywin32',
         ],
         'kerberos:sys_platform!="win32"': [
-            'gssapi>=1.4.1'
-        ]
+            'gssapi>=1.4.1',
+        ],
     },
-    python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*',
+    python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*',
     author='Jordan Borean',
     author_email='[email protected]',
     url='https://github.com/jborean93/smbprotocol',
@@ -47,7 +46,6 @@
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',