1. Background
2. Windows Package Manager
3. One Python to rule them all
   1. Python Package PIP installer
4. IDE
5. Virtual Environments to avoid ‘DLL Hell’
   1. Using correct Python version – pyenv
6. Configuration Management
7. Unit Testing
8. Terraform Python as a Data Source

Background

Python is a secondary language for me, so I don’t set up new projects or environments regularly. These notes are first and foremost an aide memoire, but I’ve also elaborated on why certain things are being done; this is less for my benefit but for the newbie who might find this info helpful. But feel free to use the information as you wish. These notes are focused on a Windows-based developer platform as most devs I know work with Python locally.

Windows Package Manager

Windows’ official package manager that operates like yum, apt, etc., is still in the ‘works’; in the meantime, it’s worth getting Chocolatey set up. As this makes some installation activities a lot easier. Details for this are located at – https://chocolatey.org/install.

One Python to rule them all

To get started and to exploit all the features and tooling we need an initial version of Python installed. The recommendation is to use the latest version for any specific use case. The downloads can be retrieved from https://www.python.org/downloads/. If you follow the links – you’ll find an exe installer that will go through setting up the path etc for you.

Python Package PIP installer

If you’re working with an older version of Python pre 2.7.9 or 3.4 then you’ll need to install the package manager – known as PIP manually – details at https://pip.pypa.io/en/stable/installing/

IDE

I’ve found Microsoft Visual Studio Code to be a really good Python IDE – lightweight but well integrated. This can be installed from https://code.visualstudio.com/

Once installed you will need to install a set of plugins, we’ve been using:

Virtual Environments to avoid ‘DLL Hell’

Python-like just about every language needs the means to address the problem of ‘DLL Hell’ (the challenge of dependencies and versions of the language). This can be overcome using PyEnv and virtualenv as a command-line capability. Details of managing it VS Code can be found here which makes use of Conda. (Conda vs PyEnv info here) A colleague of mine also pointed out that PyCharm (part of the JetBrains family of IDEs has a graceful means to manage these challenges as well, explanation here, thanks @Nagesh).

Using correct Python version – pyenv

Eventually, you’ll be working on projects that need or only support specific versions of Python, given going back to Python 2 (whilst End of Life) is still used in places. As a result, a utility to make it easy to switch between different versions of Python is essential. This is what Pyenv does. Details for installation are at – https://github.com/pyenv-win/pyenv-win#installation but with Chocolatey set up – the simplest solution is:

choco install pyenv-win

Once Pyenv is installed you can then use it to install the other versions of Python that you may need.

Configuration Management

You also need a config repository manager, if you’re using GIT then the recommendations are:

• Gitkraken – free for Open Source projects, otherwise paid. Students get it discounted
• Tortoise GIT integrated into Windows File Explorer
• VS Code plugin – Git Extension Pack – https://marketplace.visualstudio.com/items?itemName=donjayamanne.git-extension-pack

Unit Testing

We need Unit Testing capabilities, the simplest option here is to utilize the Python native capabilities. So no additional installation work.

Code Annotation and Doc Generation

Documentation relating to how code works is best when it is in line with the code rather than separate documents as this creates the best chance of the documentation actually reflecting the implementation.

The best way for documentation to be provided in Python is to use restructured (aka reST). There are a range of tools that can work with the reST approach to docs. The reST model works a lot like markdown for the most part but not entirely.

With the code documented e.g. a function description:

def find(name=None, query_type=USER, query_msg="", print_find = False):

    """

    finds xxxx

    **Parameters**`
    * name : xx
    * query_type : xx 
    * query_msg : xx
    * print_find : xx

    **Returns**

      OCID if a single entity is located. If multiple entities are found this is a list of OCIDs
"""

We would recommend pdoc which can be installed by:

pip install pdoc

This will install the tool. Then to run the tool:

pdoc <pythpon file>

Running pdoc will launch a local webserver and the browser page is opened accordingly. With it is an HTML page that dynamically updates inline with editing the document. So as documentation is applied, how it renders can be seen.

While HTML is perhaps not the best format to put into a repository such as GitHub it isn’t difficult to translate HTML to markdown using pandoc.

Terraform Python as a Data Source

Connecting Terraform to Python scripts to generate content e.g. algorithms for naming, creating JSON for tagging mechanisms can be done. On the Terraform side, this is done by defining a Data Source. When used within Python it calls the Python script passing any data through as JSON string using stdout (sys.stdin.read) rather than the args being passed in.

If you want to determine whether the Python code has been invoked from the shell or Terraform, the giveaway is that sys.argv[0] won’t provide the name of the invoking module (it’s own name when from the CLI).

Remember this means you need to ensure no logging goes to the console as this will get read back by Terraform as it is looking at stdout for a JSON structured response and logging information will simply throw a spanner in the works.

Useful Links