(2023-03-27) BiteCode on Python installation and packaging esp on Mac

Installing and running Python: the bare minimum you can get away with. Installing and running Python on a Apple Macintosh.

Macs come with Python preinstalled, but I strongly advise that you install one version from python.org and don't use the one that is provided with your OS.

• your life will be much easier if you avoid the latest major version of Python. Go here and pick the newest one labelled yellow/"security". (He picks the one newer than that.)
• If you have several 3.10 releases, how to choose the right one? Target the one with the biggest bugfix number possible
• At the end of the installation, the Finder should open automatically to "/Applications/Python 3.10" (if not, go there). You will find a "Install Certificates.command" file. Without this, you will get SSLCertVerificationError.

If you want to know what pythons are installed, just type "python" then "tab", it will offer completion and show you all pythons currently available on your system.

Back to basics with pip and venv. You can use pip to install things in Python, and virtual environments to increase the chances that it works. They are provided with python.org installers.

Summary

• On Mac and Linux, use python3.X -m venv .venv to create a virtual environment, and source .venv/bin/activate to use it.
• Once you have activated a virtual environment, you can install a thing by doing python -m pip install thing. “thing” will then be available in this virtual environment and only in this virtual environment.
• I recommend to at least use a virtual environment for each project you have, and one for all your small scripts.

What is a virtual environment, and why to use one?

It has nothing to do with a virtual machine or anything that sophisticated: it's just a folder.

A virtual environment is only useful if you plan to install third party tools. E.G: if you are going to use pip. In fact, if you are going to install third party tools, you should never do so outside of a virtual environment.

virtualenv, poetry, pipenv or virtualenvwrapper? None of them.

How to create a virtual environment

The next step is to decide of the name of your virtual environment. I often name it ".venv" (with a dot),

Then, you must decide where to put your virtual environment. Usually, I put one at the top-level directory of each of my project.

Don't start a Python shell now.

On Mac

can now create a new one with this command

python3.10 -m venv .venv

if I have a project that must work with two different versions of Python (3.9 and 3.10), I will have a ".venv39" and a ".venv310" virtual environment.

To use a virtual environment, the most convenient way is to activate it.

source .venv/bin/activate

source .venv/bin/activate

You can deactivate the virtual environment at any time by typing: deactivate And activate a different one if you wish.

You can also have several terminals, each with a different activated virtual environment. They won't see each other.

*While Python has a very rich standard library, at some point you will want to use something made by someone else.

The standard way to do this is to use "pip".*

*in a terminal, use this command:

python -m pip install NAME With "name" being the thing you wish to install.*

*The reverse of this command is:

python -m pip uninstall NAME*

At this stage, you may wonder about the “python -m” thingy. Why not call “pip” directly, isn’t the virtual environment making sure it’s the right one? It should, but sometimes it fails to do so. “python -m pip” has more chances to succeed than just “pip”, even in virtual environment. Yes, it’s annoying.

You cannot move a virtual environment, it will stop working. Also, if you change the Python used in the virtual environment, such as when uninstalling it, it will stop working.

For all these reasons, it can be handy to be able to recreate exactly the same virtual environment. To do this, run in a terminal (with a virtual environment activated): python -m pip freeze > requirements.txt This will create a new file named "requirements.txt" listing all installed packages in your virtual environment.

you can create a new virtual environment and reinstall all the packages in it by doing this: python -m pip install -r requirements.txt You can send the "requirements.txt" file to a colleague

"requirements.txt" files are not perfect. They don't separate dependencies for production and development. They always include the whole dependency graph, with pinned version, and for a specific platform.

If you have to use Anaconda, ignore this entire tutorial, and don't use pip and venv. Limit yourself to Anaconda tools.

Don't move a virtual environment. Create a "requirements.txt" file, delete the virtual environment and create a new one.

Create several virtual environments per versions of python if your project needs to support several versions. You may need several requirements.txt files as well, one for each env.

Relieving your Python packaging pain

*You can protect yourself against a lot of packaging pain by sticking to one recipe for installing and running Python:

• Don’t install the latest major version of Python
• Use only the python.org installer on Windows and Mac, or official repositories on Linux.
• Never install or run anything outside of a virtual environment
• Limit yourself to the basics: “pip” and “venv”
• If you run a command, use “-m”
• When creating a virtual environment, be explicit about which Python you use*

the new topic of complaints for the community is packaging

that’s not what most users are suffering about.

It’s the part where they want to use a package, and an invisible hand comes out of the terminal and slap them for having the audacity of wanting to be productive. This article will focus on solving that for you. Well, a lot of it, anyway.

the dirty secret of Python packaging is that most of the problems… don’t come from packaging.

They come from the deeper problem of bootstrapping Python, meaning finding, installing, configuring and running it.

1. Don't install the latest major version of Python

So if Python 3.11 is the latest major version, at the maximum, use 3.10. You can use an older one, of course. If it’s possible (I understand people don’t want to update Python every year), target down to 4 versions as your minimum. In our example, down to 3.7.

2. A. On Windows and Mac, use the official installer

2. B. On Linux, use the official repositories

You may develop on Windows/Mac and deploy on Linux. In that case, check what is available on the Linux machine, and install the same version of Windows, not the other way around.

3. To install things, use pip, and only pip

Don’t use poetry, pipenv, pdm, easy_install. Don’t use pipx. pipx is not pip. At all.

4. Always use pip in a virtual environment

This rule is the most important one.

the situation is such that there is currently no way for you to install anything reliably without a virtual environment.

5. To create a virtual environment, use “venv” and only “venv”

*There are other commands such as “virtualenv” and “virtualenvwrapper”. Don’t use them.

There are other tools such as pipx, pdm, poetry and pipenv. Don’t use them.

6. When running a Python command use "-m"

*Don't do :

pip install Do:

python -m pip install*

*Don't do :

jupyter notebook Do:

python -m jupyter notebook*

6. When creating a virtual environment, be explicitly about the Python you use

When you create a virtual environment, you should always specify which Python you want to use for the task, because that environment will be forever using this particular Python.

On Linux and Mac, you will have to use a version suffix, usually “pythonX.Y”.

And so to create a virtual environment you will end up typing some monstrous command.

*on Linux and Mac:

"python3.8 -m venv .venv*

Why not tell people to "simply" use pyenv, poetry or anaconda. Summary: I shared what I’ve witnessed to be the most reliable way to avoid many python packaging problems, for a large number of users.

This article listed the steps to take without justifying them, delegating this responsability to today’s post Send separate article, text/link missing above

Among other things, it advised not to use homebrew, pyenv, anaconda, poetry, and other tools.

Targeting the pain

The Python community is extremely diverse

So how do we help as many people as we can?

One big problem was the paradox of choice: too many ways to solve your packaging problems, too many docs or tutorials, with various levels of quality or relevance.

Another was that some errors were recurring over and over

There is no possible exhaustive list of all the causes of those issues, but fortunately, we don’t need one to do a lot of good. What we need is a Pareto list:

And so the recipe was born, refined as the years went by, from people to failure, from failure to pain. The quest to provide the maximum benefits to the largest sample of users.

You may have noticed what is not in that list:

Those problems do happen. In fact, this is exactly what poetry/anaconda/homebrew/pyenv have been designed for.

*However:

• Those problems happen less often than the ones from the other list.
• They don’t have a generic solution you can easily explain in a tutorial.
• Each tool solving those have subtle trade-offs, and add more modes of failures.*

the tooling used to solve the second list problems actually increase the risk of being exposed to the first list problems

How does the procedure deals with those problems

Using “-m” forces the users to know what Python they use.

Using a virtualenv removes most PATH, PYTHONPATH, permission and version issues,

Using the regular “python.org” installer increases the chance of getting a non-broken Python by the sheer power of statistics

Using standard pip and venv means no chicken and eggs issue for installing the tools.

Avoiding the latest major version of Python exposes the project to the part of the ecosystem that has a higher chance of compatibility

More details on why not to promote homebrew/pyenv/poetry/anaconda

They are tools that were born to solve problems, and they do.

However, it’s my experience that for the meaty part of the Gaussian curve of the user base, and I write for them as I wished someone wrote for me, the ratio cost/benefit is not in our favor.

Yes, but…

I specifically don’t talk about pip-tools, because I think once people have the procedure mastered, that’s what they should use. But that’s a separate article, that will mention you should get master pip and venv first.