ubuntu升级python版本正确姿势

前言

由于兼容性和历史原因，ubuntu（16.04 LTS）系统自带两个python版本：python2.7和python3.5,维护两个版本可能会引来了很多问题，而ubuntu的base Debian对于python安装的额外操作更是引发了更多的问题，简言之，Debian系对于python的管理就是一坨翔！

具体讨论可以参考https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e

其中python默认指向了python2.7,python3默认指向了3.5

$ ls /usr/bin/ |grep python
dh_pypy -> ../share/dh-python/dh_pypy*
dh_python2*
dh_python3 -> ../share/dh-python/dh_python3*
pdb2.7 -> ../lib/python2.7/pdb.py*
pdb3.5 -> ../lib/python3.5/pdb.py*
py3versions -> ../share/python3/py3versions.py*
pybuild -> ../share/dh-python/pybuild*
python -> python2.7*
python2 -> python2.7*
python2.7*
python2-jsondiff*
python2-jsonpatch*
python2-jsonpointer*
python3 -> python3.5*
python3.5*
python3.5m*
python3m -> python3.5m*
pyversions -> ../share/python/pyversions.py*

在一些使用场景中，我们需要更高的python版本，有时需要更换默认版本为python3，由于以上原因我们会发现升级完成后pip不可用了，甚至apt这个最基础的包管理命令都不可用了。

以下是正确的版本升级姿势。

升级新版本

安装python3.9

通过deadsnakes这个ppa源进行升级

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.9

然后通过update-alternatives切换python3的默认版本

sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 0
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 1
sudo update-alternatives --set python3 /usr/bin/python3.9

升级pip3

这时候你可能会发现一个问题：pip3失效了！！

Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
  File "/usr/lib/python3/dist-packages/pip/__init__.py", line 14, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "/usr/lib/python3/dist-packages/pip/utils/__init__.py", line 23, in <module>
    from pip.locations import (
  File "/usr/lib/python3/dist-packages/pip/locations.py", line 9, in <module>
    from distutils import sysconfig
ImportError: cannot import name 'sysconfig' from 'distutils' (/usr/lib/python3.9/distutils/__init__.py)

回头一想，python3升级了，pip是不是也要升级啊

curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py

报错如下

Traceback (most recent call last):
  File "/home/ubuntu/get-pip.py", line 27081, in <module>
    main()
  File "/home/ubuntu/get-pip.py", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/home/ubuntu/get-pip.py", line 115, in bootstrap
    monkeypatch_for_cert(tmpdir)
  File "/home/ubuntu/get-pip.py", line 96, in monkeypatch_for_cert
    from pip._internal.commands.install import InstallCommand
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/commands/__init__.py", line 9, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/cli/base_command.py", line 13, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/cli/parser.py", line 12, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/configuration.py", line 26, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/utils/logging.py", line 13, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/utils/misc.py", line 40, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/locations/__init__.py", line 14, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6zu07luz/pip.zip/pip/_internal/locations/_distutils.py", line 9, in <module>
ModuleNotFoundError: No module named 'distutils.cmd'

经过调研，这个实际是由于ubuntu的python问题导致的，简单来说就是当前解释器返回的path不对。

pip安装脚本通过当前解释器获取全局系统变量PATH，默认是/usr/lib/python3.9/site-packages/，这个路径并不在sys.path中。

python3 -c 'import sys; print(sys.path)'
['', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages']

而多个python3版本共有的sys.path是/usr/lib/python3/dist-packages/,这也将导致兼容性问题。比如说python3.5安装的模块，python3.9也会从中调用，但兼容性可能出问题。

由于以上原因，安装过程中可能会获取错误版本的模块，也会导致安装目录不在sys.path中，即使安装成功后也无法在默认环境中执行pip。

我们可以手动添加/usr/lib/python3.9/site-packages到默认的sys.path变量中，也可以指定安装目录到sys.path中。

我们先试着把这个缺失的distutils模块补全，并在pip安装的时候指定目录到系统变量PATH下。

sudo apt install python3.9-venv
python3 get-pip.py --target $(python3 -c 'import sys; print(sys.path[-1])')

或者可以更新下sys.path

mkdir /usr/lib/python3.9/dist-packages 
echo /usr/lib/python3.9/site-packages > /usr/lib/python3.9/dist-packages/site-packages.pth

发现成功了

...
Installing collected packages: pip
Successfully installed pip-21.3.1

检查一下版本

pip3 --version

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 21.3.1 from /usr/lib/python3/dist-packages/pip (python 3.9)

终于正常了！

替换python默认版本为python3

此时，如果需要替换默认版本为3,同样通过update-alternatives配置

sudo update-alternatives --install /usr/bin/python python /usr/bin/python2 0
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1
sudo update-alternatives --set python /usr/bin/python3

试试apt正常不

sudo apt help

显示一切正常

apt 1.2.32ubuntu0.2 (amd64)
Usage: apt [options] command

apt is a commandline package manager and provides commands for
searching and managing as well as querying information about packages.
It provides the same functionality as the specialized APT tools,
like apt-get and apt-cache, but enables options more suitable for
interactive use by default.

Most used commands:
  list - list packages based on package names
  search - search in package descriptions
  show - show package details
  install - install packages
  remove - remove packages
  autoremove - Remove automatically all unused packages
  update - update list of available packages
  upgrade - upgrade the system by installing/upgrading packages
  full-upgrade - upgrade the system by removing/installing/upgrading packages
  edit-sources - edit the source information file

See apt(8) for more information about the available commands.
Configuration options and syntax is detailed in apt.conf(5).
Information about how to configure sources can be found in sources.list(5).
Package and version choices can be expressed via apt_preferences(5).
Security details are available in apt-secure(8).
                                        This APT has Super Cow Powers.

问题修复

升级或者切换python默认版本是一个风险很高的操作，可能导致系统很多默认命令使用不了。

比如，输入一个无效的命令，正常会提示命令不存在，但是现在

Traceback (most recent call last):
  File "/usr/lib/python3.9/dbm/gnu.py", line 4, in <module>
    from _gdbm import *
ModuleNotFoundError: No module named '_gdbm'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/CommandNotFound/CommandNotFound.py", line 7, in <module>
    import dbm.gnu as gdbm
  File "/usr/lib/python3.9/dbm/gnu.py", line 6, in <module>
    raise ImportError(str(msg) + ', please install the python3-gdbm package')
ImportError: No module named '_gdbm', please install the python3-gdbm package

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/command-not-found", line 27, in <module>
    from CommandNotFound.util import crash_guard
  File "/usr/lib/python3/dist-packages/CommandNotFound/__init__.py", line 3, in <module>
    from CommandNotFound.CommandNotFound import CommandNotFound
  File "/usr/lib/python3/dist-packages/CommandNotFound/CommandNotFound.py", line 9, in <module>
    import gdbm
ModuleNotFoundError: No module named 'gdbm'

还是之前的问题，gdbm模块兼容性问题。默认安装的版本是3.5兼容的，而3.9调用出现异常，只能安装该版本的gdbm

sudo apt install python3.9-gdbm

再比如，常用的add-apt-repository也报错了

Traceback (most recent call last):
  File "/usr/bin/add-apt-repository", line 12, in <module>
    from softwareproperties.SoftwareProperties import SoftwareProperties, shortcut_handler
  File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 27, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

同样是apt_pkg这个模块的兼容性问题导致。不同的是，由于依赖的so包问题，我们需要链接原来的so包,然后重新安装。

cd /usr/lib/python3/dist-packages
sudo ln -s apt_pkg.cpython-35m-x86_64-linux-gnu.so apt_pkg.so
sudo apt purge python3-get
sudo apt update && sudo apt install python3-apt
sudo apt install software-properties-common

再次执行sudo add-apt-repository --yes --update ppa:ansible/ansible发现报错

Traceback (most recent call last):
  File "/usr/bin/add-apt-repository", line 173, in <module>
    if not sp.add_source_from_shortcut(shortcut, options.enable_source):
  File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 772, in add_source_from_shortcut
    if worker.isAlive():
AttributeError: 'Thread' object has no attribute 'isAlive'

究其原因，python3.9的isAlive()方法变成了is_alive(),只能改源码了

# /usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py
772 if worker.is_alive():
# thread timed out.

再次执行终于没有报错了

如果版本切换回python3.5, pip3又失效了… 重新安装下pip

python3.5 get-pip.py --target $(python3.5 -c 'import sys; print(sys.path[-1])')

来回切换python3.5和python3.9,发现正常使用，完美～

注意事项

一般来说，生产环境很少会进行python升级，基本不会进行默认版本替换。升级后的python3由于上文所说的原因，很多安装模块可能会失效，此时需要pip3重新安装。

为了避免上述的很多奇葩问题，默认版本切换最好通过用户家目录变量配置alias实现。

各个版本下的模块尽量采用python3.X -m pip install XX进行环境隔离。

多版本的运行环境还是建议使用venv或者anaconda进行环境管理。