WebSeitz/wiki

(2024-04-28) Updating my MacBookPro Python

Finally following the advice from (2023-03-27) BiteCode on Python installation and packaging esp on Mac.

  • picking 3.11... 3.11.9, download from python site
  • double-click Install Certificates.command - looks good
  • the code I run most often is still under py2 - not bothering to change that
  • I don't have a venv directly under py3 directory, have separate ones in separate projects
  • for now, the only one I care about is py3/flask
  • cd into there, do python -m pip freeze > requirements.txt -> long list in the file
  • I want to make a new venv to use my new python...
    • deactivate
    • mv venv venv_old
    • python3.11 -m venv .venv
    • source .venv/bin/activate
    • python -m pip install -r requirements.txt
      • one issue: ERROR: Could not build wheels for lxml, which is required to install pyproject.toml-based projects but I'm not even sure what that was for, so not worrying about it
  • start new terminal, activate .venv, cd into flask/wikiweb-main, do flask run -> command not found: flask
  • python -m flask run -> No module named flask
  • python3.11 -m pip install flask
  • Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them
  • python -m pip install python-dotenv
  • it's like it didn't install those requirements.txt items
  • No module named 'flask_user' -> python -m pip install flask_user -> No matching distribution found for flask_user
  • refer back to (2020-10-12) Building user management in WikiFlux
  • python -m pip install Flask-User -> No matching distribution found for Flask-User
  • it's because the library hasn't been touched/updated, so has been yanked
  • python -m pip install Flask-Login (which is a dependency anyway) -> fine
  • per comment in Yanked link above: python -m pip install git+https://github.com/lingthio/Flask-User.git@v1.0.2.2#egg=Flask-User -> Successfully installed Flask-Mail-0.9.1 Flask-User-1.0.2.2 Flask-WTF-1.2.1 bcrypt-4.1.2 cffi-1.16.0 cryptography-42.0.5 passlib-1.7.4 pycparser-2.22 wtforms-3.1.2
  • python -m flask run -> ImportError: cannot import name 'TextField' from 'wtforms' (/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/wtforms/__init__.py)
  • The above error occurs when the TextField property is used with WTForms version 3.0 or above because the wtforms.TextField deprecated in favor of wtforms.StringField. But in previous line it looks like I'm using wtf v1?
    • update: generated/checked newer requirements.txt - using v3
  • I change forms.py to use StringField where it was using TextField
  • python -m flask run -> ModuleNotFoundError: No module named 'psycopg2'
  • python3.11 -m pip install psycopg2... python -m flask run -> ModuleNotFoundError: No module named 'feedgen' -> install feedgen
  • ModuleNotFoundError: No module named 'stripe' -> install stripe
  • run -> sqlalchemy.exc.ArgumentError: Can't add unnamed column to column collection
    • in the stack trace, the last line of my code mentioned is class Node(db.Model):
  • flask --version
Python 3.11.9
Flask 3.0.3
Werkzeug 3.0.2
  • here's the Node section of models.py:
class Node(db.Model):
  __tablename__ = 'nodes'
  __table_args__ = {'extend_existing': True}
  wiki_name_lower = Column(String(128), primary_key=True)
  wiki_name = Column(String(128))
  title = Column(String(128))
  space_id = Column(Integer, ForeignKey('spaces.id'), primary_key=True)
  space = relationship("Space")
  body = Column(UnicodeText)
  created = Column(DateTime)
  modified = Column(DateTime)
  __ts_vector__ = db.Column(TSVector(),db.Computed(
         "to_tsvector('english', title || ' ' || description)",
         persisted=True))
  __table_args__ = (Index('ix_video___ts_vector__',
          __ts_vector__, postgresql_using='gin'),)
  def __init__(self, space_id=1, wiki_name=None, title=None, body=None, created=None, modified=None):
    self.wiki_name = wiki_name
    if title:
      self.title = title
    else:
      self.title = wiki_name
    self.body = body
    self.wiki_name_lower = wiki_name.lower()
    self.space_id = space_id
    if created:
      self.created = created
    else:
      self.created = datetime.datetime.now()
    if modified:
      self.modified = modified
    else:
      self.modified = datetime.datetime.now()
  def __repr__(self):
    return '<Node %r>' % (self.wiki_name)
  • temporarily showing the whole stack trace here:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask/__main__.py", line 3, in <module>
    main()
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask/cli.py", line 1105, in main
    cli.main()
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask/cli.py", line 953, in run_command
    raise e from None
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask/cli.py", line 937, in run_command
    app: WSGIApplication = info.load_app()
                           ^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask/cli.py", line 335, in load_app
    app = locate_app(import_name, name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask/cli.py", line 245, in locate_app
    __import__(module_name)
  File "/Users/billseitz/Documents/code/py3/flask/wikiweb-main/wikiweb.py", line 1, in <module>
    from app import app
  File "/Users/billseitz/Documents/code/py3/flask/wikiweb-main/app/__init__.py", line 13, in <module>
    from app import routes, models
  File "/Users/billseitz/Documents/code/py3/flask/wikiweb-main/app/models.py", line 74, in <module>
    class Node(db.Model):
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask_sqlalchemy/model.py", line 92, in __init__
    super().__init__(name, bases, d, **kwargs)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask_sqlalchemy/model.py", line 144, in __init__
    super().__init__(name, bases, d, **kwargs)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/orm/decl_api.py", line 196, in __init__
    _as_declarative(reg, cls, dict_)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/orm/decl_base.py", line 244, in _as_declarative
    return _MapperConfig.setup_mapping(registry, cls, dict_, None, {})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/orm/decl_base.py", line 325, in setup_mapping
    return _ClassScanMapperConfig(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/orm/decl_base.py", line 575, in __init__
    self._setup_table(table)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/orm/decl_base.py", line 1726, in _setup_table
    table_cls(
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/flask_sqlalchemy/model.py", line 178, in __table_cls__
    return sa.Table(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in __new__
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 431, in __new__
    return cls._new(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 485, in _new
    with util.safe_reraise():
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 481, in _new
    table.__init__(name, metadata, *args, _no_init=False, **kw)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 873, in __init__
    self._init_items(
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 231, in _init_items
    spwd(self, **kw)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/base.py", line 1324, in _set_parent_with_dispatch
    self._set_parent(parent, **kw)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 5232, in _set_parent
    ColumnCollectionMixin._set_parent(self, table)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/schema.py", line 4302, in _set_parent
    self._columns.add(col)
  File "/Users/billseitz/Documents/code/py3/flask/.venv/lib/python3.11/site-packages/sqlalchemy/sql/base.py", line 1939, in add
    raise exc.ArgumentError(
sqlalchemy.exc.ArgumentError: Can't add unnamed column to column collection
  • asked for help in sqlalchemy github-discussion

Apr29

  • I think I know what happened - when I changed models.py because of wtforms deprecating TextField I didn't change the actual db. There's probably a mismatch.

Apr30

  • Actually I hadn't changed models.py for wtforms, only forms.py - so my theory from yesterday was wrong
  • Oh derp when I started working on TeamGarden I just used the same local db as for WikiFlux, which kinda makes sense, but added a new field to the nodes table, but now obviously if I try to use the old WikiFlux models.py it won't have an entry for that field.
    • I had also added a ts_vector field for full-text-search, so should lose that, too
  • Options
    • copy the db so I have a separate db for each app
    • do my upgrading work only on the new app - not great idea in case I want to touch the old app for production changes since it's still in use
    • I guess I know the answer :) option2 it is
  • create database teamgarden with template wikiweb, then back in wikiweb
    • alter table nodes drop column node_data
    • alter table nodes drop column __ts_vector__
  • ugh actually I had ts_vector field in my local db, which matches my py3 DigitalGarden server for FluxGarden, but not my py2 Linode server for webseitz/wikiflux. Ugh have to decide which direction to go here...
    • (also I had tried ...flask run at both steps of dropping those fields, and have been getting the original error at all steps, so even once I get this right it might not solve the problem....)

May05

  • make db copies so I have 3 local dbs: fluxent, fluxgarden (adds ts_vector), and teamgarden (also adds node_data)
  • rename the 3 code folders to match, update the database.py file in each
  • the one I've been working with is fluxgarden, so it should have the ts_vector field
  • actually, changed my mind. Let's make things as simple as possible. So I'll work with fluxent which doesn't have the ts_vector
  • have to make that forms.py change for TextField->StringField
  • then it launches!
  • but when I hit http://127.0.0.1:5000/wiki/FrontPage I get sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: database "wikiweb" does not exist which is odd since I changed the db named in database.py
  • hmm maybe in an env file?
  • look in .env, see SERVER_NAME = 'flux.garden:5000' which is an issue, though maybe unrelated.... change that to webseitz.fluxent2.com then hit http://webseitz.fluxent2.com:5000/wiki/FrontPage which gives me URL-not-found. If I hit http://webseitz.fluxent2.com:5000/ I get This is a temporary static page. with link to fluxgarden registration form. Ugh I definitely have some wrong/weird stuff here....
  • I'm going to grab from server
    • already have a .zip folder there from Jan26'2021, but there are a couple files in there with newer dates.
    • do tar -zcvf fluxent.tar.gz app
  • fyi that linode server has
Python 2.7.6
Flask 1.1.2
Werkzeug 1.0.1
  • make empty fluxent folder, unzip the downloaded file inside it, which makes the app subfolder
  • edit database.py and wikiweb.py (this code is from before lots of refactoring)
  • ugh start getting all the python v2 vs v3 errors from (2021-02-07) Migrating to Python3
  • realize I added the user-mgmt bits for FluxGarden before being pushed into the py3 change.

So I need to take a step back and clarify wtf I'm trying to accomplish

  • I want to see that I have a platform/package-system to move forward with - I think I've gotten that far
  • I'd like to migrate my fluxent stuff over to Digital Ocean, but I hit roadblocks on getting nginx to deal with 2 domains. Go back to that? (2022-08-25) Move webseitz to DigitalOcean
  • If I can't do that migration in the short-run, I'd like to make improvements to fluxent, but I can't realistically do that without moving it to py3. I think I had some things converted/running at Mar28 at (2021-03-01) Massive Server Upgrade, but (a) I'm not sure, and (b) I think I just reverted everything. (I think I was only checking the changes moving toward FluxGarden into Gitlab.)

Apr'2026 starting again

Apr03'2026

  • goal: get linode and my laptop running fluxent code but with py3 and current flask, so I can make various improvements.
  • steps
    • old plan
      • get linode code running on my laptop (match python and flask versions)
      • killing this idea because just getting anything py2 installed is pretty impossible these days
    • on laptop....
    • compare schema to my local db
    • make env, install fresh py3/flask
    • get flask hello-world running
    • get simplest flask talking to my db running (in "current" flask app structure?)
    • run py3 updates on copy of old code
    • get that code working
    • repeat process at linode
  • starting...
  • go to linode;
  • cd /tmp do su postgres then pg_dump -s wikiflux > wikiflux.sql
  • cd /srv/www/wikiflux.net then tar -zcvf fluxent.tar.gz app
  • on laptop, make new folder, cd into
  • scp id@fluxent.com:/srv/www/wikiflux.net/fluxent.tar.gz .
  • scp id@fluxent.com:/tmp/wikiweb.sql .

Apr04

  • use pg_dump locally to export 'fluxent' schema (see notes above about 3 different db versions)
  • compare schemas: 3 tables at linode, 6 tables local (because of having started new features without snapshotting the db first)
    • so my plan is to rip stuff out of the local db until it matches the linode schema
    • also bunch of extra fields in table users
  • drop tables, fields, sequences in postgres command-line; also rebuild a composite primary key where the fields were in opposite order
  • re-pg_dump, compare again; I think the remaining differences are irrelevant (platform-specific, etc.)
  • since I'm going straight to working on py3, will make new folder within my py3/flask directory

Apr05

  • there's already a .env in py3/flask/, so activate it
  • there's already a py3/flask/test/ directory, so cd into it.
  • flask run - gives terminal feedback; go in brower to http://127.0.0.1:5000, get Hello, World, so that's all good.
  • flask --version
Python 3.11.9
Flask 3.0.3
Werkzeug 3.0.2
  • what's the next step? What's the purpose of the simplest possible db app?
    • To debug the linode database.py, models.py and environment variables
    • so "app" can just return a count(*) from nodes and we're good
    • I'm going to do this right in the folder I plan to put the full app in
  • as with the 3 dbs, I'm going to put this app in a `py3/flask/fluxent/ directory
    • then I expand my linode tar, which makes a fluxent/app directory. But I don't want to use that yet, so I rename it app_linode/
  • then make an empty /app/ folder beside it
  • my app is structured a little differently from the standard, but part of that is because of the uwsgi stuff. But that's not as relevant here, so I'll go the "standard" route for now..... make the __init__.py, routes.py, fluxent.py outside /app/, and set environment variable
  • flask run -> another Hello World (that's what's in routes.py)
  • I had a local-postgres version of database.py saved at linode, so I copy that into /app/ and change the db-name to fluxent
  • also copy over models.py
  • flask run -> no problem
  • copy a few lines from linode app to routes.py
import datetime
import os
import string
from flask_sqlalchemy import SQLAlchemy
from database import db_session
  • flask run -> ModuleNotFoundError: No module named 'database'
  • add from app import database to routes.py
  • flask run -> TypeError: Invalid argument(s) 'convert_unicode' sent to create_engine()
    • SqlAlchemy dropped that param; cut it

Apr06

  • flask run -> ModuleNotFoundError: No module named 'database' at __init__.py from database import db_session -> comment it out
  • flask run -> runs (just Hello World so far)
  • add couple lines to query db to routes.index(); flask run -> No module named 'database' again, from from database import Base
  • discover that changing lines like that to from app.database import Base makes them work
  • flask run -> runs; hit page -> The view function for 'index' did not return a valid response. The function either returned None or ended without a return statement. Excellent.
  • paste in some lines of code to turn nodes into lines
  • run flask -> break because of missing parts of my code
  • realize I may already have some py-2-vs-3 issues; run 2to3 on each code file (gads, they're deprecating it, it will be removed in Python3.13...); run flask -> errors on giving me node.body
  • realize (after banging head) that the node IDs (in print((node))) look odd; query db, realize there are some weird entries with empty bodies; add test for that
  • flask run -> runs, get page! (it's ugly and partial, but no errors!

Apr25 - time to move on to the main code file

  • I've been using routes.py (see Apr05 above), but it's time to change that.
    • is that "defined" just by being the import in __init__.py? It looks that way.
  • copy over wikiweb.py. Run 2to3 -w wikiweb.py
  • edit __init__.py to point to wikiweb instead of routes
  • flask run ->
Error: Could not locate a Flask application. Use the 'flask --app' option, 'FLASK_APP' environment variable, or a 'wsgi.py' or 'app.py' file in the current directory.
  • I referenced "environment variables" above, but not explicit about actually setting one, or which one. Do env to list variables, and don't see FLASK_APP.
  • but we'll try adding it anyway: export FLASK_APP=wikiweb.py; do env and see it
  • flask run ->
* Serving Flask app 'wikiweb.py'
 * Debug mode: off
Address already in use
Port 5000 is in use by another program. Either identify and stop that program, or start the server with a different port.
  • whooops, it's correct, I hadn't killed the previous process, so kill it now
  • flask run -> runs!
  • hit http://127.0.0.1:5000/ -> Error; terminal says jinja2.exceptions.TemplateNotFound: index.html not surprising
  • make /app/templates/ folder, copy index.html into it
  • hit page -> jinja2.exceptions.TemplateNotFound: layout.html not surprising, copy over layout.html
  • hit page -> get an ugly static page! Terminal lists a bunch of 404 files
  • make /app/static/ folder, copy the 12 listed missing files over
  • hit page -> get pretty page!
  • click /wiki/ link -> Server Error; ModuleNotFoundError: No module named 'CommonMark'

Edited:    |       |    Search Twitter for discussion

No Space passed/matched! - http://fluxent.com/wiki/2024-04-28-UpdatingMyMacBookProPython

Made with flux.garden