fix: the generated sdist file should contain everything needed to check and test the code, and to build the documentation as well

[jenstroeger]

This change supersedes PR #948 based on discussion pypa/flit#745, and contributes to PR #952.

The generated sdist file now contains these files:

> tar -ztvf dist/package-2.16.0.tar.gz
-rw-r--r-- 0/0            1883 2025-07-01 06:30 package-2.16.0/.flake8
-rw-r--r-- 0/0            5035 2025-07-01 06:30 package-2.16.0/.pre-commit-config.yaml
-rw-r--r-- 0/0            1063 2025-07-01 06:30 package-2.16.0/LICENSE.md
-rw-r--r-- 0/0           11471 2025-07-01 06:30 package-2.16.0/Makefile
-rw-r--r-- 0/0           28836 2025-07-01 06:30 package-2.16.0/README.md
-rw-r--r-- 0/0             675 2025-07-01 06:30 package-2.16.0/docs/Makefile
-rw-r--r-- 0/0            2006 2025-07-01 06:30 package-2.16.0/docs/source/conf.py
-rw-r--r-- 0/0             816 2025-07-01 06:30 package-2.16.0/docs/source/index.rst
-rw-r--r-- 0/0            8226 2025-07-01 06:30 package-2.16.0/pyproject.toml
-rw-r--r-- 0/0             386 2025-07-01 06:30 package-2.16.0/src/package/__init__.py
-rw-r--r-- 0/0             496 2025-07-01 06:30 package-2.16.0/src/package/__main__.py
-rw-r--r-- 0/0              80 2025-07-01 06:30 package-2.16.0/src/package/py.typed
-rw-r--r-- 0/0             520 2025-07-01 06:30 package-2.16.0/src/package/something.py
-rw-r--r-- 0/0             615 2025-07-01 06:30 package-2.16.0/tests/conftest.py
-rw-r--r-- 0/0             686 2025-07-01 06:30 package-2.16.0/tests/test_something.py
-rw-r--r-- 0/0            1352 1970-01-01 10:00 package-2.16.0/setup.py
-rw-r--r-- 0/0           31320 1970-01-01 10:00 package-2.16.0/PKG-INFO

This is essentially a git archive of the folder allowing somebody to build the package and documentation, to run all code checks, and to run the tests.

However, because this generated archive is not a git folder but our development process assumes both git and pre-commit, the archive doesn’t quite work out of the box. I think it would be ok to remove Makefile and .pre-commit-config.yaml, and let the user figure out the rest?

Tagging @takluyver

[takluyver]

I think most of your Makefile is usable without a git repo - and even pre-commit config can be used by creating a new blank git repo (that's really what pre-commit maintainers suggest 🤷 ). So I'd be inclined to leave them in the sdist. But it's definitely a matter of taste, once you go beyond the tests & docs that downstream packagers typically expect. 🙂

[takluyver]

I would also tend to include the changelog in the sdist, as part of the documentation. Also probably the security info.

[jenstroeger]

Commit 098bcd1.

[takluyver]

Heads up, what happens here will currently depend on whether you run flit build directly, or make an sdist through a generic tool that calls flit as a backend, like python -m build. With flit build, it currently uses info from git to include all checked-in files, like tests and docs sources, similar to git archive, except for those listed in exclude. However, Flit as a backend to general tools starts from only the files you need for installation. From Flit 4.0 this will be the default mode for flit build as well, although you'll be able to specify --use-vcs to keep the old behaviour.

https://flit.pypa.io/en/stable/pyproject_toml.html#including-files-committed-in-git-hg

So you might want to explicitly list includes for things you do want in the sdist.

(I can get into how we ended up in this scenario if you want. But the short version is, I like git-archive style sdists, but I don't think the backend can assume the presence of git, and I don't want it to behave differently if git is present.)

[jenstroeger]

Thank you @takluyver! With that in mind, I think it would make sense to

• document source distribution better in our README; and
• add a make build-from-sdist goal to our Makefile which takes an sdist and
  • runs the tests; then
  • produces a wheel (using build or flit build); then
  • produces the documentation.

@behnazh what do you think?

[behnazh]

One of the primary goals of using make build-from-sdist would be to reliably reproduce build artifacts I guess. To validate that the artifact built from source matches the original, it would be ideal to ensure the resulting package is bitwise equivalent. Achieving this level of reproducibility would eliminate the need for additional testing, and significantly simplifies the validation process.

[jenstroeger]

To validate that the artifact built from source matches the original, […]

With “original” I assume you mean a previously built wheel from the same commit hash? Basically:

1. Build a wheel from the repo at commit hash H, then
2. Build the sdist from the repo at commit hash H, then build a wheel from that sdist using the build timestamp from 1. and both wheels must be the same?

[behnazh]

With “original” I assume you mean a previously built wheel from the same commit hash? Basically:

1. Build a wheel from the repo at commit hash _H_, then

2. Build the sdist from the repo at commit hash _H_, then build a wheel from that sdist using the build timestamp from 1. and both wheels must be the same?

Yes, exactly.

[jenstroeger]

I think most of your Makefile is usable without a git repo - and even pre-commit config can be used by creating a new blank git repo (that's really what pre-commit maintainers suggest 🤷 ).

So, it takes a tad more to make this work on a folder unpacked from a git archive.

After unpacking the archived repository — as the thread hints — we need to git init the repo and we also need to git add all the existing files to make pre-commit work (otherwise it won’t know of any files).