support pip installation #6

New Issue

Closed

opened 2026-01-29 20:28:36 +00:00 by claunia · 10 comments

claunia commented 2026-01-29 20:28:36 +00:00

Owner

Copy Link

Originally created by @anthrotype on GitHub (Mar 30, 2015).

there is a problem if one tries to build the Python extension using the pip installer. The problem is that the setup.py is not located in the root of the repository, but in the python binding subfolder. The way pip works is to copy the source files to a temporary folder and try to build from there. But since the C/C++ source files are located higher in the repository tree (../enc, ../dec) relative to setup.py, then pip does not (cannot?) copy these over to its temporary build folder, and therefore it fails to build the extension.

A solution would be to make a hard copy (instead of a symlink) of the enc and dec folders inside the python subfolder. I believe git can efficiently handle such duplicate files and store them under the same object, as long as they have the same content.

That means one would have to synchronise it every time there is a change. I don't know if anyone has a better solution...

The reason I want to use pip is to allow publishing the Brotli extension to the official Python Package Index (PyPI) repository. Once that is done, one could simply do pip install brotli to download, build and install the extension. Besides, one could add brotli to the list of dependencies for other packages (e.g. fontTools, etc.).

I wonder whether @khaledhosny has already thought about publishing the Brotli extension to PyPI?

I'm already experimenting in this direction in https://github.com/anthrotype/brotli-wheels
I'm trying to use Travis and Appveyor to automatically build pre-compiled Python wheel packages for Windows and OS X.

Please let me know what you think.
Thank you,

Cosimo

Originally created by @anthrotype on GitHub (Mar 30, 2015). there is a problem if one tries to build the Python extension using the `pip` installer. The problem is that the setup.py is not located in the root of the repository, but in the `python` binding subfolder. The way pip works is to copy the source files to a temporary folder and try to build from there. But since the C/C++ source files are located higher in the repository tree (`../enc`, `../dec`) relative to setup.py, then pip does not (cannot?) copy these over to its temporary build folder, and therefore it fails to build the extension. A solution would be to make a hard copy (instead of a symlink) of the `enc` and `dec` folders inside the `python` subfolder. I believe git can efficiently handle such duplicate files and store them under the same object, as long as they have the same content. That means one would have to synchronise it every time there is a change. I don't know if anyone has a better solution... The reason I want to use pip is to allow publishing the Brotli extension to the official Python Package Index (PyPI) repository. Once that is done, one could simply do `pip install brotli` to download, build and install the extension. Besides, one could add brotli to the list of dependencies for other packages (e.g. fontTools, etc.). I wonder whether @khaledhosny has already thought about publishing the Brotli extension to PyPI? I'm already experimenting in this direction in https://github.com/anthrotype/brotli-wheels I'm trying to use Travis and Appveyor to automatically build pre-compiled Python [wheel](http://pythonwheels.com/) packages for Windows and OS X. Please let me know what you think. Thank you, Cosimo

claunia closed this issue 2026-01-29 20:28:36 +00:00

claunia commented 2026-01-29 20:28:37 +00:00

Author

Owner

Copy Link

@khaledhosny commented on GitHub (Mar 30, 2015):

How are you building with pip? I do pip install -e . routinely and had no issue with it.

@khaledhosny commented on GitHub (Mar 30, 2015): How are you building with pip? I do `pip install -e .` routinely and had no issue with it.

claunia commented 2026-01-29 20:28:37 +00:00

Author

Owner

Copy Link

@anthrotype commented on GitHub (Mar 30, 2015):

I know, but that only runs python setup develop for testing a package while developing it.
The problem arises when doing pip install ., without the -e option.
If you look at the traceback, the reason why pip fails is because it cannot find the C/C++ sources, since they are referenced using a relative path which is "above" the setup.py working directory:

$ pip install .
Processing /Users/cosimolupo/Documents/Github/brotli/python
Installing collected packages: Brotli
  Running setup.py install for Brotli
    clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DOS_MACOSX=1 -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c -o build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.o
    clang: error: no such file or directory: '/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c'
    clang: error: no input files
    error: command 'clang' failed with exit status 1
    Complete output from command /usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-R4EIB0-record/install-record.txt --single-version-externally-managed --compile:
    running install

    running build

    running build_ext

    creating build

    creating build/temp.macosx-10.9-x86_64-2.7

    creating build/temp.macosx-10.9-x86_64-2.7/private

    creating build/temp.macosx-10.9-x86_64-2.7/private/var

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T

    creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec

    clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DOS_MACOSX=1 -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c -o build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.o

    clang: error: no such file or directory: '/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c'

    clang: error: no input files

    error: command 'clang' failed with exit status 1

    ----------------------------------------
    Command "/usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-R4EIB0-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build

@anthrotype commented on GitHub (Mar 30, 2015): I know, but that only runs `python setup develop` for testing a package while developing it. The problem arises when doing `pip install .`, without the `-e` option. If you look at the traceback, the reason why pip fails is because it cannot find the C/C++ sources, since they are referenced using a relative path which is "above" the setup.py working directory: ``` $ pip install . Processing /Users/cosimolupo/Documents/Github/brotli/python Installing collected packages: Brotli Running setup.py install for Brotli clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DOS_MACOSX=1 -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c -o build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.o clang: error: no such file or directory: '/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c' clang: error: no input files error: command 'clang' failed with exit status 1 Complete output from command /usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-R4EIB0-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_ext creating build creating build/temp.macosx-10.9-x86_64-2.7 creating build/temp.macosx-10.9-x86_64-2.7/private creating build/temp.macosx-10.9-x86_64-2.7/private/var creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T creating build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DOS_MACOSX=1 -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c -o build/temp.macosx-10.9-x86_64-2.7/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.o clang: error: no such file or directory: '/private/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/dec/bit_reader.c' clang: error: no input files error: command 'clang' failed with exit status 1 ---------------------------------------- Command "/usr/local/opt/python/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-R4EIB0-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /var/folders/1w/6098jtks69sgwz0vtz2ysmsm0000gn/T/pip-oaQuzp-build ```

claunia commented 2026-01-29 20:28:37 +00:00

Author

Owner

Copy Link

@khaledhosny commented on GitHub (Mar 30, 2015):

I think we need to fix sdist command, since it seems not to copy any of the Brotli source files. If this does not fix pip, it will at least allow uploading to PyPI.

@khaledhosny commented on GitHub (Mar 30, 2015): I think we need to fix `sdist` command, since it seems not to copy any of the Brotli source files. If this does not fix `pip`, it will at least allow uploading to PyPI.

claunia commented 2026-01-29 20:28:38 +00:00

Author

Owner

Copy Link

@anthrotype commented on GitHub (Mar 30, 2015):

I'll have look at sdist.
The thing is, the way pip (but also distutils, setuptools, etc.) expects python packages to be organised is with a setup.py at the root of the package, and then everything else positioned either on the same level or inside sub-folders reachable from the root. Whereas here we want to keep the python folder isolated from the C/C++ source files, which is understandable, but this seems to conflict with the way Python packaging works. Maybe the solution could simply be to clone brotli as a submodule in another repository with a copy of setup.py positioned higher in the directory tree.

@anthrotype commented on GitHub (Mar 30, 2015): I'll have look at sdist. The thing is, the way pip (but also distutils, setuptools, etc.) expects python packages to be organised is with a setup.py at the root of the package, and then everything else positioned either on the same level or inside sub-folders reachable from the root. Whereas here we want to keep the python folder isolated from the C/C++ source files, which is understandable, but this seems to conflict with the way Python packaging works. Maybe the solution could simply be to clone brotli as a submodule in another repository with a copy of setup.py positioned higher in the directory tree.

claunia commented 2026-01-29 20:28:38 +00:00

Author

Owner

Copy Link

@khaledhosny commented on GitHub (Mar 30, 2015):

Maybe the solution could simply be to clone brotli as a submodule in another repository with a copy of setup.py positioned higher in the directory tree.

That is how it was done originally :)

@khaledhosny commented on GitHub (Mar 30, 2015): > Maybe the solution could simply be to clone brotli as a submodule in another repository with a copy of setup.py positioned higher in the directory tree. That is [how it was done originally](https://github.com/khaledhosny/brotli-python/issues/2) :)

claunia commented 2026-01-29 20:28:38 +00:00

Author

Owner

Copy Link

@anthrotype commented on GitHub (Mar 30, 2015):

yes, I do remember.

@anthrotype commented on GitHub (Mar 30, 2015): yes, I do remember.

claunia commented 2026-01-29 20:28:38 +00:00

Author

Owner

Copy Link

@khaledhosny commented on GitHub (Mar 31, 2015):

Thinking about it more, I think it shouldn’t harm to move the setup.py file to the top level directory, and leave the C file where it is, if it will simplify things.

@khaledhosny commented on GitHub (Mar 31, 2015): Thinking about it more, I think it shouldn’t harm to move the setup.py file to the top level directory, and leave the C file where it is, if it will simplify things.

claunia commented 2026-01-29 20:28:39 +00:00

Author

Owner

Copy Link

@anthrotype commented on GitHub (Mar 31, 2015):

It would definitely make the Python extension's setup much easier.

On Tue, Mar 31, 2015 at 8:42 AM, Khaled Hosny notifications@github.com
wrote:

Thinking about it more, I think it shouldn’t harm to move the setup.py file to the top level directory, and leave the C file where it is, if it will simplify things.

Reply to this email directly or view it on GitHub:
https://github.com/google/brotli/issues/52#issuecomment-87980117

@anthrotype commented on GitHub (Mar 31, 2015): It would definitely make the Python extension's setup much easier. On Tue, Mar 31, 2015 at 8:42 AM, Khaled Hosny notifications@github.com wrote: > ## Thinking about it more, I think it shouldn’t harm to move the setup.py file to the top level directory, and leave the C file where it is, if it will simplify things. > > Reply to this email directly or view it on GitHub: > https://github.com/google/brotli/issues/52#issuecomment-87980117

claunia commented 2026-01-29 20:28:39 +00:00

Author

Owner

Copy Link

@szabadka commented on GitHub (Mar 31, 2015):

If it involves just the setup.py file, it should be ok to have it in the top-level directory.

@szabadka commented on GitHub (Mar 31, 2015): If it involves just the setup.py file, it should be ok to have it in the top-level directory.

claunia commented 2026-01-29 20:28:39 +00:00

Author

Owner

Copy Link

@anthrotype commented on GitHub (Mar 31, 2015):

I can confirm pip is happy after 5e3e97e294

@anthrotype commented on GitHub (Mar 31, 2015): I can confirm pip is happy after 5e3e97e2947d82971bcc9f1bb099de7d7da91ac2

claunia referenced this issue 2026-01-29 20:40:25 +00:00

BrotliDecoderDecompress() crashed if inbuf is part of mmaped file #231

claunia referenced this issue 2026-01-29 20:46:13 +00:00

[PR #6] [MERGED] Move the block switch storing functions to the brotli_bit_stream library... #575

claunia referenced this issue 2026-01-29 20:46:14 +00:00

[PR #6] Move the block switch storing functions to the brotli_bit_stream library... #578

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

master

dependabot/github_actions/everything-f734e6b871

test_843619313

eustas-patch-1

test_902775910

dependabot/github_actions/actions/upload-artifact-7.0.1

test_889816903

test_884615862

test_883357601

test_835167838

v1.2

test_803421474

test_774593500

v1.1

recompression

v1.0

custom-dictionary

v1.2.0

v1.2.0rc2

dev/null

v1.2.0rc1

go/brotli/v1.1.1-rc0

go/cbrotli/v1.1.1-rc0

go/cbrotli/v1.1.0.bcr.1

go/cbrotli/v1.1.0.bcr.0

go/cbrotli/v1.1.0

v1.1.0

v1.1.0rc

v1.0.9

v1.0.8

v1.0.7

v1.0.6

v1.0.5

v1.0.4

v1.0.3

v1.0.2

v1.0.1

v1.0.0

v0.6.0

v0.5.2

v0.4.0

v0.3.0

v0.2.0

v0.1.0

Labels

Clear labels

bug

enhancement

feature

help wanted

pull-request

Mirrored from GitHub Pull Request

release-v1.1

release-v1.1.1

release-v1.2

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

claunia

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/brotli#6