python language

System specifications for deep scan

Before you proceed to run a deep scan, ensure that your system meets the following specification.

Software prerequisites

Complete the following prerequisites:

• Install Python 3.6 or higher versions. Refer to the Python documentation for instructions on how to install Python.
• For uv managed projects, install Python 3.8 or higher.
• Use the package manager pip, Poetry, PDM, uv, or Pipenv in your projects to build your software packages.
• For Pipenv managed projects, install Pipenv and set the ENDOR_PYTHON_PIPENV_ENABLED environment variable to true. Projects that contain a Pipfile need Pipenv to scan, even if they build with another package manager.
• If you are using pip with Python 3.12 or higher versions, install setuptools.
• Set up any build, code generation, or other dependencies that your project’s packages require.
• Organize the project as one or more packages using setup.py, setup.cfg, pyproject.toml, or requirements.txt package manifest files.
• Install Bazel version 5.x.x, 6.x.x, 7.x.x, 8.x.x, or 9.x.x if your project uses Bazel. Endor Labs supports rules_python 0.9.0 up to (but not including) 0.30.0 with the WORKSPACE model only. Bzlmod requires rules_python 0.30.0 or higher and Bazel aspects. See Bazel for more information.
• Your repository must include a WORKSPACE file for WORKSPACE-based projects, or MODULE.bazel for Bzlmod projects when you use Bazel with Python.
• Make sure your repository includes one or more files with .py extension or pass either one of requirements.txt, setup.py, setup.cfg or pyproject.toml using the --include-path flag. See Scoping scans.

Build Python projects

Create a virtual environment and build your Python projects before running the endorctl scan for the most accurate results. Endor Labs attempts to automatically create and configure a virtual environment when you do not provide one, but this may not work for complex projects. Verify that packages exist in the local package caches and build artifacts exist in the standard locations.

1. Configure any private repositories
   • If you use dependencies from a PyPI compatible repository other thanpypi.org, configure it in the Integrations section of the Endor Labs web application. See Configure private PyPI package repositories for more details.
2. Clone the repository and optionally create a virtual environment inside it
   1. Clone the repository using git clone or an equivalent workflow.
   2. Enter the working copy root directory that’s created.
   3. Create a virtual environment based on your package manager: For pip or setuptools

• Use python3 -m venv venv. Set up the virtual environment in the root folder that you want to scan and name it venv or .venv. - Install your project’s dependencies using venv/bin/python -m pip install -r requirements.txt or venv/bin/python -m pip install. - If you create the virtual environment outside the project, use one of the methods in Virtual environment support to specify the path of the Python virtual environment to Endor Labs.

For Poetry projects

• Install your project’s dependencies using poetry install.

For PDM projects

• Install your project’s dependencies using pdm install.

For Pipenv projects - Run pipenv install in the project directory. This creates a Pipfile.lock if it doesn’t exist, and sets up a virtual environment while installing the required packages. - Run export ENDOR_PYTHON_PIPENV_ENABLED=true before you run the scan.

Virtual environment support

Create a virtual environment to ensure consistent and accurate scan results and to verify that all dependencies install correctly before scanning. Automatic setup may encounter issues such as:

• Complex dependency chains or conflicting package requirements
• Private packages requiring authentication
• System-level dependencies not available in the scan environment
• Non-standard project structures or custom build scripts

Endor Labs attempts to automatically detect, create, or configure virtual environments for your projects. The behavior varies by package manager.

Poetry, Pipenv, and PDM

endorctl automatically detects and uses existing virtual environments managed by these tools.

uv

endorctl automatically creates a temporary virtual environment and deletes it after the scan completes. Install uv on your system for this automatic management to work.

pip

endorctl attempts to detect virtual environments in standard locations, such as venv or .venv directories in your project root. You can also use one of the following methods to specify the virtual environment:

• Set up the virtual environment in the root folder that you want to scan and name it venv or .venv, it is automatically picked up by the Endor Labs application.

export PYTHONPATH=/usr/tmp/venv:/usr/tmp/another-venv

• Set the environment variable ENDOR_SCAN_PYTHON_VIRTUAL_ENV to the path of the virtual environment of your Python project.

export ENDOR_SCAN_PYTHON_VIRTUAL_ENV=/usr/tmp/venv

• Set the environment variable ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES to true to indicate that a virtual environment is not present and Endor Labs can use the system-wide Python installation packages and modules.

export ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES=true

Setting both ENDOR_SCAN_PYTHON_VIRTUAL_ENV and ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES environment variables at the same time is currently not supported, and the scan may not be successful.

If you do not set up the virtual environment, Endor Labs attempts to set it up with all the code dependencies. Install all dependencies in a virtual environment for the most accurate results. If you use custom scripts without manifest files to assemble your dependencies, set up the virtual environment and install the dependencies first.

Configure private PyPI package repositories

Endor Labs supports fetching and scanning dependencies from private PyPI package registries. Endor Labs will fetch resources from authenticated endpoints and perform the scan, allowing you to view the resolved dependencies and findings. See PyPI package manager integrations for more information on configuring private registries.

Scan Bazel projects

To scan Python projects that use Bazel, see Bazel for build instructions, supported rules, and scan commands. Endor Labs supports rules_python 0.9.0 up to (but not including) 0.30.0 with the WORKSPACE model only. Bzlmod requires rules_python 0.30.0 or higher with Bazel aspects. Software composition analysis includes reachability analysis. See Bazel Aspects for more information.

Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition and perform dependency resolution. It discovers dependencies that the package has explicitly declared. If the package’s build file is incomplete then the dependency list will also be incomplete. This scan won’t perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully. The deep scan performs the following operations for the Python projects:

• Discovers explicitly declared dependencies,
• Discovers project dependent OSS packages present in the venv/global and scope/python.
• Performs reachability analysis and generates call graphs.
• Detects dependencies used in source code but not declared in the package’s manifest files called phantom dependencies.

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

During a deep scan, Endor Labs analyzes all private software dependencies in full by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but won’t impact subsequent scans. Organizations might not own some parts of the software internally and the related findings are not actionable by them. They can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

Understand the scan process

Endor Labs uses the following two methods to analyze your Python code.

• Dependency resolution using manifest files
• Dependency resolution using static analysis

Endor Labs uses the results from both these methods to perform superior dependency resolution, identify security issues, detect open-source vulnerabilities, and generate call graphs.

Dependency resolution using manifest files

In this method, Endor Labs analyzes the manifest files present in a project to detect and resolve dependencies. Endor Labs analyzes the manifest files in the following priority.

Set ENDOR_PYTHON_PIPENV_ENABLED to true if you want Endor Labs to analyze Pipenv manifests. For Poetry, PDM, and uv, when both lock and toml files are present, Endor Labs analyzes both files to detect and resolve dependencies. For pip, Endor Labs analyzes the first available file in the priority list to detect and resolve dependencies, and ignores others.

When a scan starts, Endor Labs identifies the package manager by inspecting files such as pyproject.toml, poetry.lock, pdm.lock, setup.py, and requirements.txt. If Endor Labs discovers poetry.lock or pyproject.toml, it uses Poetry to build the project. If it discovers pdm.lock or pyproject.toml, it uses PDM. Otherwise, it uses pip3.

Example

This is an example that demonstrates scanning a Python repository from GitHub on your local system using the endorctl scan. This example assumes you run the scan on a Linux or Mac operating system and have the following Endor Labs API key and secret stored in environment variables. See endorctl flags and variables.

• ENDOR_API_CREDENTIALS_KEY set to the API key
• ENDOR_API_CREDENTIALS_SECRET set to the API secret
• ENDOR_NAMESPACE set to your namespace (you can find this when logged into Endor Labs by looking at your URL: https://app.endorlabs.com/t/NAMESPACE/...; it is typically a form of your organization’s name)

pip

git clone https://github.com/HybirdCorp/creme_crm.git
 cd creme_crm
 python3 -m venv venv
 source venv/bin/activate
 venv/bin/python3 -m pip install
 endorctl scan

Poetry

 git clone https://github.com/HybirdCorp/creme_crm.git
 cd creme_crm
 poetry lock
 endorctl scan

PDM

 git clone https://github.com/HybirdCorp/creme_crm.git
 cd creme_crm
 pdm install
 endorctl scan

uv

 git clone https://github.com/example/repo.git
 cd repo
 endorctl scan

Pipenv

 git clone https://github.com/example/repo.git
 cd repo
 pipenv install
 export ENDOR_PYTHON_PIPENV_ENABLED=true
 endorctl scan

Handling custom and multiple requirement files in pip

Repositories often use non-standard names for pip requirement files (for example default.txt) or split dependencies across multiple .txt files. You can let Endor Labs discover those files automatically so you do not have to maintain a fixed list.

Auto detection of requirement files

Endor Labs can automatically detect alternate pip requirement files in your workspace. It identifies .txt files that follow pip requirements format using version specifiers and environment markers and treats them as manifests. To enable auto detection, set ENDOR_SCAN_ENABLE_PYTHON_REQUIREMENTS_AUTO_DETECT=true and then run the endorctl scan.

export ENDOR_SCAN_ENABLE_PYTHON_REQUIREMENTS_AUTO_DETECT=true
endorctl scan

Specify requirement files

To specify custom file names as requirement files, export the file name using the ENDOR_SCAN_PYTHON_REQUIREMENTS environment variable and then run the endorctl scan.

export ENDOR_SCAN_PYTHON_REQUIREMENTS=default.txt

To resolve dependencies from multiple requirement files, export them as a comma-separated list using the ENDOR_SCAN_PYTHON_REQUIREMENTS environment variable and then run the endorctl scan.

export ENDOR_SCAN_PYTHON_REQUIREMENTS=default.txt,requirements.txt

Dependency resolution using static analysis

Not all Python projects include manifest files. A project can consist of a series of install statements that custom scripts assemble. Even when manifest files are present, the dependency information and version declared in the manifest file may differ drastically from what the project actually uses. To solve this problem, Endor Labs performs a static analysis on the code, giving you complete visibility of what your code actually uses.

• Endor Labs enumerates all Python packages and recognizes the import statements within the project. An import statement brings external modules or libraries into your Python script.
• Endor Labs performs a static analysis of the code to match import statements with pre-installed packages and recursively traverses all files to create a dependency tree with the actual versions installed in the virtual environment.
• Endor Labs detects dependencies at the system level, identifies which ones resolve, and retrieves the precise name and version information from the library currently in use.
• Endor Labs gives you accurate visibility into your project components and helps you understand how they depend on one another.

Endor Labs performs dependency resolution using static analysis on deep scans only.

Known Limitations

• Endor Labs does not support Python versions older than 3.7, but they may work as expected.
• If you do not provide a virtual environment, Endor Labs does not assume Python version constraints based on the CI runtime environment. Dependencies appear for all possible Python versions at runtime. If you provide a virtual environment, Endor Labs respects what you installed in it.
• Symbolic links in manifest files may cause Endor Labs to duplicate the same package in the project.
• If a dependency is not available in the PyPI repository or in a configured private package repository, Endor Labs cannot build the software. Build the package in your local environment before scanning.
• Endor Labs treats a project as uv-managed if its pyproject.toml file contains the tool.uv key. Any member of a uv workspace is also uv-managed, even if its individual manifest file does not include the tool.uv key.
• When scanning uv workspaces, Endor Labs uses the workspace-level lock file for dependency resolution. Endor Labs does not scan individual workspace members as independent projects, ensuring consistency with uv’s workspace architecture.