## System specifications for deep scan

Before you proceed to run a deep scan, ensure that your system meets the following specification.

| Project Size      | Processor         | Memory |
| ------------------| ------------------| ------ |
| Small projects    | 4-core processor  | 16 GB  |
| Mid-size projects | 8-core processor  | 32 GB  |
| Large projects    | 16-core processor | 64 GB  |

## Software prerequisites

Complete the following prerequisites:

- Install Python 3.6 or higher versions. Refer to the [Python documentation](https://www.python.org/downloads/) for instructions on how to install Python.
- For uv managed projects, install Python 3.8 or higher.
- Use the package manager [pip](https://pip.pypa.io/en/stable/installation/), [Poetry](https://python-poetry.org/docs/), [PDM](https://pdm-project.org/en/latest/), [uv](https://docs.astral.sh/uv/), or [Pipenv](https://pipenv.pypa.io/en/latest/installation.html#installing-pipenv) in your projects to build your software packages.
- For Pipenv managed projects, install Pipenv and set the `ENDOR_PYTHON_PIPENV_ENABLED` environment variable to `true`. Projects that contain a `Pipfile` need Pipenv to scan, even if they build with another package manager.
- If you are using pip with Python 3.12 or higher versions, install [setuptools](https://pypi.org/project/setuptools/).
- Set up any build, code generation, or other dependencies that your project’s packages require.
- Organize the project as one or more packages using `setup.py`, `setup.cfg`, `pyproject.toml`, or `requirements.txt` package manifest files.
- Install Bazel version `5.x.x`, `6.x.x`, `7.x.x`, `8.x.x`, or `9.x.x` if your project uses Bazel. Endor Labs supports `rules_python` 0.9.0 up to (but not including) 0.30.0 with the WORKSPACE model only. Bzlmod requires `rules_python` 0.30.0 or higher and Bazel aspects. See [Bazel](https://docs.endorlabs.com/scan/bazel) for more information.
- Your repository must include a `WORKSPACE` file for WORKSPACE-based projects, or `MODULE.bazel` for Bzlmod projects when you use Bazel with Python.
- Make sure your repository includes one or more files with `.py` extension or pass either one of `requirements.txt`, `setup.py`, `setup.cfg` or `pyproject.toml` using the `--include-path` flag. See [Scoping scans](https://docs.endorlabs.com/best-practices/scoping-scans).

## Build Python projects

Create a virtual environment and build your Python projects before running the endorctl scan for the most accurate results. Endor Labs attempts to automatically create and configure a virtual environment when you do not provide one, but this may not work for complex projects. Verify that packages exist in the local package caches and build artifacts exist in the standard locations.

1. Configure any private repositories
   - If you use dependencies from a PyPI compatible repository _other than_`pypi.org`, configure it in the **Integrations** section of the Endor Labs web application. See [Configure private PyPI package repositories](https://docs.endorlabs.com/scan/sca/python#configure-private-pypi-package-repositories) for more details.
2. Clone the repository and optionally create a virtual environment inside it
   1. Clone the repository using `git clone` or an equivalent workflow.
   2. Enter the working copy root directory that’s created.
   3. Create a virtual environment based on your package manager:
      For **pip or setuptools**

- Use `python3 -m venv venv`. Set up the virtual environment in the root folder that you want to scan and name it `venv` or `.venv`.
          - Install your project’s dependencies using `venv/bin/python -m pip install -r requirements.txt` or `venv/bin/python -m pip install`.
          - If you create the virtual environment outside the project, use one of the methods in [Virtual environment support](https://docs.endorlabs.com/scan/sca/python#virtual-environment-support) to specify the path of the Python virtual environment to Endor Labs.

For **Poetry projects**

- Install your project’s dependencies using `poetry install`.

For **PDM projects**

- Install your project’s dependencies using `pdm install`.

For **Pipenv projects**
          - Run `pipenv install` in the project directory. This creates a `Pipfile.lock` if it doesn’t exist, and sets up a virtual environment while installing the required packages.
          - Run `export ENDOR_PYTHON_PIPENV_ENABLED=true` before you run the scan.

### Virtual environment support

Create a virtual environment to ensure consistent and accurate scan results and to verify that all dependencies install correctly before scanning. Automatic setup may encounter issues such as:

- Complex dependency chains or conflicting package requirements
- Private packages requiring authentication
- System-level dependencies not available in the scan environment
- Non-standard project structures or custom build scripts

Endor Labs attempts to automatically detect, create, or configure virtual environments for your projects. The behavior varies by package manager.

Poetry, Pipenv, and PDM

endorctl automatically detects and uses existing virtual environments managed by these tools.

uv

endorctl automatically creates a temporary virtual environment and deletes it after the scan completes. Install uv on your system for this automatic management to work.

pip

endorctl attempts to detect virtual environments in standard locations, such as `venv` or `.venv` directories in your project root. You can also use one of the following methods to specify the virtual environment:

- Set up the virtual environment in the root folder that you want to scan and name it **venv** or **.venv**, it is automatically picked up by the Endor Labs application.

```bash
export PYTHONPATH=/usr/tmp/venv:/usr/tmp/another-venv
```

- Set the environment variable `ENDOR_SCAN_PYTHON_VIRTUAL_ENV` to the path of the virtual environment of your Python project.

```bash
export ENDOR_SCAN_PYTHON_VIRTUAL_ENV=/usr/tmp/venv
```

- Set the environment variable `ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES` to true to indicate that a virtual environment is not present and Endor Labs can use the system-wide Python installation packages and modules.

```bash
export ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES=true
```

Setting both `ENDOR_SCAN_PYTHON_VIRTUAL_ENV` and `ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES` environment variables at the same time is currently not supported, and the scan may not be successful.

If you do not set up the virtual environment, Endor Labs attempts to set it up with all the code dependencies. Install all dependencies in a virtual environment for the most accurate results. If you use custom scripts without manifest files to assemble your dependencies, set up the virtual environment and install the dependencies first.

### Configure private PyPI package repositories

Endor Labs supports fetching and scanning dependencies from private PyPI package registries. Endor Labs will fetch resources from authenticated endpoints and perform the scan, allowing you to view the resolved dependencies and findings. See [PyPI package manager integrations](https://docs.endorlabs.com/integrations/package-managers/pypi-private-package-manager) for more information on configuring private registries.

## Scan Bazel projects

To scan Python projects that use Bazel, see [Bazel](https://docs.endorlabs.com/scan/bazel) for build instructions, supported rules, and scan commands. Endor Labs supports `rules_python` 0.9.0 up to (but not including) 0.30.0 with the WORKSPACE model only. Bzlmod requires `rules_python` 0.30.0 or higher with Bazel aspects. Software composition analysis includes reachability analysis. See [Bazel Aspects](https://docs.endorlabs.com/scan/bazel/bazel-aspects) for more information.

## Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

### Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition and perform dependency resolution. It discovers dependencies that the package has explicitly declared. If the package’s build file is incomplete then the dependency list will also be incomplete. This scan won’t perform reachability analysis to help you prioritize vulnerabilities.

```bash
endorctl scan --quick-scan
```

You can perform the scan from within the root directory of the Git project repository, and save the local results to a _results.json_ file. The results and related analysis information are available on the Endor Labs user interface.

```bash
endorctl scan --quick-scan -o json | tee /path/to/results.json
```

### Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully. The deep scan performs the following operations for the Python projects:

- Discovers explicitly declared dependencies,
- Discovers project dependent OSS packages present in the `venv/global` and `scope/python`.
- Performs reachability analysis and generates call graphs.
- Detects dependencies used in source code but not declared in the package’s manifest files called `phantom dependencies`.

```bash
endorctl scan
```

Use the following flags to save the local results to a _results.json_ file. The results and related analysis information are available on the Endor Labs user interface.

```bash
endorctl scan -o json | tee /path/to/results.json
```

During a deep scan, Endor Labs analyzes all private software dependencies in full by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but won’t impact subsequent scans. Organizations might not own some parts of the software internally and the related findings are not actionable by them. They can choose to disable this analysis using the flag `disable-private-package-analysis`. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

## Understand the scan process

Endor Labs uses the following two methods to analyze your Python code.

- [Dependency resolution using manifest files](https://docs.endorlabs.com/scan/sca/python#dependency-resolution-using-manifest-files)
- [Dependency resolution using static analysis](https://docs.endorlabs.com/scan/sca/python#dependency-resolution-using-static-analysis)

Endor Labs uses the results from both these methods to perform superior dependency resolution, identify security issues, detect open-source vulnerabilities, and generate call graphs.

### Dependency resolution using manifest files

In this method, Endor Labs analyzes the manifest files present in a project to detect and resolve dependencies. Endor Labs analyzes the manifest files in the following priority.

| Package manager | Priority | Build solution |
| ----------------| ---------| ----------------| 
| Poetry          | 1        | `poetry.lock`,`pyproject.toml` |
| Pipenv          | 2        | `Pipfile.lock`,`Pipfile` |
| PDM             | 3        | `pdm.lock`,`pyproject.toml` |
| uv              | 4        | `uv.lock`,`pyproject.toml` |
| pip             | 5        | `setup.py`,`setup.cfg`,`requirements.txt`,`pyproject.toml` |

Set `ENDOR_PYTHON_PIPENV_ENABLED` to `true` if you want Endor Labs to analyze Pipenv manifests. For Poetry, PDM, and uv, when both `lock` and `toml` files are present, Endor Labs analyzes both files to detect and resolve dependencies. For pip, Endor Labs analyzes the first available file in the priority list to detect and resolve dependencies, and ignores others.

| Build solution | Priority |
| -------------- | -------- |
| `setup.py`    | 1        |
| `setup.cfg`   | 2        |
| `pyproject.toml` | 3      |
| `requirements.txt` | 4     |

When a scan starts, Endor Labs identifies the package manager by inspecting files such as `pyproject.toml`, `poetry.lock`, `pdm.lock`, `setup.py`, and `requirements.txt`. If Endor Labs discovers `poetry.lock` or `pyproject.toml`, it uses Poetry to build the project. If it discovers `pdm.lock` or `pyproject.toml`, it uses PDM. Otherwise, it uses pip3.

### Example

This is an example that demonstrates scanning a Python repository from GitHub on your local system using the **endorctl** scan. This example assumes you run the scan on a Linux or Mac operating system and have the following Endor Labs API key and secret stored in environment variables. See [endorctl flags and variables](https://docs.endorlabs.com/developers-api/cli/environment-variables).

- `ENDOR_API_CREDENTIALS_KEY` set to the API key
- `ENDOR_API_CREDENTIALS_SECRET` set to the API secret
- `ENDOR_NAMESPACE` set to your namespace (you can find this when logged into Endor Labs by looking at your URL: `https://app.endorlabs.com/t/NAMESPACE/...`; it is typically a form of your organization’s name)

##### pip

```bash

git clone https://github.com/HybirdCorp/creme_crm.git
 cd creme_crm
 python3 -m venv venv
 source venv/bin/activate
 venv/bin/python3 -m pip install
 endorctl scan
```

##### Poetry

```bash
 git clone https://github.com/HybirdCorp/creme_crm.git
 cd creme_crm
 poetry lock
 endorctl scan
```

##### PDM

```bash
 git clone https://github.com/HybirdCorp/creme_crm.git
 cd creme_crm
 pdm install
 endorctl scan
```

##### uv

```bash
 git clone https://github.com/example/repo.git
 cd repo
 endorctl scan
```

##### Pipenv

```bash
 git clone https://github.com/example/repo.git
 cd repo
 pipenv install
 export ENDOR_PYTHON_PIPENV_ENABLED=true
 endorctl scan
```

### Handling custom and multiple requirement files in pip

Repositories often use non-standard names for pip requirement files (for example `default.txt`) or split dependencies across multiple `.txt` files. You can let Endor Labs discover those files automatically so you do not have to maintain a fixed list.

#### Auto detection of requirement files

Endor Labs can automatically detect alternate pip requirement files in your workspace. It identifies `.txt` files that follow pip requirements format using version specifiers and environment markers and treats them as manifests. To enable auto detection, set `ENDOR_SCAN_ENABLE_PYTHON_REQUIREMENTS_AUTO_DETECT=true` and then run the endorctl scan.

```bash
export ENDOR_SCAN_ENABLE_PYTHON_REQUIREMENTS_AUTO_DETECT=true
endorctl scan
```

#### Specify requirement files

To specify custom file names as requirement files, export the file name using the `ENDOR_SCAN_PYTHON_REQUIREMENTS` environment variable and then run the endorctl scan.

```bash
export ENDOR_SCAN_PYTHON_REQUIREMENTS=default.txt
```

To resolve dependencies from multiple requirement files, export them as a comma-separated list using the `ENDOR_SCAN_PYTHON_REQUIREMENTS` environment variable and then run the endorctl scan.

```bash
export ENDOR_SCAN_PYTHON_REQUIREMENTS=default.txt,requirements.txt
```

### Dependency resolution using static analysis

Not all Python projects include manifest files. A project can consist of a series of install statements that custom scripts assemble. Even when manifest files are present, the dependency information and version declared in the manifest file may differ drastically from what the project actually uses. To solve this problem, Endor Labs performs a static analysis on the code, giving you complete visibility of what your code actually uses.
- Endor Labs enumerates all Python packages and recognizes the import statements within the project. An import statement brings external modules or libraries into your Python script.
- Endor Labs performs a static analysis of the code to match import statements with pre-installed packages and recursively traverses all files to create a dependency tree with the actual versions installed in the virtual environment.
- Endor Labs detects dependencies at the system level, identifies which ones resolve, and retrieves the precise name and version information from the library currently in use.
- Endor Labs gives you accurate visibility into your project components and helps you understand how they depend on one another.

Endor Labs performs dependency resolution using static analysis on deep scans only.

### Known Limitations

- Endor Labs does not support Python versions older than 3.7, but they may work as expected.
- If you do not provide a virtual environment, Endor Labs does not assume Python version constraints based on the CI runtime environment. Dependencies appear for all possible Python versions at runtime. If you provide a virtual environment, Endor Labs respects what you installed in it.
- Symbolic links in manifest files may cause Endor Labs to duplicate the same package in the project.
- If a dependency is not available in the PyPI repository or in a configured private package repository, Endor Labs cannot build the software. Build the package in your local environment before scanning.
- Endor Labs treats a project as uv-managed if its `pyproject.toml` file contains the `tool.uv` key. Any member of a uv workspace is also uv-managed, even if its individual manifest file does not include the `tool.uv` key.
- When scanning uv workspaces, Endor Labs uses the workspace-level lock file for dependency resolution. Endor Labs does not scan individual workspace members as independent projects, ensuring consistency with uv’s workspace architecture.