README.md

# 🥞 The Stack v3 — full corpus

`stack-v3-full` is the **complete corpus** behind [The Stack v3](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train): **110+ TB** of source code from **224M** GitHub repositories (**770** languages), *before* near-dedup selection and quality filtering. It retains all near-duplicates (with cluster IDs), pre-filtering signals, and a stub entry for every excluded file — so you can build your own training mix from scratch.

> [!NOTE]
> **If you just want to train a model, use [`stack-v3-train`](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train) instead** — the deduplicated, quality-filtered, PII-redacted training dataset with contents inline, grouped by repository. It is also the main entry point for all Stack v3 documentation: dataset creation, licensing, opt-out, and usage considerations are covered in detail there.

> [!WARNING]
> Unlike `stack-v3-train`, the file contents in this bucket are **not PII-redacted**. They are the raw file contents as crawled from GitHub and may contain emails, credentials, API keys, and other personal data that was public on GitHub. **You are responsible for performing PII redaction on any data you derive from `stack-v3-full`.**

- [Layout](#layout)
- [How to access it](#how-to-access-it)
- [Data fields](#data-fields)
- [Stub entries](#stub-entries)
- [Near-duplicates: rolling your own dedup and filters](#near-duplicates-rolling-your-own-dedup-and-filters)
- [Dataset statistics](#dataset-statistics)
- [Licensing, opt-out, and limitations](#licensing-opt-out-and-limitations)

## Layout

The corpus is split into two parts so that identical file blobs shared across many repositories are stored only **once**:

- **`metadata/`** — one row per repository, with a `files[]` array describing **every** file (path, language, licenses, dedup cluster, …). Each file references its content by `content_id`.
- **`contents/`** — one row per **unique** file blob, keyed by `content_id`, Hive-partitioned by language (`contents/language=Python/`, `contents/language=JavaScript/`, …). This is where the actual `content` text lives. `language` is encoded in the directory name, not stored as a column inside the Parquet files — engines with Hive-partitioning support (Spark, DuckDB, PyArrow datasets) reconstruct it as a virtual column.

To attach content to files, join the two parts on **`content_id`**.

## How to access it

`stack-v3-full` is hosted as a [Hugging Face Storage Bucket](https://huggingface.co/docs/hub/storage-buckets) (not a regular dataset repo). See the [bucket access documentation](https://huggingface.co/docs/hub/storage-buckets-access) for all access methods, including the [S3-compatible API](https://huggingface.co/docs/hub/storage-buckets-s3). At 110+ TB, downloading the whole bucket is rarely what you want — start with a language partition or a metadata sample.

**With DuckDB** (register the HF filesystem, then query directly):

```python
import duckdb
from huggingface_hub import HfFileSystem

duckdb.register_filesystem(HfFileSystem())

duckdb.sql("""
    WITH files AS (
        SELECT repo_path, UNNEST(files) AS f
        FROM 'hf://buckets/HuggingFaceCode/stack-v3-full/metadata/*.parquet'
    )
    SELECT
        files.repo_path,
        files.f.file_path,
        files.f.language,
        files.f.license_type,
        c.content
    FROM files
    JOIN 'hf://buckets/HuggingFaceCode/stack-v3-full/contents/language=Python/*.parquet' AS c
      ON files.f.content_id = c.content_id
    LIMIT 10
""").show()
```

**With pandas / PyArrow** (reads via `hf://buckets/` fsspec paths):

```python
import pandas as pd
from huggingface_hub import HfFileSystem

fs = HfFileSystem()

# Read the first metadata partition
meta_files = fs.glob("hf://buckets/HuggingFaceCode/stack-v3-full/metadata/*.parquet")
meta = pd.read_parquet(f"hf://{meta_files[0]}")

# Read all contents for a single language
contents = pd.read_parquet("hf://buckets/HuggingFaceCode/stack-v3-full/contents/language=OCaml/")
```

**With PySpark** (for large-scale joins — use the [S3-compatible API](https://huggingface.co/docs/hub/storage-buckets-s3), or use [`pyspark_huggingface`](https://github.com/huggingface/pyspark_huggingface)):

```python
from pyspark.sql import SparkSession, functions as F

spark = (
    SparkSession.builder
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.4.1")
    # Point s3a at the HF Storage Bucket gateway
    .config("spark.hadoop.fs.s3a.endpoint", "https://s3.hf.co/HuggingFaceCode")
    .config("spark.hadoop.fs.s3a.access.key", "HFAK...")
    .config("spark.hadoop.fs.s3a.secret.key", "...")
    .config("spark.hadoop.fs.s3a.path.style.access", "true")
    .config("spark.hadoop.fs.s3a.endpoint.region", "us-east-1")
    # The HF gateway returns 302 redirects to CDN on GetObject.  It only
    # proxies bytes directly for user-agents it knows can't follow redirects
    # (aws-cli, botocore, aws-sdk-rust).  The AWS Java SDK used by Hadoop
    # s3a doesn't follow the redirect, so we spoof the UA.
    .config("spark.hadoop.fs.s3a.user.agent.prefix", "botocore/1.35.0")
    .config("spark.hadoop.fs.s3a.change.detection.mode", "none")
    .config("spark.hadoop.fs.s3a.aws.credentials.provider",
            "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
    .getOrCreate()
)

meta = spark.read.parquet("s3a://stack-v3-full/metadata/")
contents = spark.read.parquet("s3a://stack-v3-full/contents/")

files = meta.select("repo_path", F.explode("files").alias("f"))
joined = (
    files.join(contents, files["f.content_id"] == contents["content_id"])
         .select("repo_path", "f.file_path", "f.language", "f.license_type", "content")
)
```

## Data fields

### `metadata` (one row per repository)

* `repo_path` (`string`): `owner/name` slug of the repository on GitHub.
* `repo_id` (`int64`): Numeric GitHub repository ID.
* `commit_id` (`string`): Commit (HEAD of the default branch) the snapshot was taken from.
* `github_metadata` (`struct`): Repository-level metadata scraped from the GitHub repository page: `branch`, `commit_count`, `repo_created_at`, `is_fork`, `is_org_owned`, `forked_from`, `stars`, `forks`, `issues`, `pull_requests` (see the [`stack-v3-train` card](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train#dataset-structure) for per-field descriptions).
* `num_files` (`int64`): Number of files in this repository row.
* `files` (`list[struct]`): Every file in the repository, including [stubs](#stub-entries):
  * `content_id` (`string`): SHA-1 of the UTF-8 file content. **Join key** into `contents`. `null` when the content was never captured (see stubs).
  * `file_path` (`string`): Path within the repository.
  * `file_timestamp` (`int64`): File modification time (Unix seconds). Only as accurate as the GitHub metadata.
  * `size_bytes` (`int64`): Length of the file content in UTF-8 bytes.
  * `language` (`string`): Language detected by [`go-enry`](https://github.com/go-enry/go-enry). `null` when detection failed (see stubs).
  * `is_generated` (`bool`): Whether the file is auto-generated (per `go-enry`). May be unreliable.
  * `is_vendor` (`bool`): Whether the file is vendored (per `go-enry`).
  * `detected_licenses` (`list[string]`): SPDX license identifiers detected by [ScanCode](https://github.com/nexB/scancode-toolkit).
  * `license_type` (`string`): `permissive` or `no_license`.
  * `dedup_cluster` (`int64`): Near-duplicate cluster ID — files sharing an ID are near-duplicates of each other. `null` for stubs and for very short files (fewer than 5 tokens) that were not indexed for deduplication.

### `contents` (one row per unique file blob)

* `content_id` (`string`): SHA-1 of the UTF-8 content. **Join key** into `metadata`.
* `content` (`string`): The UTF-8 file content (**not** PII-redacted — see the warning above).
* `size_bytes` (`int64`): Length in UTF-8 bytes.
* `dedup_cluster` (`int64`): Near-duplicate cluster ID.
* `repo_ids` (`list[int64]`): All repository IDs that contain this blob.

The partition directory (`language=…`) carries the detected language; there is no partition for undetected languages — those files exist only as stubs in `metadata`.

Jupyter notebooks in `contents` are the raw `.ipynb` JSON as crawled (outputs included, subject to the crawl-time size and binary limits below) — unlike `stack-v3-train`, where cell outputs, inline images, and volatile metadata are stripped. Use `stack-v3-full` if you want to apply your own notebook processing.

## Stub entries

`metadata` lists the **complete file listing** of every repository — 43.9B file entries, of which **28.3 billion** are *stubs*: files present in the repository but whose content is not included in `contents`. A stub is any entry with `content_id IS NULL OR language IS NULL`:

* **`content_id = null`** (13.5B entries): the content was never captured during the crawl — binary files, files larger than 5 MB, symlinks and other non-regular files. `language` is also `null` for these.
* **`content_id` set, `language = null`** (14.8B entries): the file was decoded as text, but `go-enry` could not detect a language (data files, PDFs-as-text, domain-specific formats, …). The content is **not** included in `contents`; other fields such as `size_bytes` and `license_type` may still be populated.

Stubs carry `file_path`, so together with the repository's `repo_path` and `commit_id` you can locate any excluded file on GitHub and re-crawl it yourself — e.g. if you need PDFs, binary assets, or a language that language detection missed.

## Near-duplicates: rolling your own dedup and filters

`stack-v3-full` deliberately retains **all** near-duplicates. MinHash signatures (256 permutations, 5-grams, min. 5 tokens) were indexed with LSH; candidate pairs with Jaccard similarity ≥ **0.7** (verified, language-agnostic) were grouped into clusters via connected components. Every file's cluster ID is exposed as `dedup_cluster` in both parts.

To deduplicate, keep one file per `dedup_cluster`. For `stack-v3-train`, the representative was chosen by (in order): highest stars, highest forks, permissive license, earliest repository creation — but with the cluster IDs and the pre-filtering signals (`is_generated`, `is_vendor`, licenses, `github_metadata`) you can apply any selection or quality filtering you like.

## Dataset statistics

Per-language and per-license breakdowns for the full corpus are published alongside the train dataset: [`stats/full/`](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train/tree/main/stats/full) (totals, by language, by license, stub counts, repository-size distributions).

## Licensing, opt-out, and limitations

The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0** [license](https://opendatacommons.org/licenses/by/1-0/).

Both released datasets contain only files classified as `permissive` or `no_license`; files with non-permissive detected licenses are excluded. Any use of the code must abide by the terms of the original licenses, including attribution; `repo_path`, `commit_id`, and `detected_licenses` provide provenance for each file.

Developers can check inclusion via the ["Am I in The Stack?"](https://huggingface.co/spaces/HuggingFaceCode/in-the-stack) Space and request removal following the [opt-out instructions](https://github.com/bigcode-project/opt-out-v2).

For the full documentation — dataset creation pipeline, license detection methodology, known limitations, social impact, and citation — see the [`stack-v3-train` dataset card](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train).