Datasets:
Tensor Valley
Add instructed_lint_grpo rule frequency stats artifact (link in README)
6123dd6 verified | license: other | |
| license_name: multiple | |
| license_link: LICENSE | |
| task_categories: | |
| - text-generation | |
| language: | |
| - code | |
| tags: | |
| - python | |
| - lint | |
| - ruff | |
| - code-quality | |
| - the-stack | |
| - static-analysis | |
| pretty_name: Instructed Lint Python Files | |
| size_categories: | |
| - 10M<n<100M | |
| extra_gated_prompt: >- | |
| ## Terms of Use | |
| This dataset is derived from [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup), | |
| which is a gated dataset with its own Terms of Use. **Before accessing this dataset, you must first | |
| accept the Terms of Use for The Stack** by visiting | |
| [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) and clicking | |
| "Access repository". | |
| By requesting access to this dataset, you acknowledge that: | |
| 1. **You have accepted The Stack's Terms of Use** at | |
| [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup). | |
| 2. **License compliance**: This dataset contains source code from repositories with various | |
| licenses. Any use of the code must abide by the terms of the original licenses, including | |
| attribution clauses when relevant. License information is provided in the `max_stars_repo_licenses` | |
| field of each record. | |
| 3. **Data removal**: The Stack is regularly updated to enact validated data removal requests. | |
| You agree to update your version of this dataset when notified of removals. Follow the | |
| [update thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7) for | |
| notifications, and use the [community discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new) | |
| for questions about dataset versions and allowed uses. | |
| 4. **Redistribution**: To host, share, or otherwise provide access to this dataset, you must | |
| include these Terms of Use and require users to agree to them. | |
| 5. **Contact sharing**: Your contact information (email address and username) may be shared | |
| with the dataset maintainers. | |
| 6. **Security warning**: This dataset contains raw source code from public GitHub repositories. | |
| Some files may contain security vulnerabilities, malicious code, or exploits. **Do not execute | |
| any code from this dataset without thorough security review.** The lint annotations may help | |
| identify some issues but are not a substitute for security auditing. | |
| 7. **No warranty**: This dataset is provided "as is" without warranty of any kind. The lint | |
| annotations are generated automatically by ruff and may contain false positives or miss | |
| real issues. | |
| extra_gated_fields: | |
| Email: text | |
| I have accepted The Stack's Terms of Use at bigcode/the-stack-dedup: checkbox | |
| I agree to update my copy when notified of data removals: checkbox | |
| I will comply with the original licenses of included source code: checkbox | |
| # Instructed Lint Python Files | |
| Lint-annotated Python source code from [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup), | |
| processed with [ruff](https://github.com/astral-sh/ruff) (all 800 stable rules enabled). | |
| ## Dataset Description | |
| This dataset pairs **12,962,249 Python files** from The Stack (deduplicated) with their complete | |
| ruff lint diagnostics. Each record contains the original source code, file metadata, license | |
| information, and structured lint results. | |
| ### Motivation | |
| Building training data for code quality models requires large-scale lint annotations. Running | |
| ruff at scale on millions of files is expensive with the standard CLI approach. We built | |
| [ruff-batch](https://github.com/astral-sh/ruff), a Rust binary that calls ruff's internal | |
| `lint_only()` API directly via Rayon parallel iteration, achieving **26,337 files/sec** with all | |
| 800 rules enabled — a **10x speedup** over the ruff CLI. | |
| ### Source Data | |
| - **Origin**: [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup), Python split | |
| - **Files**: 12,962,249 | |
| - **Linter**: ruff (all 800 stable rules, preview disabled) | |
| - **Processing time**: 8.2 minutes on Apple M3 Ultra (28 cores) | |
| ## Dataset Structure | |
| Each record contains: | |
| | Field | Type | Description | | |
| |-------|------|-------------| | |
| | `id` | string | Unique file identifier (git hexsha from The Stack) | | |
| | `code` | string | Python source code | | |
| | `path` | string | Original file path in the source repository | | |
| | `max_stars_repo_licenses` | list[string] | SPDX license identifiers for the most-starred repo containing this file | | |
| | `max_stars_repo_name` | string | GitHub repository name (owner/repo) | | |
| | `size` | int | File size in bytes | | |
| | `diagnostics` | list[object] | Lint diagnostics (see below) | | |
| | `num_diagnostics` | int | Number of lint issues found | | |
| | `valid_syntax` | bool | Whether the file has valid Python syntax | | |
| | `lint_elapsed_us` | int | Microseconds spent linting this file | | |
| ### Diagnostic Structure | |
| Each diagnostic in the `diagnostics` list contains: | |
| | Field | Type | Description | | |
| |-------|------|-------------| | |
| | `code` | string | Ruff rule code (e.g., "F401", "E711") | | |
| | `message` | string | Human-readable description of the issue | | |
| | `row` | int | 1-indexed start line number | | |
| | `col` | int | 1-indexed start column number | | |
| | `end_row` | int | 1-indexed end line number | | |
| | `end_col` | int | 1-indexed end column number | | |
| ## Statistics | |
| - **Total files**: 12,962,249 | |
| - **Valid syntax**: 95.6% | |
| - **Average diagnostics per file**: ~102 (with all 800 rules) | |
| - **Total diagnostics**: ~1.3 billion | |
| ## Usage | |
| ```python | |
| from datasets import load_dataset | |
| ds = load_dataset("tensorvalley/instructed_lint_python_files", split="train", streaming=True) | |
| for record in ds: | |
| print(f"File: {record['path']}") | |
| print(f"License: {record['max_stars_repo_licenses']}") | |
| print(f"Issues: {record['num_diagnostics']}") | |
| for diag in record['diagnostics']: | |
| span = f"L{diag['row']}:{diag['col']}-L{diag['end_row']}:{diag['end_col']}" | |
| print(f" {diag['code']} [{span}]: {diag['message']}") | |
| break | |
| ``` | |
| ## Licensing and Attribution | |
| This dataset contains source code from GitHub repositories with **various open-source licenses**. | |
| The license for each file is provided in the `max_stars_repo_licenses` field. **Any use of the | |
| code must comply with the terms of the original licenses**, including attribution requirements. | |
| The lint annotations themselves are provided under [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0). | |
| For data provenance, each record includes: | |
| - `id`: The git hexsha, which can be used to trace back to the original commit | |
| - `max_stars_repo_name`: The GitHub repository name for attribution | |
| - `path`: The original file path within the repository | |
| ## Data Removal | |
| This dataset inherits The Stack's data removal policy. If your code is included and you wish to | |
| have it removed, please follow the opt-out process at | |
| [Am I in The Stack?](https://huggingface.co/spaces/bigcode/in-the-stack) | |
| When The Stack publishes data removal updates, this dataset will be updated accordingly. Follow | |
| the [update thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7) for | |
| notifications. | |
| ## Security Warning | |
| **This dataset contains raw source code from public GitHub repositories. Some files may contain:** | |
| - Security vulnerabilities (SQL injection, XSS, command injection, etc.) | |
| - Intentionally malicious code | |
| - Backdoors or exploits | |
| - Credential leaks or hardcoded secrets | |
| **Do not execute any code from this dataset without thorough security review.** The ruff lint | |
| annotations identify some code quality issues but are **not a security audit** and should not | |
| be treated as one. | |
| ## Citation | |
| ```bibtex | |
| @misc{{instructed_lint_python_files, | |
| title={{Instructed Lint Python Files}}, | |
| author={{Tensor Valley}}, | |
| year={{2026}}, | |
| howpublished={{\url{{https://huggingface.co/datasets/tensorvalley/instructed_lint_python_files}}}} | |
| }} | |
| ``` | |
| If you use the underlying source code, please also cite The Stack: | |
| ```bibtex | |
| @inproceedings{{kocetkov2022thestack, | |
| title={{The Stack: 3 TB of permissively licensed source code}}, | |
| author={{Denis Kocetkov and Raymond Li and Loubna Ben Allal and Jia Li and Chenghao Mou | |
| and Carlos Mu{{\~n}}oz Ferrandis and Yacine Jernite and Margaret Mitchell | |
| and Sean Hughes and Thomas Wolf and Dzmitry Bahdanau and Leandro von Werra | |
| and Harm de Vries}}, | |
| booktitle={{Transactions on Machine Learning Research}}, | |
| year={{2022}} | |
| }} | |
| ``` | |
| ## Terms of Use for The Stack | |
| The Stack dataset is a collection of source code in over 300 programming languages. We ask that | |
| you read and acknowledge the following points before using the dataset: | |
| 1. The Stack is a collection of source code from repositories with various licenses. Any use of | |
| all or part of the code gathered in The Stack must abide by the terms of the original licenses, | |
| including attribution clauses when relevant. We facilitate this by providing provenance | |
| information for each data point. | |
| 2. The Stack is regularly updated to enact validated data removal requests. By accessing this | |
| dataset, you agree to update your own version of The Stack to the most recent usable version | |
| specified by the maintainers in the | |
| [following thread](https://huggingface.co/datasets/bigcode/the-stack/discussions/7). If you | |
| have questions about dataset versions and allowed uses, please also ask them in the dataset's | |
| [community discussions](https://huggingface.co/datasets/bigcode/the-stack/discussions/new). | |
| We will also notify users via email when the latest usable version changes. | |
| 3. To host, share, or otherwise provide access to The Stack dataset, you must include these | |
| Terms of Use and require users to agree to it. | |
| 4. By accessing this dataset, you accept that your contact information (email address and | |
| username) can be shared with the dataset maintainers as well. | |
| ## Artifacts | |
| - Rule frequency stats (`rule_stats.json`): per-rule counts over the dataset split (how many files contain each diagnostic code + total diagnostic occurrences), used to pick a balanced ~100-rule subset for GRPO. | |
| - https://huggingface.co/datasets/tensorvalley/instructed_lint_python_files/blob/main/artifacts/rule_stats.json | |
| - Rule stats: https://huggingface.co/datasets/tensorvalley/instructed_lint_python_files/blob/main/artifacts/rule_stats.json | |