Use sccache on CI to reduce the build time #11520

Closed
opened 2026-01-31 02:50:02 +00:00 by claunia · 2 comments
Owner

Originally created by @skyline75489 on GitHub (Nov 21, 2020).

Description of the new feature/enhancement

I just found this amazing project sccache by Mozilla, which is a improved version of ccache and can be used to speed up the build.

I found it in the PyTorch CircleCI run like this one.

Per the documentation of sccache:

sccache is a ccache-like compiler caching tool. It is used as a compiler wrapper and avoids compilation when possible, storing cached results either on local disk or in one of several cloud storage backends.

sccache includes support for caching the compilation of C/C++ code, Rust, as well as NVIDIA's CUDA using nvcc.
sccache also provides icecream-style distributed compilation

Proposed technical implementation details (optional)

By simply storing cached compilation result on cloud storage, I think we can speed up the compilation dramatically. And there's more - distributed compilation!

I'm not sure Azure DevOps builtin sccache support, though. That would be a problem.

Originally created by @skyline75489 on GitHub (Nov 21, 2020). <!-- 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨 I ACKNOWLEDGE THE FOLLOWING BEFORE PROCEEDING: 1. If I delete this entire template and go my own path, the core team may close my issue without further explanation or engagement. 2. If I list multiple bugs/concerns in this one issue, the core team may close my issue without further explanation or engagement. 3. If I write an issue that has many duplicates, the core team may close my issue without further explanation or engagement (and without necessarily spending time to find the exact duplicate ID number). 4. If I leave the title incomplete when filing the issue, the core team may close my issue without further explanation or engagement. 5. If I file something completely blank in the body, the core team may close my issue without further explanation or engagement. All good? Then proceed! --> # Description of the new feature/enhancement I just found this amazing project [sccache](https://github.com/mozilla/sccache) by Mozilla, which is a improved version of ccache and can be used to speed up the build. I found it in the PyTorch CircleCI run like [this one](https://app.circleci.com/pipelines/github/pytorch/pytorch/241977/workflows/d0eb985d-94c9-428c-9c2e-e58bb8217fcf/jobs/9103296). Per the documentation of sccache: >sccache is a ccache-like compiler caching tool. It is used as a compiler wrapper and avoids compilation when possible, storing cached results either on local disk or in one of several cloud storage backends. > >sccache includes support for caching the compilation of C/C++ code, Rust, as well as NVIDIA's CUDA using nvcc. >sccache also provides icecream-style distributed compilation <!-- A clear and concise description of what the problem is that the new feature would solve. Describe why and how a user would use this new functionality (if applicable). --> # Proposed technical implementation details (optional) By simply storing cached compilation result on cloud storage, I think we can speed up the compilation dramatically. And there's more - distributed compilation! I'm not sure Azure DevOps builtin sccache support, though. That would be a problem. <!-- A clear and concise description of what you want to happen. -->
Author
Owner

@zadjii-msft commented on GitHub (Nov 23, 2020):

I'm not sure Azure DevOps builtin sccache support, though. That would be a problem.

That's certainly problem number 1. The other thought I have is, and I might just not be understanding how sccache works, but what artifacts would we exactly be caching? The majority of the build time is spent on pch.h generation, and we already can't share those across projects

@zadjii-msft commented on GitHub (Nov 23, 2020): > I'm not sure Azure DevOps builtin sccache support, though. That would be a problem. That's certainly problem number 1. The other thought I have is, and I might just not be understanding how `sccache` works, but what artifacts would we exactly be caching? The majority of the build time is spent on pch.h generation, and we already can't share those across projects
Author
Owner

@ghost commented on GitHub (Nov 27, 2020):

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

@ghost commented on GitHub (Nov 27, 2020): This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for **4 days**. It will be closed if no further activity occurs **within 3 days of this comment**.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#11520