An Elegant Puzzle: Systems of Engineering Management
by
Will Larson
Published 19 May 2019
This paper is particularly impressive when you consider the difficulties that companies run into as they scale Git monorepos (talk to an ex-Twitter employee near you for war stories). “Large-Scale Automated Refactoring Using ClangMR” Large codebases tend to age poorly, especially in the case of monorepos storing hundreds or thousands of different teams collaborating on different projects. This paper covers one of Google’s attempts to reduce the burden of maintaining their large monorepo through tooling that makes it easy to rewrite abstract syntax trees (ASTs) across the entire codebase. From the abstract: In this paper, we present a real-world implementation of a system to refactor large C++ codebases efficiently.
Software Engineering at Google: Lessons Learned From Programming Over Time
by
Titus Winters
,
Tom Manshreck
and
Hyrum Wright
Published 17 Mar 2020
Effectively, it’s never safe to add a new dependency from a target to any third-party library with multiple versions, since any of that target’s users could already be depending on a different version. Following the one-version rule makes this conflict impossible - if a target adds a dependency on a third-party library, any existing dependencies will already be on that same version, so they can happily coexist. The one-version rule is discussed further in the context of a large monorepo in “Dependency Management.” Transitive External Dependencies Dealing with the transitive dependencies of an external dependency can be particularly difficult. Many artifact repositories such as Maven Central allow artifacts to specify dependencies on particular versions of other artifacts in the repository.
…
This effectively means that all of the project’s external dependencies are converted to internal dependencies. Google uses this approach internally, checking every third-party library referenced throughout Google into a third_party directory at the root of Google’s source tree. However, this works at Google only because Google’s source control system is custom-built to handle an extremely large monorepo, so vendoring might not be an option for other organizations. Conclusion A build system is one of the most important parts of an engineering organization. Each developer will interact with it potentially dozens or hundreds of times per day, and in many situations it can be the rate-limiting step in determining their productivity.
Working in Public: The Making and Maintenance of Open Source Software
by
Nadia Eghbal
Published 3 Aug 2020
(Reading: Addison-Wesley, 1995), 121. 214 Devon Zuegel, “The City Guide to Open Source,” Increment 9, May 2019, https://increment.com/open-source/the-city-guide-to-open-source/. 215 Robert Glass, Facts and Fallacies of Software Engineering (Boston: Addison-Wesley, 2010), 174. 216 “Section 230 of the Communications Decency Act,” Electronic Frontier Foundation, n.d., https://www.eff.org/issues/cda230. 217 Ben Balter, “Open Source License Usage on GitHub.com,” The GitHub Blog, GitHub, March 9, 2015, https://github.blog/2015-03-09-open-source-license-usage-on-github-com/. 218 “The MIT License,” Open Source Initiative, n.d., https://opensource.org/licenses/MIT. 219 “Open Source Survey,” Open Source Survey, 2017, https://opensourcesurvey.org/2017/. 220 “The Developer Coefficient: A $300B Opportunity for Business,” Stripe, September 2018, https://stripe.com/reports/developer-coefficient-2018. 221 Ensmenger, “When Good Software Goes Bad.” 222 “About Required Status Checks,” GitHub Help, n.d., https://help.github.com/en/github/administering-a-repository/about-required-status-checks. 223 Alan Zeino, “Faster Together: Uber Engineering’s IOS Monorepo,” Uber Engineering (blog), March 6, 2017, https://eng.uber.com/ios-monorepo/ 224 Steve Klabnik (@steveklabnik), “Today I glanced at some numbers . . .,” Twitter, June 14, 2019, 11:17 a.m., https://twitter.com/steveklabnik/status/1139552342842458112. 225 “The State of the Octoverse,” GitHub, 2019, https://octoverse.github.com/. 226 Russ Cox, “Our Software Dependency Problem,” Research!
…
Is-sorted, a module that checks if an array is sorted and contains less than thirty lines of code, counts as one repository.77 So does CPython, a repository that contains the entire codebase for the Python programming language.78 As with anything else, the size and structure of a repository are influenced by technical decisions and personal preferences. Some projects like to break up their code into many repositories, each of which contains a smaller amount of code. These repositories can all live under a single GitHub organization. Other projects adopt the monorepo philosophy, managing large amounts of code under a single repository. GitHub repository for CPython.79 At minimum, open source projects hosted on GitHub can be broken into three parts: code (the final output of a project), an issue tracker (a way to discuss changes), and pull requests (a way to make changes).